index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
*
html ingest: remove whitespace around relative URLs (eg, for d-lib)
Bryan Newbold
2021-05-21
1
-1
/
+1
*
add cdx_collection.py python script (from scratch repo)
Bryan Newbold
2021-05-04
1
-0
/
+80
*
sql: do periodically retry spn2-wayback-error
Bryan Newbold
2021-04-27
2
-2
/
+0
*
ingest: cap max body size to ~128 MByte
Bryan Newbold
2021-04-27
1
-0
/
+6
*
persist: skip very long URLs
Bryan Newbold
2021-04-12
1
-0
/
+4
*
reingest scripts to run as sandcrawler
Bryan Newbold
2021-04-09
2
-12
/
+12
*
sql: notes on sql restore
Bryan Newbold
2021-04-09
1
-0
/
+9
*
sql: update paths to work with svc506 machine
Bryan Newbold
2021-04-09
12
-49
/
+49
*
sql: before/after pg13 table size stats
Bryan Newbold
2021-04-09
2
-1
/
+43
*
sql: update periodic retry/reingest scripts
Bryan Newbold
2021-04-09
4
-6
/
+14
*
update default postgrest ('db') API endpoint
Bryan Newbold
2021-04-09
1
-1
/
+1
*
SQL snapshot doc update
Bryan Newbold
2021-04-07
1
-2
/
+5
*
gitlab CI: make python_hadoop tests manual
Bryan Newbold
2021-04-07
1
-0
/
+1
*
gitlab-ci: gpg-agent
Bryan Newbold
2021-04-07
1
-1
/
+1
*
gitlab CI: switch to ubuntu:focal
Bryan Newbold
2021-04-07
1
-14
/
+10
*
2021-04-07 sandcrawler DB stats
Bryan Newbold
2021-04-07
1
-0
/
+428
*
grobid: disable biblio-glutton consolidation
Bryan Newbold
2021-04-07
1
-3
/
+3
*
ingest: handle current degruyter PDF link pattern
Bryan Newbold
2021-03-26
1
-0
/
+8
*
add missing dotfiles (due to gitignore oops)
Bryan Newbold
2021-01-18
2
-0
/
+12
*
pipenv: lock minio S3 library to <7.0.0
Bryan Newbold
2021-01-14
2
-242
/
+196
*
more expansive python/.gitignore rules (all .gz)
Bryan Newbold
2021-01-05
1
-1
/
+1
*
doaj ingest request updates (from prod)
Bryan Newbold
2021-01-05
1
-1
/
+5
*
gitlab CI: update kitware (updated cmake on xenial) signing key
Bryan Newbold
2021-01-05
1
-2
/
+2
*
gitlab CI: add libmagic1 dep (was a recommends)
Bryan Newbold
2021-01-05
1
-1
/
+1
*
gitlab CI: add build deps
Bryan Newbold
2021-01-05
1
-1
/
+1
*
gitlab CI: install fewer packages (no recommends)
Bryan Newbold
2021-01-05
1
-2
/
+2
*
python makefile: don't duplicate 'lint' commands during 'test'
Bryan Newbold
2021-01-05
1
-2
/
+0
*
update to python3.8
Bryan Newbold
2021-01-05
4
-403
/
+417
*
pdf: yet more bad SHA1 (commiting lines from prod)
Bryan Newbold
2021-01-05
1
-0
/
+20
*
ia CDX: handle bad CDX rows
Bryan Newbold
2021-01-05
1
-2
/
+4
*
late-2020 OA DOI crawl ingest notes
Bryan Newbold
2021-01-04
1
-3
/
+46
*
DOAJ crawl ingest stats
Bryan Newbold
2020-12-31
1
-0
/
+295
*
progress notes on OA DOI ingest (still running)
Bryan Newbold
2020-12-28
1
-11
/
+102
*
update HTML ingest proposal
Bryan Newbold
2020-12-23
1
-1
/
+3
*
spn: more status codes
Bryan Newbold
2020-12-21
1
-1
/
+2
*
HTML ingest deployment notes
Bryan Newbold
2020-12-16
1
-1
/
+71
*
persist: html_meta is ON CONFLICT DO UPDATE
Bryan Newbold
2020-12-15
1
-1
/
+1
*
persist: don't expect HTML TEI-XML in result object
Bryan Newbold
2020-12-15
1
-1
/
+1
*
unpaywall crawl/ingest update (from Oct 2020)
Bryan Newbold
2020-12-08
1
-0
/
+134
*
commit sept 2020 scielo ingest notes
Bryan Newbold
2020-12-08
1
-0
/
+21
*
handle more wayback error conditions
Bryan Newbold
2020-11-20
1
-0
/
+6
*
kafka docs for rolling back a consumer group
Bryan Newbold
2020-11-20
1
-0
/
+9
*
html: more conservative parsing of element attr
Bryan Newbold
2020-11-20
1
-2
/
+4
*
xml: catch parse error
Bryan Newbold
2020-11-19
1
-3
/
+8
*
SQL: more ingest monitoring
Bryan Newbold
2020-11-16
3
-1
/
+660
*
spn 'forbidden' status code
Bryan Newbold
2020-11-12
1
-1
/
+1
*
html biblio: handle 'content not in attrs' case
Bryan Newbold
2020-11-12
1
-2
/
+2
*
add implementation notes about HTML ingest
Bryan Newbold
2020-11-10
1
-0
/
+248
*
fuzzy matching notes
Bryan Newbold
2020-11-10
1
-0
/
+148
*
blacklist -> denylist
Bryan Newbold
2020-11-10
2
-9
/
+9
[next]