index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
*
pig: move count_lines helper to pighelper.py
Bryan Newbold
2019-12-22
3
-7
/
+6
*
refactor: use print(..., file=sys.stderr)
Bryan Newbold
2019-12-18
5
-32
/
+34
*
refactor: sort keys in JSON output
Bryan Newbold
2019-12-18
4
-6
/
+7
*
refactor: improve argparse usage
Bryan Newbold
2019-12-18
5
-13
/
+27
*
update ingest proposal source/link naming
Bryan Newbold
2019-12-13
2
-17
/
+27
*
sql schema change proposals
Bryan Newbold
2019-12-11
1
-0
/
+40
*
pdftotext proposal
Bryan Newbold
2019-12-11
1
-0
/
+123
*
add some GROBID metadata schema docs to SQL schema
Bryan Newbold
2019-12-11
1
-0
/
+11
*
update ingest proposal
Bryan Newbold
2019-12-11
1
-11
/
+145
*
fixes for large GROBID result skip
Bryan Newbold
2019-12-02
1
-2
/
+2
*
count empty blobs as 'failed' instead of crashing
Bryan Newbold
2019-12-01
1
-1
/
+2
*
cleanup unused import
Bryan Newbold
2019-12-01
1
-1
/
+0
*
filter out very large GROBID XML bodies
Bryan Newbold
2019-12-01
1
-0
/
+6
*
updated re-GROBID job log entry
Bryan Newbold
2019-11-15
1
-0
/
+31
*
CI: make some jobs manual
Bryan Newbold
2019-11-15
2
-6
/
+12
*
handle wayback fetch redirect loop in ingest code
Bryan Newbold
2019-11-14
1
-2
/
+5
*
bump kafka max poll interval for consumers
Bryan Newbold
2019-11-14
1
-2
/
+2
*
handle WaybackError during ingest
Bryan Newbold
2019-11-14
1
-0
/
+4
*
handle SPNv1 redirect loop
Bryan Newbold
2019-11-14
1
-0
/
+2
*
handle SPNv2 polling timeout
Bryan Newbold
2019-11-14
1
-6
/
+10
*
update ingest-file batch size to 1
Bryan Newbold
2019-11-14
2
-4
/
+4
*
start of hrmars.com ingest support
Bryan Newbold
2019-11-14
2
-2
/
+7
*
treat failure to get terminal capture as a SavePageNowError
Bryan Newbold
2019-11-13
1
-1
/
+1
*
citation_pdf_url with host-relative URLs
Bryan Newbold
2019-11-13
1
-1
/
+3
*
status_forcelist is on session, not request
Bryan Newbold
2019-11-13
1
-2
/
+2
*
handle SPNv1 remote server HTTP status codes better
Bryan Newbold
2019-11-13
1
-8
/
+15
*
grobid2json: make lang detection flexible
Bryan Newbold
2019-11-13
1
-1
/
+2
*
handle requests (http) redirect loop from wayback
Bryan Newbold
2019-11-13
1
-1
/
+4
*
handle wayback client return status correctly
Bryan Newbold
2019-11-13
1
-2
/
+2
*
allow way more errors in SPN path
Bryan Newbold
2019-11-13
1
-2
/
+11
*
clean up redirect-following CDX API path
Bryan Newbold
2019-11-13
1
-8
/
+15
*
fix lint errors
Bryan Newbold
2019-11-13
2
-6
/
+11
*
improve ingest worker remote failure behavior
Bryan Newbold
2019-11-13
1
-5
/
+12
*
have SPN client differentiate between SPN and remote errors
Bryan Newbold
2019-11-13
2
-3
/
+11
*
correct ingest-file consumer group
Bryan Newbold
2019-11-13
1
-1
/
+1
*
new kafka topics for file ingest
Bryan Newbold
2019-11-13
1
-0
/
+15
*
ingest/backfill notes
Bryan Newbold
2019-11-13
3
-0
/
+47
*
notes about running 'regrobid' batches manually (not kafka)
Bryan Newbold
2019-11-13
1
-0
/
+41
*
commit old notes about munging GROBID output
Bryan Newbold
2019-11-13
1
-0
/
+70
*
add structure of ingest proposal
Bryan Newbold
2019-11-13
1
-0
/
+129
*
add basic sandcrawler worker (kafka)
Bryan Newbold
2019-11-13
1
-0
/
+74
*
note that kafka_grobid.py is deprecated
Bryan Newbold
2019-11-13
1
-0
/
+3
*
rename FileIngestWorker
Bryan Newbold
2019-11-13
3
-10
/
+16
*
refactor consume_topic name out of make_kafka_consumer()
Bryan Newbold
2019-11-13
1
-5
/
+5
*
more progress on file ingest
Bryan Newbold
2019-11-13
4
-17
/
+75
*
add note to CDX backfill script that we should be filtering (oops)
Bryan Newbold
2019-11-12
1
-0
/
+1
*
SQL stats and commands (mostly from sept 2019)
Bryan Newbold
2019-11-12
4
-0
/
+96
*
much progress on file ingest path
Bryan Newbold
2019-10-22
6
-335
/
+338
*
remove spurious debug print from grobid2json
Bryan Newbold
2019-10-22
1
-1
/
+1
*
new/additional GWB CDX filter scripts
Bryan Newbold
2019-10-17
7
-0
/
+142
[next]