sandcrawler - [no description]

	Commit message (Collapse)	Author	Age	Files	Lines
*	rename FileIngestWorker	Bryan Newbold	2019-11-13	2	-5/+10
\|
*	refactor consume_topic name out of make_kafka_consumer()	Bryan Newbold	2019-11-13	1	-5/+5
\| \| \| \|	Best to do this in wrapping code for full flexibility.
*	more progress on file ingest	Bryan Newbold	2019-11-13	3	-16/+73
\|
*	much progress on file ingest path	Bryan Newbold	2019-10-22	5	-15/+334
\|
*	we do actually want consolidateHeader=2, not 1	Bryan Newbold	2019-10-04	1	-3/+3
\|
*	grobid: consolidateHeaders typo	Bryan Newbold	2019-10-04	1	-1/+1
\|
*	disable citation consolidation by default	Bryan Newbold	2019-10-04	1	-1/+1
\| \| \| \| \| \| \|	with this consolidation enabled, the glutton_fatcat elasticsearch server was totally pegged over 90% CPU with only 10 PDF worker threads; the glutton load seemed to be the bottleneck even for this low degree of parallelism. Disabled for now, will debug with GROBID/glutton folks.
*	fix GROBID POST flags	Bryan Newbold	2019-10-04	1	-1/+3
\|
*	workers: better generic batch-size arg handling	Bryan Newbold	2019-10-03	1	-0/+6
\|
*	handle GROBID fetch empty blob condition	Bryan Newbold	2019-10-03	1	-1/+2
\|
*	have grobidworker error status indicate issues instead of bailing	Bryan Newbold	2019-10-02	1	-4/+13
\|
*	more counts and bugfixes in grobid_tool	Bryan Newbold	2019-09-26	2	-4/+6
\|
*	off-by-one error in batch sizes	Bryan Newbold	2019-09-26	1	-1/+1
\|
*	small improvements to GROBID tool	Bryan Newbold	2019-09-26	1	-0/+4
\|
*	lots of grobid tool implementation (still WIP)	Bryan Newbold	2019-09-26	5	-9/+632
\|
*	re-write parse_cdx_line for sandcrawler lib	Bryan Newbold	2019-09-25	2	-1/+85
\|
*	start refactoring sandcrawler python common code	Bryan Newbold	2019-09-23	3	-0/+90