sandcrawler - [no description]

	Commit message (Collapse)	Author	Age	Files	Lines
*	skip-db option also for worker	Bryan Newbold	2020-03-19	1	-0/+4
\|
*	ingest: bulk workers don't hit SPNv2	Bryan Newbold	2020-02-13	1	-0/+2
\|
*	pdftrio basic python code	Bryan Newbold	2020-02-12	1	-0/+19
\| \| \| \|	This is basically just a copy/paste of GROBID code, only simpler!
*	sandcrawler_worker: ingest worker distinct consumer groups	Bryan Newbold	2020-01-29	1	-1/+3
\| \| \| \| \| \|	I'm in the process of resetting these consumer groups, so might as well take the opportunity to split by topic and use the new canonical naming format.
*	make grobid-extract worker batch size 1	Bryan Newbold	2020-01-28	1	-0/+1
\| \| \| \| \|	This is part of attempts to fix Kafka errors that look like they might be timeouts.
*	improve sentry reporting with 'release' git hash	Bryan Newbold	2020-01-15	1	-1/+5
\|
*	bulk ingest file request topic support	Bryan Newbold	2020-01-14	1	-1/+7
\|
*	grobid-to-kafka support in ingest worker	Bryan Newbold	2020-01-14	1	-0/+6
\|
*	update persist worker invocation to use batches	Bryan Newbold	2020-01-02	1	-15/+55
\|
*	fix sandcrawler persist workers	Bryan Newbold	2020-01-02	1	-8/+36
\|
*	start work on persist workers and tool	Bryan Newbold	2020-01-02	1	-5/+15
\|
*	refactor: improve argparse usage	Bryan Newbold	2019-12-18	1	-4/+8
\| \| \| \| \|	use ArgumentDefaultsHelpFormatter and add help messages to all sub-commands
*	update ingest-file batch size to 1	Bryan Newbold	2019-11-14	1	-1/+1
\| \| \| \| \| \| \| \|	Was defaulting to 100, which I think was resulting in lots of consumer group timeouts, resulting in UNKNOWN_MEMBER_ID errors. Will probably switch back to batches of 10 or so, but multi-processing or some other concurrent dispatch/processing.
*	fix lint errors	Bryan Newbold	2019-11-13	1	-5/+10
\|
*	correct ingest-file consumer group	Bryan Newbold	2019-11-13	1	-1/+1
\|
*	add basic sandcrawler worker (kafka)	Bryan Newbold	2019-11-13	1	-0/+74