sandcrawler - [no description]

	Commit message (Collapse)	Author	Age	Files	Lines
*	html: actually publish HTML TEI-XML to body; fix dataflow though ingest a bit	Bryan Newbold	2020-11-04	1	-0/+6
\|
*	small fixes from local testing for XML ingest	Bryan Newbold	2020-11-03	1	-2/+0
\|
*	persist: XML and HTML persist workers	Bryan Newbold	2020-11-03	1	-0/+47
\|
*	ingest: handle publishing XML docs to kafka	Bryan Newbold	2020-11-03	1	-0/+6
\|
*	refactor 'minio' to 'seaweedfs'; and BLOB env vars	Bryan Newbold	2020-11-03	1	-10/+10
\| \| \| \| \|	This goes along with changes to ansible deployment to use the correct key names and values.
*	better default CLI output (show usage)	Bryan Newbold	2020-10-29	1	-1/+1
\|
*	persist PDF extraction in ingest pipeline	Bryan Newbold	2020-10-20	1	-4/+16
\| \| \| \| \|	Ooof, didn't realize that this wasn't happening. Explains a lot of missing thumbnails in scholar!
*	customize timeout per worker; 120sec for pdf-extract	Bryan Newbold	2020-06-29	1	-0/+1
\| \| \| \| \|	This is a stab-in-the-dark attempt to resolve long timeouts with this worker in prod.
*	args.kafka_env refactor didn't happen (yet)	Bryan Newbold	2020-06-25	1	-2/+2
\|
*	s3-only mode persist workers use different consumer group	Bryan Newbold	2020-06-25	1	-2/+8
\|
*	sandcrawler_worker: remove duplicate run_pdf_extract()	Bryan Newbold	2020-06-25	1	-29/+0
\|
*	pdfextract worker	Bryan Newbold	2020-06-25	1	-1/+34
\|
*	fixes and tweaks from testing locally	Bryan Newbold	2020-06-17	1	-1/+2
\|
*	tweak kafka topic names and seaweedfs layout	Bryan Newbold	2020-06-17	1	-9/+10
\|
*	add new pdf workers/persisters	Bryan Newbold	2020-06-17	1	-0/+83
\|
*	skip-db option also for worker	Bryan Newbold	2020-03-19	1	-0/+4
\|
*	ingest: bulk workers don't hit SPNv2	Bryan Newbold	2020-02-13	1	-0/+2
\|
*	pdftrio basic python code	Bryan Newbold	2020-02-12	1	-0/+19
\| \| \| \|	This is basically just a copy/paste of GROBID code, only simpler!
*	sandcrawler_worker: ingest worker distinct consumer groups	Bryan Newbold	2020-01-29	1	-1/+3
\| \| \| \| \| \|	I'm in the process of resetting these consumer groups, so might as well take the opportunity to split by topic and use the new canonical naming format.
*	make grobid-extract worker batch size 1	Bryan Newbold	2020-01-28	1	-0/+1
\| \| \| \| \|	This is part of attempts to fix Kafka errors that look like they might be timeouts.
*	improve sentry reporting with 'release' git hash	Bryan Newbold	2020-01-15	1	-1/+5
\|
*	bulk ingest file request topic support	Bryan Newbold	2020-01-14	1	-1/+7
\|
*	grobid-to-kafka support in ingest worker	Bryan Newbold	2020-01-14	1	-0/+6
\|
*	update persist worker invocation to use batches	Bryan Newbold	2020-01-02	1	-15/+55
\|
*	fix sandcrawler persist workers	Bryan Newbold	2020-01-02	1	-8/+36
\|
*	start work on persist workers and tool	Bryan Newbold	2020-01-02	1	-5/+15
\|
*	refactor: improve argparse usage	Bryan Newbold	2019-12-18	1	-4/+8
\| \| \| \| \|	use ArgumentDefaultsHelpFormatter and add help messages to all sub-commands
*	update ingest-file batch size to 1	Bryan Newbold	2019-11-14	1	-1/+1
\| \| \| \| \| \| \| \|	Was defaulting to 100, which I think was resulting in lots of consumer group timeouts, resulting in UNKNOWN_MEMBER_ID errors. Will probably switch back to batches of 10 or so, but multi-processing or some other concurrent dispatch/processing.
*	fix lint errors	Bryan Newbold	2019-11-13	1	-5/+10
\|
*	correct ingest-file consumer group	Bryan Newbold	2019-11-13	1	-1/+1
\|
*	add basic sandcrawler worker (kafka)	Bryan Newbold	2019-11-13	1	-0/+74