sandcrawler - [no description]

	Commit message (Collapse)	Author	Age	Files	Lines
...
*	remove unused pytype tool	Bryan Newbold	2020-11-06	3	-76/+25
\| \| \| \| \|	Having trouble getting this to install on Xenial, and we aren't even using it in tests/lint yet. Can revisit after Focal upgrade.
*	many bad PDF sha1 from prod	Bryan Newbold	2020-11-06	1	-0/+36
\|
*	html: catch and report exceptions at process_hit() stage	Bryan Newbold	2020-11-06	1	-4/+27
\|
*	html: pdf and html extract similar to XML	Bryan Newbold	2020-11-06	2	-22/+55
\| \| \| \|	Note that the primary PDF URL extraction path is a separate code path.
*	html: refactors/tweaks from testing	Bryan Newbold	2020-11-06	3	-17/+23
\|
*	ia: use newer gwb (petabox) loading class	Bryan Newbold	2020-11-04	1	-5/+8
\| \| \| \|	This fixes zstandard WARC reading.
*	pipenv: fix lock file; add zstandard; update wayback+gwb deps	Bryan Newbold	2020-11-04	2	-27/+1069
\|
*	persist: fix worker API/typing hacks (raw_key, key, key_str)	Bryan Newbold	2020-11-04	1	-9/+9
\|
*	html: actually publish HTML TEI-XML to body; fix dataflow though ingest a bit	Bryan Newbold	2020-11-04	2	-5/+31
\|
*	initial implementation of HTML ingest in existing worker	Bryan Newbold	2020-11-04	4	-15/+94
\|
*	small fixes from local testing for XML ingest	Bryan Newbold	2020-11-03	3	-6/+9
\|
*	xml: re-encode XML docs into UTF-8 for persisting	Bryan Newbold	2020-11-03	4	-1/+364
\|
*	persist: XML and HTML persist workers	Bryan Newbold	2020-11-03	2	-3/+121
\|
*	ingest: handle publishing XML docs to kafka	Bryan Newbold	2020-11-03	2	-3/+27
\|
*	refactor 'minio' to 'seaweedfs'; and BLOB env vars	Bryan Newbold	2020-11-03	5	-25/+27
\| \| \| \| \|	This goes along with changes to ansible deployment to use the correct key names and values.
*	gen_file_metadata: detect JATS XML and use application/jats+xml	Bryan Newbold	2020-11-03	1	-0/+4
\|
*	basic support for XML ingest in worker	Bryan Newbold	2020-11-03	1	-23/+40
\|
*	html: improve XML fulltext extraction for scielo	Bryan Newbold	2020-11-03	1	-4/+17
\|
*	tweak html_meta SQL schema	Bryan Newbold	2020-11-03	1	-12/+19
\|
*	html: some refactoring	Bryan Newbold	2020-11-03	3	-24/+57
\|
*	move transfer encoding helper to sandcrawler/ia.py	Bryan Newbold	2020-11-03	2	-23/+42
\|
*	ingest: cleanups, typing, start generalizing to xml and html	Bryan Newbold	2020-11-03	1	-122/+118
\|
*	ingest: tweak debug printing alignment	Bryan Newbold	2020-11-03	2	-11/+10
\|
*	pytest: ignore DW in justext	Bryan Newbold	2020-11-03	1	-0/+1
\|
*	ingest: add more IA domains	Bryan Newbold	2020-11-03	1	-0/+2
\|
*	cdx: fix 'closest' support	Bryan Newbold	2020-11-03	1	-3/+2
\|
*	html: start on SQL table	Bryan Newbold	2020-11-03	1	-0/+44
\|
*	html: syntax fixes; resolve relative URLs; extract more XML fulltext URLs	Bryan Newbold	2020-10-30	3	-15/+23
\|
*	html: work around firstmonday DOCTYPE issue	Bryan Newbold	2020-10-30	3	-0/+458
\|
*	tests: fix conditional on poppler version check	Bryan Newbold	2020-10-30	1	-1/+1
\|
*	cdx datetime parsing improvements	Bryan Newbold	2020-10-30	1	-0/+11
\|
*	cdx: add support for 'closest' time parameter	Bryan Newbold	2020-10-30	1	-3/+9
\|
*	html: more ingest improvements	Bryan Newbold	2020-10-30	2	-18/+120
\|
*	html ingest: improve data flow	Bryan Newbold	2020-10-29	1	-18/+41
\|
*	better default CLI output (show usage)	Bryan Newbold	2020-10-29	6	-6/+6
\|
*	improve test running and config	Bryan Newbold	2020-10-29	3	-3/+7
\|
*	misc: type annotations, fix parse_cdx_datetime	Bryan Newbold	2020-10-29	1	-14/+18
\|
*	html: initial ingest implementation	Bryan Newbold	2020-10-29	1	-0/+193
\|
*	html: more metadata tests	Bryan Newbold	2020-10-29	2	-0/+2453
\|
*	pipenv: braveblocker, dynaconf, sentry-sdk	Bryan Newbold	2020-10-29	2	-29/+34
\|
*	html: more biblio selectors; resource extraction	Bryan Newbold	2020-10-29	1	-0/+102
\|
*	HTML meta: more from online hunting/research	Bryan Newbold	2020-10-27	1	-3/+54
\|
*	HTML metadata: fix type warnings	Bryan Newbold	2020-10-27	2	-2/+5
\|
*	grobid2json: fix deprecation warning	Bryan Newbold	2020-10-27	1	-1/+1
\|
*	pylint: fix error on selectolax import	Bryan Newbold	2020-10-27	1	-0/+1
\|
*	pytest: remove unknown 'ignore' from config file	Bryan Newbold	2020-10-27	1	-2/+0
\|
*	new dependencies for HTML metadata parsing	Bryan Newbold	2020-10-27	2	-926/+184
\|
*	start HTML metadata extraction code	Bryan Newbold	2020-10-27	6	-0/+2858
\|
*	ingest: skip JSTOR DOI prefixes	Bryan Newbold	2020-10-23	1	-0/+3
\|
*	Revert "reimplement worker timeout with multiprocessing"	Bryan Newbold	2020-10-22	1	-17/+23
\| \| \| \| \| \| \|	This reverts commit 031f51752e79dbdde47bbc95fe6b3600c9ec711a. Didn't actually work when testing; can't pickle the Kafka Producer object (and probably other objects)