sandcrawler - [no description]

	Commit message (Collapse)	Author	Age	Files	Lines
...
\| *	kafka: new XML+HTML topics	Bryan Newbold	2020-11-04	1	-0/+15
\| \|
\| *	initial implementation of HTML ingest in existing worker	Bryan Newbold	2020-11-04	4	-15/+94
\| \|
\| *	small fixes from local testing for XML ingest	Bryan Newbold	2020-11-03	3	-6/+9
\| \|
\| *	xml: re-encode XML docs into UTF-8 for persisting	Bryan Newbold	2020-11-03	5	-2/+382
\| \|
\| *	XML ingest proposal	Bryan Newbold	2020-11-03	1	-0/+64
\| \|
\| *	persist: XML and HTML persist workers	Bryan Newbold	2020-11-03	2	-3/+121
\| \|
\| *	ingest: handle publishing XML docs to kafka	Bryan Newbold	2020-11-03	2	-3/+27
\| \|
\| *	refactor 'minio' to 'seaweedfs'; and BLOB env vars	Bryan Newbold	2020-11-03	5	-25/+27
\| \| \| \| \| \| \| \| \| \|	This goes along with changes to ansible deployment to use the correct key names and values.
\| *	gen_file_metadata: detect JATS XML and use application/jats+xml	Bryan Newbold	2020-11-03	1	-0/+4
\| \|
\| *	basic support for XML ingest in worker	Bryan Newbold	2020-11-03	1	-23/+40
\| \|
\| *	html: improve XML fulltext extraction for scielo	Bryan Newbold	2020-11-03	1	-4/+17
\| \|
\| *	tweak html_meta SQL schema	Bryan Newbold	2020-11-03	2	-14/+21
\| \|
\| *	html: some refactoring	Bryan Newbold	2020-11-03	3	-24/+57
\| \|
\| *	move transfer encoding helper to sandcrawler/ia.py	Bryan Newbold	2020-11-03	2	-23/+42
\| \|
\| *	ingest: cleanups, typing, start generalizing to xml and html	Bryan Newbold	2020-11-03	1	-122/+118
\| \|
\| *	ingest: tweak debug printing alignment	Bryan Newbold	2020-11-03	2	-11/+10
\| \|
\| *	pytest: ignore DW in justext	Bryan Newbold	2020-11-03	1	-0/+1
\| \|
\| *	ingest: add more IA domains	Bryan Newbold	2020-11-03	1	-0/+2
\| \|
\| *	SQL: unmatched glutton query (old)	Bryan Newbold	2020-11-03	1	-0/+19
\| \|
\| *	commit WIP HTML ingest proposal	Bryan Newbold	2020-11-03	1	-0/+97
\| \|
\| *	monitoring: past-7-days summary query	Bryan Newbold	2020-11-03	1	-0/+26
\| \|
\| *	cdx: fix 'closest' support	Bryan Newbold	2020-11-03	1	-3/+2
\| \|
\| *	html: start on SQL table	Bryan Newbold	2020-11-03	2	-0/+59
\| \|
\| *	please: better usage output	Bryan Newbold	2020-11-02	1	-1/+1
\| \|
\| *	html: syntax fixes; resolve relative URLs; extract more XML fulltext URLs	Bryan Newbold	2020-10-30	3	-15/+23
\| \|
\| *	html: work around firstmonday DOCTYPE issue	Bryan Newbold	2020-10-30	3	-0/+458
\| \|
\| *	tests: fix conditional on poppler version check	Bryan Newbold	2020-10-30	1	-1/+1
\| \|
\| *	cdx datetime parsing improvements	Bryan Newbold	2020-10-30	1	-0/+11
\| \|
\| *	cdx: add support for 'closest' time parameter	Bryan Newbold	2020-10-30	1	-3/+9
\| \|
\| *	html: more ingest improvements	Bryan Newbold	2020-10-30	2	-18/+120
\| \|
\| *	html ingest: improve data flow	Bryan Newbold	2020-10-29	1	-18/+41
\| \|
\| *	better default CLI output (show usage)	Bryan Newbold	2020-10-29	6	-6/+6
\| \|
\| *	improve test running and config	Bryan Newbold	2020-10-29	3	-3/+7
\| \|
\| *	misc: type annotations, fix parse_cdx_datetime	Bryan Newbold	2020-10-29	1	-14/+18
\| \|
\| *	html: initial ingest implementation	Bryan Newbold	2020-10-29	1	-0/+193
\| \|
\| *	html: more metadata tests	Bryan Newbold	2020-10-29	2	-0/+2453
\| \|
\| *	pipenv: braveblocker, dynaconf, sentry-sdk	Bryan Newbold	2020-10-29	2	-29/+34
\| \|
\| *	html: more biblio selectors; resource extraction	Bryan Newbold	2020-10-29	1	-0/+102
\| \|
\| *	HTML meta: more from online hunting/research	Bryan Newbold	2020-10-27	1	-3/+54
\| \|
\| *	HTML metadata: fix type warnings	Bryan Newbold	2020-10-27	2	-2/+5
\| \|
\| *	grobid2json: fix deprecation warning	Bryan Newbold	2020-10-27	1	-1/+1
\| \|
\| *	pylint: fix error on selectolax import	Bryan Newbold	2020-10-27	1	-0/+1
\| \|
\| *	pytest: remove unknown 'ignore' from config file	Bryan Newbold	2020-10-27	1	-2/+0
\| \|
\| *	new dependencies for HTML metadata parsing	Bryan Newbold	2020-10-27	2	-926/+184
\| \|
\| *	start HTML metadata extraction code	Bryan Newbold	2020-10-27	6	-0/+2858
\| \|
* \|	unpaywall oct 2020 crawl notes	Bryan Newbold	2020-11-02	1	-45/+82
\| \|
* \|	more notes on unpaywall ingest from last week	Bryan Newbold	2020-10-27	1	-0/+73
\| \|
* \|	kafka topics for fatcat -> scholar pipeline	Bryan Newbold	2020-10-27	1	-0/+19
\|/
*	ingest: skip JSTOR DOI prefixes	Bryan Newbold	2020-10-23	1	-0/+3
\|
*	Revert "reimplement worker timeout with multiprocessing"	Bryan Newbold	2020-10-22	1	-17/+23
\| \| \| \| \| \| \|	This reverts commit 031f51752e79dbdde47bbc95fe6b3600c9ec711a. Didn't actually work when testing; can't pickle the Kafka Producer object (and probably other objects)