sandcrawler - [no description]

	Commit message (Collapse)	Author	Age	Files	Lines
...
\| * \|	pytest: ignore DW in justext	Bryan Newbold	2020-11-03	1	-0/+1
\| \| \|
\| * \|	ingest: add more IA domains	Bryan Newbold	2020-11-03	1	-0/+2
\| \| \|
\| * \|	SQL: unmatched glutton query (old)	Bryan Newbold	2020-11-03	1	-0/+19
\| \| \|
\| * \|	commit WIP HTML ingest proposal	Bryan Newbold	2020-11-03	1	-0/+97
\| \| \|
\| * \|	monitoring: past-7-days summary query	Bryan Newbold	2020-11-03	1	-0/+26
\| \| \|
\| * \|	cdx: fix 'closest' support	Bryan Newbold	2020-11-03	1	-3/+2
\| \| \|
\| * \|	html: start on SQL table	Bryan Newbold	2020-11-03	2	-0/+59
\| \| \|
\| * \|	please: better usage output	Bryan Newbold	2020-11-02	1	-1/+1
\| \| \|
\| * \|	html: syntax fixes; resolve relative URLs; extract more XML fulltext URLs	Bryan Newbold	2020-10-30	3	-15/+23
\| \| \|
\| * \|	html: work around firstmonday DOCTYPE issue	Bryan Newbold	2020-10-30	3	-0/+458
\| \| \|
\| * \|	tests: fix conditional on poppler version check	Bryan Newbold	2020-10-30	1	-1/+1
\| \| \|
\| * \|	cdx datetime parsing improvements	Bryan Newbold	2020-10-30	1	-0/+11
\| \| \|
\| * \|	cdx: add support for 'closest' time parameter	Bryan Newbold	2020-10-30	1	-3/+9
\| \| \|
\| * \|	html: more ingest improvements	Bryan Newbold	2020-10-30	2	-18/+120
\| \| \|
\| * \|	html ingest: improve data flow	Bryan Newbold	2020-10-29	1	-18/+41
\| \| \|
\| * \|	better default CLI output (show usage)	Bryan Newbold	2020-10-29	6	-6/+6
\| \| \|
\| * \|	improve test running and config	Bryan Newbold	2020-10-29	3	-3/+7
\| \| \|
\| * \|	misc: type annotations, fix parse_cdx_datetime	Bryan Newbold	2020-10-29	1	-14/+18
\| \| \|
\| * \|	html: initial ingest implementation	Bryan Newbold	2020-10-29	1	-0/+193
\| \| \|
\| * \|	html: more metadata tests	Bryan Newbold	2020-10-29	2	-0/+2453
\| \| \|
\| * \|	pipenv: braveblocker, dynaconf, sentry-sdk	Bryan Newbold	2020-10-29	2	-29/+34
\| \| \|
\| * \|	html: more biblio selectors; resource extraction	Bryan Newbold	2020-10-29	1	-0/+102
\| \| \|
\| * \|	HTML meta: more from online hunting/research	Bryan Newbold	2020-10-27	1	-3/+54
\| \| \|
\| * \|	HTML metadata: fix type warnings	Bryan Newbold	2020-10-27	2	-2/+5
\| \| \|
\| * \|	grobid2json: fix deprecation warning	Bryan Newbold	2020-10-27	1	-1/+1
\| \| \|
\| * \|	pylint: fix error on selectolax import	Bryan Newbold	2020-10-27	1	-0/+1
\| \| \|
\| * \|	pytest: remove unknown 'ignore' from config file	Bryan Newbold	2020-10-27	1	-2/+0
\| \| \|
\| * \|	new dependencies for HTML metadata parsing	Bryan Newbold	2020-10-27	2	-926/+184
\| \| \|
\| * \|	start HTML metadata extraction code	Bryan Newbold	2020-10-27	6	-0/+2858
\| \| \|
* \| \|	unpaywall oct 2020 crawl notes	Bryan Newbold	2020-11-02	1	-45/+82
\| \| \|
* \| \|	more notes on unpaywall ingest from last week	Bryan Newbold	2020-10-27	1	-0/+73
\| \| \|
* \| \|	kafka topics for fatcat -> scholar pipeline	Bryan Newbold	2020-10-27	1	-0/+19
\|/ /
* \|	ingest: skip JSTOR DOI prefixes	Bryan Newbold	2020-10-23	1	-0/+3
\| \|
* \|	Revert "reimplement worker timeout with multiprocessing"	Bryan Newbold	2020-10-22	1	-17/+23
\| \| \| \| \| \| \| \| \| \| \| \| \| \|	This reverts commit 031f51752e79dbdde47bbc95fe6b3600c9ec711a. Didn't actually work when testing; can't pickle the Kafka Producer object (and probably other objects)
* \|	ingest: decrease CDX timeout retries again	Bryan Newbold	2020-10-22	1	-1/+1
\| \|
* \|	make: coverage report and HTML in one go	Bryan Newbold	2020-10-22	2	-5/+2
\| \|
* \|	reimplement worker timeout with multiprocessing	Bryan Newbold	2020-10-22	1	-23/+17
\| \|
* \|	SQL: update weekly/quarterly ingest retry scripts	Bryan Newbold	2020-10-21	5	-18/+119
\| \|
* \|	sql stats: larger limits (more complete lists)	Bryan Newbold	2020-10-21	1	-8/+8
\| \|
* \|	ingest: fix WaybackContentError typo	Bryan Newbold	2020-10-21	1	-1/+1
\| \|
* \|	ingest: add a check for blocked-cookie before trying PDF url extraction	Bryan Newbold	2020-10-21	1	-0/+11
\| \|
* \|	differential wayback-error from wayback-content-error	Bryan Newbold	2020-10-21	7	-18/+22
\| \| \| \| \| \| \| \| \| \| \| \|	The motivation here is to distinguish errors due to current content in wayback (eg, in WARCs) from operational errors (eg, wayback machine is down, or network failures/disruption).
* \|	persist PDF extraction in ingest pipeline	Bryan Newbold	2020-10-20	1	-4/+16
\| \| \| \| \| \| \| \| \| \|	Ooof, didn't realize that this wasn't happening. Explains a lot of missing thumbnails in scholar!
* \|	ingest: add a cdx-error slowdown delay	Bryan Newbold	2020-10-19	1	-0/+3
\| \|
* \|	SPN CDX delay now seems reasonable; increase to 40sec to catch most	Bryan Newbold	2020-10-19	1	-1/+1
\| \|
* \|	ingest: fix old_failure datetime	Bryan Newbold	2020-10-19	1	-1/+1
\| \|
* \|	CDX: when retrying, do so every 3 seconds up to limit	Bryan Newbold	2020-10-19	1	-5/+9
\| \|
* \|	ingest: try SPNv2 for no-capture and old failures	Bryan Newbold	2020-10-19	1	-1/+5
\| \|
* \|	SPN: more verbose status logging	Bryan Newbold	2020-10-19	1	-0/+4
\| \|
* \|	ingest: disable soft404 and non-hit SPNv2 retries	Bryan Newbold	2020-10-19	1	-4/+5
\| \| \| \| \| \| \| \| \| \| \| \|	This might have made sense at some point, but I had forgotten about this code path and it makes no sense now. Has been resulting in very many extraneous SPN requests.