sandcrawler - [no description]

	Commit message (Collapse)	Author	Age	Files	Lines
*	block springer page-one domain	Bryan Newbold	2020-01-28	1	-0/+3
\|
*	re-enable figshare and zenodo crawling	Bryan Newbold	2020-01-21	1	-8/+0
\| \| \| \|	For daily imports
*	ingest: check for null-body before file_meta	Bryan Newbold	2020-01-21	1	-0/+3
\| \| \| \| \|	gen_file_metadata raises an assert error if body is None (or false-y in general)
*	add SKIP log line for skip-url-blocklist path	Bryan Newbold	2020-01-17	1	-0/+1
\|
*	ingest: add URL blocklist feature	Bryan Newbold	2020-01-17	1	-4/+32
\| \| \| \|	And, temporarily, block zenodo and figshare.
*	clarify ingest result schema and semantics	Bryan Newbold	2020-01-15	1	-4/+11
\|
*	pass through revisit_cdx	Bryan Newbold	2020-01-15	1	-0/+3
\|
*	ingest: sketch out more of how 'existing' path would work	Bryan Newbold	2020-01-14	1	-8/+22
\|
*	ingest: check existing GROBID; also push results to sink	Bryan Newbold	2020-01-14	1	-4/+22
\|
*	filter out archive.org and web.archive.org (until implemented)	Bryan Newbold	2020-01-14	1	-1/+12
\|
*	basic FTP ingest support; revist record resolution	Bryan Newbold	2020-01-14	1	-1/+1
\| \| \| \| \| \| \|	- supporting revisits means more wayback hits (fewer crawls) => faster - ... but this is only partial support. will also need to work through sandcrawler db schema, etc. current status should be safe to merge/use. - ftp support via treating an ftp hit as a 200
*	better print() output	Bryan Newbold	2020-01-10	1	-1/+1
\|
*	fix trivial typo	Bryan Newbold	2020-01-10	1	-1/+1
\|
*	hack/workaround for protocols.io octet PDFs	Bryan Newbold	2020-01-10	1	-2/+4
\|
*	limit length of error messages	Bryan Newbold	2020-01-10	1	-4/+4
\|
*	more general ingest teaks and affordances	Bryan Newbold	2020-01-10	1	-10/+24
\|
*	improve ingest robustness (for legacy requests)	Bryan Newbold	2020-01-10	1	-6/+12
\|
*	support forwarding url types other than pdf_url	Bryan Newbold	2020-01-09	1	-4/+5
\|
*	refactor ingest to a loop, allowing multiple hops	Bryan Newbold	2020-01-09	1	-25/+48
\|
*	lots of progress on wayback refactoring	Bryan Newbold	2020-01-09	1	-11/+15
\| \| \| \| \| \|	- too much to list - canonical flags to control crawling - cdx_to_dict helper
*	wrap up basic (locally testable) ingest refactor	Bryan Newbold	2020-01-09	1	-159/+196
\|
*	remove SPNv1 code paths	Bryan Newbold	2020-01-07	1	-30/+24
\|
*	refactor: use print(..., file=sys.stderr)	Bryan Newbold	2019-12-18	1	-4/+4
\| \| \| \|	Should use logging soon, but this seems more idiomatic in the meanwhile.
*	handle wayback fetch redirect loop in ingest code	Bryan Newbold	2019-11-14	1	-2/+5
\|
*	handle WaybackError during ingest	Bryan Newbold	2019-11-14	1	-0/+4
\|
*	start of hrmars.com ingest support	Bryan Newbold	2019-11-14	1	-2/+5
\|
*	treat failure to get terminal capture as a SavePageNowError	Bryan Newbold	2019-11-13	1	-1/+1
\|
*	handle wayback client return status correctly	Bryan Newbold	2019-11-13	1	-2/+2
\|
*	allow way more errors in SPN path	Bryan Newbold	2019-11-13	1	-2/+11
\|
*	fix lint errors	Bryan Newbold	2019-11-13	1	-1/+1
\|
*	improve ingest worker remote failure behavior	Bryan Newbold	2019-11-13	1	-5/+12
\|
*	rename FileIngestWorker	Bryan Newbold	2019-11-13	1	-5/+9
\|
*	more progress on file ingest	Bryan Newbold	2019-11-13	1	-10/+37
\|
*	much progress on file ingest path	Bryan Newbold	2019-10-22	1	-0/+150