sandcrawler - [no description]

	Commit message (Collapse)	Author	Age	Files	Lines
*	add live tests for ftp, revisits	Bryan Newbold	2020-01-14	1	-1/+36
\|
*	more live tests (for regressions)	Bryan Newbold	2020-01-10	1	-0/+41
\|
*	refactor ingest to a loop, allowing multiple hops	Bryan Newbold	2020-01-09	1	-2/+9
\|
*	add (skipped) live tests for wayback services	Bryan Newbold	2020-01-09	1	-0/+73
\|
*	add ingest test file	Bryan Newbold	2020-01-09	1	-0/+120
\| \| \| \|	Forgot to commit earlier!
*	lots of progress on wayback refactoring	Bryan Newbold	2020-01-09	1	-1/+7
\| \| \| \| \| \|	- too much to list - canonical flags to control crawling - cdx_to_dict helper
*	location comes as a string, not list	Bryan Newbold	2020-01-09	1	-4/+4
\|
*	wrap up basic (locally testable) ingest refactor	Bryan Newbold	2020-01-09	1	-4/+48
\|
*	basic elife+plos extraction tests	Bryan Newbold	2020-01-09	3	-0/+4842
\| \| \| \| \|	Ripped out some HTML, but these could have been minimized even further to keep repository from growing large.
*	fix grobid test (ISO-8859-1 encoding)	Bryan Newbold	2020-01-09	1	-6/+4
\| \| \| \|	Also changes for wayback refactor
*	fix grobid tests for new wayback refactors	Bryan Newbold	2020-01-09	2	-12/+14
\|
*	more wayback and SPN tests and fixes	Bryan Newbold	2020-01-09	2	-13/+67
\|
*	refactor CdxApiClient, add tests	Bryan Newbold	2020-01-08	1	-0/+110
\| \| \| \| \| \|	- always use auth token and get full CDX rows - simplify to "fetch" (exact url/dt match) and "lookup_best" methods - all redirect stuff will be moved to a higher level
*	refactor SavePaperNowClient and add test	Bryan Newbold	2020-01-07	1	-0/+160
\| \| \| \| \| \|	- response as a namedtuple - "remote" errors (aka, SPN API was HTTP 200 but returned error) aren't an exception
*	teixml2json test update for skipping null JSON keys	Bryan Newbold	2020-01-02	1	-10/+1
\|
*	grobid2json: language_code	Bryan Newbold	2019-10-04	1	-1/+2
\|
*	python tests for pusher classes	Bryan Newbold	2019-10-02	2	-0/+28
\|
*	add tests for affiliation extraction	Bryan Newbold	2019-10-02	2	-1/+25
\|
*	lots of grobid tool implementation (still WIP)	Bryan Newbold	2019-09-26	2	-7/+29
\|
*	test of GROBID client	Bryan Newbold	2019-09-25	1	-0/+53
\|
*	refactor old python hadoop code into new directory	Bryan Newbold	2019-09-25	4	-591/+0
\|
*	re-write parse_cdx_line for sandcrawler lib	Bryan Newbold	2019-09-25	1	-1/+31
\|
*	fix test grobid2json test	Bryan Newbold	2019-09-25	1	-1/+4
\| \| \| \|	For new extra fields
*	start refactoring sandcrawler python common code	Bryan Newbold	2019-09-23	2	-0/+41
\|
*	update grobid2json to include given_name/surname	Bryan Newbold	2019-05-13	1	-3/+3
\|
*	python test fixes	Bryan Newbold	2019-02-21	1	-1/+1
\|
*	fix ungrobid extraction tests	Bryan Newbold	2018-11-22	1	-2/+4
\|
*	longtail grobid metadata parse/filter WIP	Bryan Newbold	2018-09-22	1	-0/+5
\|
*	WIP: ungrobided doesn't inherit (copypasta)	Bryan Newbold	2018-08-25	1	-4/+4
\|
*	ungrobided: example real output	Bryan Newbold	2018-08-25	1	-0/+20
\|
*	ungrobided: add real results to tests	Bryan Newbold	2018-08-25	1	-1/+51
\|
*	python extraction_ungrobided job	Bryan Newbold	2018-08-24	1	-0/+126
\|
*	rename ./mapreduce to ./python	Bryan Newbold	2018-08-24	8	-0/+2632