| Commit message (Collapse) | Author | Age | Files | Lines | |
|---|---|---|---|---|---|
| * | refactor ingest to a loop, allowing multiple hops | Bryan Newbold | 2020-01-09 | 1 | -2/+9 |
| | | |||||
| * | add (skipped) live tests for wayback services | Bryan Newbold | 2020-01-09 | 1 | -0/+73 |
| | | |||||
| * | add ingest test file | Bryan Newbold | 2020-01-09 | 1 | -0/+120 |
| | | | | | Forgot to commit earlier! | ||||
| * | lots of progress on wayback refactoring | Bryan Newbold | 2020-01-09 | 1 | -1/+7 |
| | | | | | | | - too much to list - canonical flags to control crawling - cdx_to_dict helper | ||||
| * | location comes as a string, not list | Bryan Newbold | 2020-01-09 | 1 | -4/+4 |
| | | |||||
| * | wrap up basic (locally testable) ingest refactor | Bryan Newbold | 2020-01-09 | 1 | -4/+48 |
| | | |||||
| * | basic elife+plos extraction tests | Bryan Newbold | 2020-01-09 | 3 | -0/+4842 |
| | | | | | | Ripped out some HTML, but these could have been minimized even further to keep repository from growing large. | ||||
| * | fix grobid test (ISO-8859-1 encoding) | Bryan Newbold | 2020-01-09 | 1 | -6/+4 |
| | | | | | Also changes for wayback refactor | ||||
| * | fix grobid tests for new wayback refactors | Bryan Newbold | 2020-01-09 | 2 | -12/+14 |
| | | |||||
| * | more wayback and SPN tests and fixes | Bryan Newbold | 2020-01-09 | 2 | -13/+67 |
| | | |||||
| * | refactor CdxApiClient, add tests | Bryan Newbold | 2020-01-08 | 1 | -0/+110 |
| | | | | | | | - always use auth token and get full CDX rows - simplify to "fetch" (exact url/dt match) and "lookup_best" methods - all redirect stuff will be moved to a higher level | ||||
| * | refactor SavePaperNowClient and add test | Bryan Newbold | 2020-01-07 | 1 | -0/+160 |
| | | | | | | | - response as a namedtuple - "remote" errors (aka, SPN API was HTTP 200 but returned error) aren't an exception | ||||
| * | teixml2json test update for skipping null JSON keys | Bryan Newbold | 2020-01-02 | 1 | -10/+1 |
| | | |||||
| * | grobid2json: language_code | Bryan Newbold | 2019-10-04 | 1 | -1/+2 |
| | | |||||
| * | python tests for pusher classes | Bryan Newbold | 2019-10-02 | 2 | -0/+28 |
| | | |||||
| * | add tests for affiliation extraction | Bryan Newbold | 2019-10-02 | 2 | -1/+25 |
| | | |||||
| * | lots of grobid tool implementation (still WIP) | Bryan Newbold | 2019-09-26 | 2 | -7/+29 |
| | | |||||
| * | test of GROBID client | Bryan Newbold | 2019-09-25 | 1 | -0/+53 |
| | | |||||
| * | refactor old python hadoop code into new directory | Bryan Newbold | 2019-09-25 | 4 | -591/+0 |
| | | |||||
| * | re-write parse_cdx_line for sandcrawler lib | Bryan Newbold | 2019-09-25 | 1 | -1/+31 |
| | | |||||
| * | fix test grobid2json test | Bryan Newbold | 2019-09-25 | 1 | -1/+4 |
| | | | | | For new extra fields | ||||
| * | start refactoring sandcrawler python common code | Bryan Newbold | 2019-09-23 | 2 | -0/+41 |
| | | |||||
| * | update grobid2json to include given_name/surname | Bryan Newbold | 2019-05-13 | 1 | -3/+3 |
| | | |||||
| * | python test fixes | Bryan Newbold | 2019-02-21 | 1 | -1/+1 |
| | | |||||
| * | fix ungrobid extraction tests | Bryan Newbold | 2018-11-22 | 1 | -2/+4 |
| | | |||||
| * | longtail grobid metadata parse/filter WIP | Bryan Newbold | 2018-09-22 | 1 | -0/+5 |
| | | |||||
| * | WIP: ungrobided doesn't inherit (copypasta) | Bryan Newbold | 2018-08-25 | 1 | -4/+4 |
| | | |||||
| * | ungrobided: example real output | Bryan Newbold | 2018-08-25 | 1 | -0/+20 |
| | | |||||
| * | ungrobided: add real results to tests | Bryan Newbold | 2018-08-25 | 1 | -1/+51 |
| | | |||||
| * | python extraction_ungrobided job | Bryan Newbold | 2018-08-24 | 1 | -0/+126 |
| | | |||||
| * | rename ./mapreduce to ./python | Bryan Newbold | 2018-08-24 | 8 | -0/+2632 |
 |
index : sandcrawler | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |