Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | pass through revisit_cdx | Bryan Newbold | 2020-01-15 | 1 | -0/+3 |
| | |||||
* | ingest: sketch out more of how 'existing' path would work | Bryan Newbold | 2020-01-14 | 1 | -8/+22 |
| | |||||
* | ingest: check existing GROBID; also push results to sink | Bryan Newbold | 2020-01-14 | 1 | -4/+22 |
| | |||||
* | filter out archive.org and web.archive.org (until implemented) | Bryan Newbold | 2020-01-14 | 1 | -1/+12 |
| | |||||
* | basic FTP ingest support; revist record resolution | Bryan Newbold | 2020-01-14 | 1 | -1/+1 |
| | | | | | | | - supporting revisits means more wayback hits (fewer crawls) => faster - ... but this is only partial support. will also need to work through sandcrawler db schema, etc. current status should be safe to merge/use. - ftp support via treating an ftp hit as a 200 | ||||
* | better print() output | Bryan Newbold | 2020-01-10 | 1 | -1/+1 |
| | |||||
* | fix trivial typo | Bryan Newbold | 2020-01-10 | 1 | -1/+1 |
| | |||||
* | hack/workaround for protocols.io octet PDFs | Bryan Newbold | 2020-01-10 | 1 | -2/+4 |
| | |||||
* | limit length of error messages | Bryan Newbold | 2020-01-10 | 1 | -4/+4 |
| | |||||
* | more general ingest teaks and affordances | Bryan Newbold | 2020-01-10 | 1 | -10/+24 |
| | |||||
* | improve ingest robustness (for legacy requests) | Bryan Newbold | 2020-01-10 | 1 | -6/+12 |
| | |||||
* | support forwarding url types other than pdf_url | Bryan Newbold | 2020-01-09 | 1 | -4/+5 |
| | |||||
* | refactor ingest to a loop, allowing multiple hops | Bryan Newbold | 2020-01-09 | 1 | -25/+48 |
| | |||||
* | lots of progress on wayback refactoring | Bryan Newbold | 2020-01-09 | 1 | -11/+15 |
| | | | | | | - too much to list - canonical flags to control crawling - cdx_to_dict helper | ||||
* | wrap up basic (locally testable) ingest refactor | Bryan Newbold | 2020-01-09 | 1 | -159/+196 |
| | |||||
* | remove SPNv1 code paths | Bryan Newbold | 2020-01-07 | 1 | -30/+24 |
| | |||||
* | refactor: use print(..., file=sys.stderr) | Bryan Newbold | 2019-12-18 | 1 | -4/+4 |
| | | | | Should use logging soon, but this seems more idiomatic in the meanwhile. | ||||
* | handle wayback fetch redirect loop in ingest code | Bryan Newbold | 2019-11-14 | 1 | -2/+5 |
| | |||||
* | handle WaybackError during ingest | Bryan Newbold | 2019-11-14 | 1 | -0/+4 |
| | |||||
* | start of hrmars.com ingest support | Bryan Newbold | 2019-11-14 | 1 | -2/+5 |
| | |||||
* | treat failure to get terminal capture as a SavePageNowError | Bryan Newbold | 2019-11-13 | 1 | -1/+1 |
| | |||||
* | handle wayback client return status correctly | Bryan Newbold | 2019-11-13 | 1 | -2/+2 |
| | |||||
* | allow way more errors in SPN path | Bryan Newbold | 2019-11-13 | 1 | -2/+11 |
| | |||||
* | fix lint errors | Bryan Newbold | 2019-11-13 | 1 | -1/+1 |
| | |||||
* | improve ingest worker remote failure behavior | Bryan Newbold | 2019-11-13 | 1 | -5/+12 |
| | |||||
* | rename FileIngestWorker | Bryan Newbold | 2019-11-13 | 1 | -5/+9 |
| | |||||
* | more progress on file ingest | Bryan Newbold | 2019-11-13 | 1 | -10/+37 |
| | |||||
* | much progress on file ingest path | Bryan Newbold | 2019-10-22 | 1 | -0/+150 |