Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | filter out archive.org and web.archive.org (until implemented) | Bryan Newbold | 2020-01-14 | 1 | -1/+12 |
| | |||||
* | SPNv2 doesn't support FTP; add a live test for non-revist FTP | Bryan Newbold | 2020-01-14 | 1 | -0/+10 |
| | |||||
* | basic FTP ingest support; revist record resolution | Bryan Newbold | 2020-01-14 | 2 | -35/+78 |
| | | | | | | | - supporting revisits means more wayback hits (fewer crawls) => faster - ... but this is only partial support. will also need to work through sandcrawler db schema, etc. current status should be safe to merge/use. - ftp support via treating an ftp hit as a 200 | ||||
* | better print() output | Bryan Newbold | 2020-01-10 | 2 | -4/+4 |
| | |||||
* | fix trivial typo | Bryan Newbold | 2020-01-10 | 1 | -1/+1 |
| | |||||
* | hack/workaround for protocols.io octet PDFs | Bryan Newbold | 2020-01-10 | 1 | -2/+4 |
| | |||||
* | html extract: protocols.io, fix americanarchivist | Bryan Newbold | 2020-01-10 | 1 | -1/+7 |
| | |||||
* | fix redirect replay fetch method | Bryan Newbold | 2020-01-10 | 1 | -1/+4 |
| | |||||
* | limit length of error messages | Bryan Newbold | 2020-01-10 | 1 | -4/+4 |
| | |||||
* | handle SPNv2-then-CDX lookup failures | Bryan Newbold | 2020-01-10 | 1 | -6/+23 |
| | | | | | | - use a 10 second delay if CDX result isn't immediately available. blech. - if there is a lookup failure, call it a wayback-error and move on | ||||
* | SPNv2 hack specifically for elsevier lookups | Bryan Newbold | 2020-01-10 | 1 | -0/+15 |
| | | | | | I'm not really sure why this is needed, and maybe with more careful testing it isn't. But it works! | ||||
* | add support for redirect lookups from replay | Bryan Newbold | 2020-01-10 | 1 | -9/+69 |
| | |||||
* | more general ingest teaks and affordances | Bryan Newbold | 2020-01-10 | 2 | -15/+42 |
| | |||||
* | add sleep-and-retry workaround for CDX after SPNv2 | Bryan Newbold | 2020-01-10 | 1 | -1/+9 |
| | |||||
* | more ingest HTML extraction hacks | Bryan Newbold | 2020-01-10 | 1 | -6/+46 |
| | |||||
* | more live tests (for regressions) | Bryan Newbold | 2020-01-10 | 1 | -0/+1 |
| | |||||
* | disable CDX best lookup 'collapse'; leave comment | Bryan Newbold | 2020-01-10 | 1 | -1/+3 |
| | |||||
* | hack: reverse sort of CDX exact seems broken with SPNv2 results | Bryan Newbold | 2020-01-10 | 1 | -1/+1 |
| | |||||
* | many publisher-specific ingest improvements | Bryan Newbold | 2020-01-10 | 1 | -4/+96 |
| | |||||
* | improve ingest robustness (for legacy requests) | Bryan Newbold | 2020-01-10 | 1 | -6/+12 |
| | |||||
* | support forwarding url types other than pdf_url | Bryan Newbold | 2020-01-09 | 1 | -4/+5 |
| | |||||
* | wayback: datetime mismatch as an error | Bryan Newbold | 2020-01-09 | 1 | -1/+2 |
| | |||||
* | fill in more html extraction techniques | Bryan Newbold | 2020-01-09 | 1 | -7/+6 |
| | |||||
* | refactor ingest to a loop, allowing multiple hops | Bryan Newbold | 2020-01-09 | 1 | -25/+48 |
| | |||||
* | lots of progress on wayback refactoring | Bryan Newbold | 2020-01-09 | 2 | -50/+138 |
| | | | | | | - too much to list - canonical flags to control crawling - cdx_to_dict helper | ||||
* | location comes as a string, not list | Bryan Newbold | 2020-01-09 | 1 | -1/+1 |
| | |||||
* | fix http/https issue with GlobalWayback library | Bryan Newbold | 2020-01-09 | 1 | -1/+2 |
| | |||||
* | wayback fetch via replay; confirm hashes in crawl_resource() | Bryan Newbold | 2020-01-09 | 1 | -5/+40 |
| | |||||
* | wrap up basic (locally testable) ingest refactor | Bryan Newbold | 2020-01-09 | 2 | -178/+219 |
| | |||||
* | fix grobid tests for new wayback refactors | Bryan Newbold | 2020-01-09 | 1 | -3/+3 |
| | |||||
* | more wayback and SPN tests and fixes | Bryan Newbold | 2020-01-09 | 2 | -39/+153 |
| | |||||
* | refactor CdxApiClient, add tests | Bryan Newbold | 2020-01-08 | 1 | -40/+130 |
| | | | | | | - always use auth token and get full CDX rows - simplify to "fetch" (exact url/dt match) and "lookup_best" methods - all redirect stuff will be moved to a higher level | ||||
* | refactor SavePaperNowClient and add test | Bryan Newbold | 2020-01-07 | 1 | -28/+154 |
| | | | | | | - response as a namedtuple - "remote" errors (aka, SPN API was HTTP 200 but returned error) aren't an exception | ||||
* | remove SPNv1 code paths | Bryan Newbold | 2020-01-07 | 2 | -65/+25 |
| | |||||
* | handle grobid2json errors in calling code instead | Bryan Newbold | 2020-01-02 | 1 | -1/+7 |
| | |||||
* | db: move duplicate row filtering into DB insert helpers | Bryan Newbold | 2020-01-02 | 2 | -15/+26 |
| | |||||
* | remove unused filter in grobid worker | Bryan Newbold | 2020-01-02 | 1 | -1/+0 |
| | |||||
* | fix dict typo | Bryan Newbold | 2020-01-02 | 1 | -1/+1 |
| | |||||
* | improvements to grobid persist worker | Bryan Newbold | 2020-01-02 | 1 | -13/+16 |
| | |||||
* | set mimetype when PUT to minio | Bryan Newbold | 2020-01-02 | 1 | -0/+4 |
| | |||||
* | fix DB import counting | Bryan Newbold | 2020-01-02 | 1 | -4/+5 |
| | |||||
* | fix small errors found by pylint | Bryan Newbold | 2020-01-02 | 2 | -1/+2 |
| | |||||
* | fix sandcrawler persist workers | Bryan Newbold | 2020-01-02 | 1 | -0/+1 |
| | |||||
* | filter ingest results to not have key conflicts within batch | Bryan Newbold | 2020-01-02 | 1 | -1/+16 |
| | | | | | This handles a corner case with ON CONFLICT ... DO UPDATE where you can't do multiple such updates in the same batch transaction. | ||||
* | db: fancy insert/update separation using postgres xmax | Bryan Newbold | 2020-01-02 | 2 | -24/+45 |
| | |||||
* | add PersistGrobidDiskWorker | Bryan Newbold | 2020-01-02 | 1 | -0/+33 |
| | | | | To help with making dumps directly from Kafka (eg, for partner delivery) | ||||
* | flush out minio helper, add to grobid persist | Bryan Newbold | 2020-01-02 | 2 | -22/+71 |
| | |||||
* | implement counts properly for persist workers | Bryan Newbold | 2020-01-02 | 1 | -15/+19 |
| | |||||
* | improve DB helpers | Bryan Newbold | 2020-01-02 | 1 | -26/+81 |
| | | | | | - return insert/update row counts - implement ON CONFLICT ... DO UPDATE on some tables | ||||
* | be more parsimonious with GROBID metadata | Bryan Newbold | 2020-01-02 | 1 | -2/+4 |
| | | | | | Because these are getting persisted in database (as well as kafka), don't write out empty keys. |