Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | fix trivial typo | Bryan Newbold | 2020-01-10 | 1 | -1/+1 |
| | |||||
* | hack/workaround for protocols.io octet PDFs | Bryan Newbold | 2020-01-10 | 1 | -2/+4 |
| | |||||
* | html extract: protocols.io, fix americanarchivist | Bryan Newbold | 2020-01-10 | 1 | -1/+7 |
| | |||||
* | fix redirect replay fetch method | Bryan Newbold | 2020-01-10 | 1 | -1/+4 |
| | |||||
* | limit length of error messages | Bryan Newbold | 2020-01-10 | 1 | -4/+4 |
| | |||||
* | setup notes and CI (for psycopg2 dependency) | Bryan Newbold | 2020-01-10 | 2 | -1/+6 |
| | |||||
* | handle SPNv2-then-CDX lookup failures | Bryan Newbold | 2020-01-10 | 1 | -6/+23 |
| | | | | | | - use a 10 second delay if CDX result isn't immediately available. blech. - if there is a lookup failure, call it a wayback-error and move on | ||||
* | SPNv2 hack specifically for elsevier lookups | Bryan Newbold | 2020-01-10 | 1 | -0/+15 |
| | | | | | I'm not really sure why this is needed, and maybe with more careful testing it isn't. But it works! | ||||
* | add support for redirect lookups from replay | Bryan Newbold | 2020-01-10 | 1 | -9/+69 |
| | |||||
* | more general ingest teaks and affordances | Bryan Newbold | 2020-01-10 | 2 | -15/+42 |
| | |||||
* | add sleep-and-retry workaround for CDX after SPNv2 | Bryan Newbold | 2020-01-10 | 1 | -1/+9 |
| | |||||
* | more ingest HTML extraction hacks | Bryan Newbold | 2020-01-10 | 1 | -6/+46 |
| | |||||
* | more live tests (for regressions) | Bryan Newbold | 2020-01-10 | 2 | -0/+42 |
| | |||||
* | disable CDX best lookup 'collapse'; leave comment | Bryan Newbold | 2020-01-10 | 1 | -1/+3 |
| | |||||
* | hack: reverse sort of CDX exact seems broken with SPNv2 results | Bryan Newbold | 2020-01-10 | 1 | -1/+1 |
| | |||||
* | many publisher-specific ingest improvements | Bryan Newbold | 2020-01-10 | 1 | -4/+96 |
| | |||||
* | improve ingest robustness (for legacy requests) | Bryan Newbold | 2020-01-10 | 1 | -6/+12 |
| | |||||
* | update Pipfile to be xenial-compatible | Bryan Newbold | 2020-01-09 | 2 | -16/+26 |
| | |||||
* | support forwarding url types other than pdf_url | Bryan Newbold | 2020-01-09 | 1 | -4/+5 |
| | |||||
* | wayback: datetime mismatch as an error | Bryan Newbold | 2020-01-09 | 1 | -1/+2 |
| | |||||
* | fill in more html extraction techniques | Bryan Newbold | 2020-01-09 | 1 | -7/+6 |
| | |||||
* | refactor ingest to a loop, allowing multiple hops | Bryan Newbold | 2020-01-09 | 2 | -27/+57 |
| | |||||
* | add (skipped) live tests for wayback services | Bryan Newbold | 2020-01-09 | 1 | -0/+73 |
| | |||||
* | add ingest test file | Bryan Newbold | 2020-01-09 | 1 | -0/+120 |
| | | | | Forgot to commit earlier! | ||||
* | lots of progress on wayback refactoring | Bryan Newbold | 2020-01-09 | 3 | -51/+145 |
| | | | | | | - too much to list - canonical flags to control crawling - cdx_to_dict helper | ||||
* | location comes as a string, not list | Bryan Newbold | 2020-01-09 | 2 | -5/+5 |
| | |||||
* | fix http/https issue with GlobalWayback library | Bryan Newbold | 2020-01-09 | 1 | -1/+2 |
| | |||||
* | wayback fetch via replay; confirm hashes in crawl_resource() | Bryan Newbold | 2020-01-09 | 1 | -5/+40 |
| | |||||
* | wrap up basic (locally testable) ingest refactor | Bryan Newbold | 2020-01-09 | 3 | -182/+267 |
| | |||||
* | cli: allow multiple ingest single types | Bryan Newbold | 2020-01-09 | 1 | -3/+4 |
| | |||||
* | example.env: more required vars | Bryan Newbold | 2020-01-09 | 1 | -2/+7 |
| | |||||
* | basic elife+plos extraction tests | Bryan Newbold | 2020-01-09 | 3 | -0/+4842 |
| | | | | | Ripped out some HTML, but these could have been minimized even further to keep repository from growing large. | ||||
* | fix grobid test (ISO-8859-1 encoding) | Bryan Newbold | 2020-01-09 | 1 | -6/+4 |
| | | | | Also changes for wayback refactor | ||||
* | fix grobid tests for new wayback refactors | Bryan Newbold | 2020-01-09 | 3 | -15/+17 |
| | |||||
* | more wayback and SPN tests and fixes | Bryan Newbold | 2020-01-09 | 4 | -52/+220 |
| | |||||
* | pipenv: mock testing library | Bryan Newbold | 2020-01-09 | 2 | -309/+325 |
| | |||||
* | refactor CdxApiClient, add tests | Bryan Newbold | 2020-01-08 | 2 | -40/+240 |
| | | | | | | - always use auth token and get full CDX rows - simplify to "fetch" (exact url/dt match) and "lookup_best" methods - all redirect stuff will be moved to a higher level | ||||
* | refactor SavePaperNowClient and add test | Bryan Newbold | 2020-01-07 | 2 | -28/+314 |
| | | | | | | - response as a namedtuple - "remote" errors (aka, SPN API was HTTP 200 but returned error) aren't an exception | ||||
* | remove SPNv1 code paths | Bryan Newbold | 2020-01-07 | 2 | -65/+25 |
| | |||||
* | sql: more cool random queries | Bryan Newbold | 2020-01-02 | 1 | -0/+5 |
| | |||||
* | TODO: check out python ratelimit library | Bryan Newbold | 2020-01-02 | 1 | -1/+1 |
| | |||||
* | minio version/compression notes | Bryan Newbold | 2020-01-02 | 1 | -0/+11 |
| | |||||
* | handle grobid2json errors in calling code instead | Bryan Newbold | 2020-01-02 | 1 | -1/+7 |
| | |||||
* | db: move duplicate row filtering into DB insert helpers | Bryan Newbold | 2020-01-02 | 2 | -15/+26 |
| | |||||
* | remove unused filter in grobid worker | Bryan Newbold | 2020-01-02 | 1 | -1/+0 |
| | |||||
* | fix dict typo | Bryan Newbold | 2020-01-02 | 1 | -1/+1 |
| | |||||
* | improvements to grobid persist worker | Bryan Newbold | 2020-01-02 | 1 | -13/+16 |
| | |||||
* | small (syntax?) changes to pig join script | Bryan Newbold | 2020-01-02 | 1 | -2/+2 |
| | |||||
* | update persist worker invocation to use batches | Bryan Newbold | 2020-01-02 | 1 | -15/+55 |
| | |||||
* | set mimetype when PUT to minio | Bryan Newbold | 2020-01-02 | 1 | -0/+4 |
| |