| Commit message (Expand) | Author | Age | Files | Lines |
... | |
* | small fixups to SandcrawlerPostgrestClient | Bryan Newbold | 2020-01-14 | 2 | -1/+11 |
* | filter out archive.org and web.archive.org (until implemented) | Bryan Newbold | 2020-01-14 | 1 | -1/+12 |
* | clarify pmc/pmcid pairing | Bryan Newbold | 2020-01-14 | 1 | -3/+3 |
* | SPNv2 doesn't support FTP; add a live test for non-revist FTP | Bryan Newbold | 2020-01-14 | 2 | -0/+26 |
* | more ftp status 226 support | Bryan Newbold | 2020-01-14 | 5 | -9/+23 |
* | add live tests for ftp, revisits | Bryan Newbold | 2020-01-14 | 1 | -1/+36 |
* | basic FTP ingest support; revist record resolution | Bryan Newbold | 2020-01-14 | 2 | -35/+78 |
* | arabesque2ingestrequest: ingest type flag | Bryan Newbold | 2020-01-14 | 1 | -1/+4 |
* | better print() output | Bryan Newbold | 2020-01-10 | 2 | -4/+4 |
* | fix trivial typo | Bryan Newbold | 2020-01-10 | 1 | -1/+1 |
* | hack/workaround for protocols.io octet PDFs | Bryan Newbold | 2020-01-10 | 1 | -2/+4 |
* | html extract: protocols.io, fix americanarchivist | Bryan Newbold | 2020-01-10 | 1 | -1/+7 |
* | fix redirect replay fetch method | Bryan Newbold | 2020-01-10 | 1 | -1/+4 |
* | limit length of error messages | Bryan Newbold | 2020-01-10 | 1 | -4/+4 |
* | setup notes and CI (for psycopg2 dependency) | Bryan Newbold | 2020-01-10 | 2 | -1/+6 |
* | handle SPNv2-then-CDX lookup failures | Bryan Newbold | 2020-01-10 | 1 | -6/+23 |
* | SPNv2 hack specifically for elsevier lookups | Bryan Newbold | 2020-01-10 | 1 | -0/+15 |
* | add support for redirect lookups from replay | Bryan Newbold | 2020-01-10 | 1 | -9/+69 |
* | more general ingest teaks and affordances | Bryan Newbold | 2020-01-10 | 2 | -15/+42 |
* | add sleep-and-retry workaround for CDX after SPNv2 | Bryan Newbold | 2020-01-10 | 1 | -1/+9 |
* | more ingest HTML extraction hacks | Bryan Newbold | 2020-01-10 | 1 | -6/+46 |
* | more live tests (for regressions) | Bryan Newbold | 2020-01-10 | 2 | -0/+42 |
* | disable CDX best lookup 'collapse'; leave comment | Bryan Newbold | 2020-01-10 | 1 | -1/+3 |
* | hack: reverse sort of CDX exact seems broken with SPNv2 results | Bryan Newbold | 2020-01-10 | 1 | -1/+1 |
* | many publisher-specific ingest improvements | Bryan Newbold | 2020-01-10 | 1 | -4/+96 |
* | improve ingest robustness (for legacy requests) | Bryan Newbold | 2020-01-10 | 1 | -6/+12 |
* | update Pipfile to be xenial-compatible | Bryan Newbold | 2020-01-09 | 2 | -16/+26 |
* | support forwarding url types other than pdf_url | Bryan Newbold | 2020-01-09 | 1 | -4/+5 |
* | wayback: datetime mismatch as an error | Bryan Newbold | 2020-01-09 | 1 | -1/+2 |
* | fill in more html extraction techniques | Bryan Newbold | 2020-01-09 | 1 | -7/+6 |
* | refactor ingest to a loop, allowing multiple hops | Bryan Newbold | 2020-01-09 | 2 | -27/+57 |
* | add (skipped) live tests for wayback services | Bryan Newbold | 2020-01-09 | 1 | -0/+73 |
* | add ingest test file | Bryan Newbold | 2020-01-09 | 1 | -0/+120 |
* | lots of progress on wayback refactoring | Bryan Newbold | 2020-01-09 | 3 | -51/+145 |
* | location comes as a string, not list | Bryan Newbold | 2020-01-09 | 2 | -5/+5 |
* | fix http/https issue with GlobalWayback library | Bryan Newbold | 2020-01-09 | 1 | -1/+2 |
* | wayback fetch via replay; confirm hashes in crawl_resource() | Bryan Newbold | 2020-01-09 | 1 | -5/+40 |
* | wrap up basic (locally testable) ingest refactor | Bryan Newbold | 2020-01-09 | 3 | -182/+267 |
* | cli: allow multiple ingest single types | Bryan Newbold | 2020-01-09 | 1 | -3/+4 |
* | example.env: more required vars | Bryan Newbold | 2020-01-09 | 1 | -2/+7 |
* | basic elife+plos extraction tests | Bryan Newbold | 2020-01-09 | 3 | -0/+4842 |
* | fix grobid test (ISO-8859-1 encoding) | Bryan Newbold | 2020-01-09 | 1 | -6/+4 |
* | fix grobid tests for new wayback refactors | Bryan Newbold | 2020-01-09 | 3 | -15/+17 |
* | more wayback and SPN tests and fixes | Bryan Newbold | 2020-01-09 | 4 | -52/+220 |
* | pipenv: mock testing library | Bryan Newbold | 2020-01-09 | 2 | -309/+325 |
* | refactor CdxApiClient, add tests | Bryan Newbold | 2020-01-08 | 2 | -40/+240 |
* | refactor SavePaperNowClient and add test | Bryan Newbold | 2020-01-07 | 2 | -28/+314 |
* | remove SPNv1 code paths | Bryan Newbold | 2020-01-07 | 2 | -65/+25 |
* | sql: more cool random queries | Bryan Newbold | 2020-01-02 | 1 | -0/+5 |
* | TODO: check out python ratelimit library | Bryan Newbold | 2020-01-02 | 1 | -1/+1 |