Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | ingest: experimentally reduce CDX API retry delay | Bryan Newbold | 2020-10-17 | 1 | -1/+1 |
| | | | | | | | This code path is only working about 1/7 times in production. Going to try with a much shorter retry delay and see if we get no success with that. Considering also just disabling this attempt all together and relying on retries after hours/days. | ||||
* | notes on 2020-09 re-ingest passes | Bryan Newbold | 2020-10-17 | 1 | -0/+197 |
| | |||||
* | OA DOIs: partial notes | Bryan Newbold | 2020-10-17 | 1 | -0/+218 |
| | |||||
* | notes/status on daily ingest | Bryan Newbold | 2020-10-17 | 1 | -0/+193 |
| | |||||
* | update SQL ingest monitoring commands to be past-month by default | Bryan Newbold | 2020-10-17 | 1 | -5/+5 |
| | |||||
* | ingest: handle cookieAbsent and partial SPNv2 URL reponse cases better | Bryan Newbold | 2020-10-17 | 1 | -0/+31 |
| | |||||
* | and another sha1 | Bryan Newbold | 2020-10-13 | 1 | -0/+1 |
| | |||||
* | another day, another bad PDF sha1 | Bryan Newbold | 2020-10-13 | 1 | -0/+1 |
| | |||||
* | store no-capture URLs in terminal_url | Bryan Newbold | 2020-10-12 | 3 | -3/+39 |
| | |||||
* | start 2020-10 ingest notes | Bryan Newbold | 2020-10-11 | 1 | -0/+42 |
| | |||||
* | update unpaywall 2020-04 notes | Bryan Newbold | 2020-10-11 | 1 | -0/+32 |
| | |||||
* | OAI-PMH ingest progress timestamps | Bryan Newbold | 2020-10-11 | 1 | -0/+13 |
| | |||||
* | another bad PDF sha1 | Bryan Newbold | 2020-10-11 | 1 | -0/+1 |
| | |||||
* | yet more bad sha1 PDFs to skip | Bryan Newbold | 2020-10-10 | 1 | -0/+20 |
| | |||||
* | notes on file_meta task (from august) | Bryan Newbold | 2020-10-01 | 1 | -0/+66 |
| | |||||
* | dump_file_meta helper | Bryan Newbold | 2020-10-01 | 1 | -0/+12 |
| | |||||
* | update README (public) | Bryan Newbold | 2020-10-01 | 1 | -17/+27 |
| | |||||
* | ingest: small bugfix to print pdfextract status on SUCCESS | Bryan Newbold | 2020-09-17 | 1 | -1/+1 |
| | |||||
* | more bad PDF sha1 | Bryan Newbold | 2020-09-17 | 1 | -0/+2 |
| | |||||
* | yet another broken PDF (sha1) | Bryan Newbold | 2020-09-16 | 1 | -0/+1 |
| | |||||
* | html: handle JMIR URL pattern | Bryan Newbold | 2020-09-15 | 1 | -0/+6 |
| | |||||
* | updated sandcrawler-db stats | Bryan Newbold | 2020-09-15 | 2 | -6/+346 |
| | |||||
* | skip citation_pdf_url if it is a link loop | Bryan Newbold | 2020-09-14 | 1 | -2/+8 |
| | | | | This may help get around link-loop errors for a specific version of OJS | ||||
* | html parse: add another generic fulltext pattern | Bryan Newbold | 2020-09-14 | 1 | -1/+10 |
| | |||||
* | ingest: treat text/xml as XHTML in pdf ingest | Bryan Newbold | 2020-09-14 | 1 | -1/+1 |
| | |||||
* | OAI-PMH ingest notes | Bryan Newbold | 2020-09-03 | 1 | -0/+232 |
| | |||||
* | daily ingest notes | Bryan Newbold | 2020-09-02 | 1 | -0/+202 |
| | |||||
* | follow-up notes on processing 'holes' | Bryan Newbold | 2020-09-02 | 1 | -0/+19 |
| | |||||
* | unpaywall ingest follow-up | Bryan Newbold | 2020-09-02 | 1 | -0/+115 |
| | |||||
* | more bad SHA1 PDF | Bryan Newbold | 2020-09-02 | 1 | -0/+2 |
| | |||||
* | another bad PDF sha1 | Bryan Newbold | 2020-09-01 | 1 | -0/+1 |
| | |||||
* | another bad PDF sha1 | Bryan Newbold | 2020-08-24 | 1 | -0/+1 |
| | |||||
* | html: handle embed with mangled 'src' attribute | Bryan Newbold | 2020-08-24 | 1 | -1/+1 |
| | |||||
* | WIP weekly re-ingest script | Bryan Newbold | 2020-08-17 | 2 | -0/+97 |
| | |||||
* | another bad PDF sha1 | Bryan Newbold | 2020-08-17 | 1 | -0/+1 |
| | |||||
* | another bad PDF sha1 | Bryan Newbold | 2020-08-15 | 1 | -0/+1 |
| | |||||
* | more bad sha1 | Bryan Newbold | 2020-08-14 | 1 | -0/+1 |
| | |||||
* | yet more bad PDF sha1 | Bryan Newbold | 2020-08-14 | 1 | -0/+2 |
| | |||||
* | more bad SHA1 | Bryan Newbold | 2020-08-13 | 1 | -0/+2 |
| | |||||
* | yet another PDF sha1 | Bryan Newbold | 2020-08-12 | 1 | -0/+1 |
| | |||||
* | another bad sha1; maybe the last for this batch? | Bryan Newbold | 2020-08-12 | 1 | -0/+1 |
| | |||||
* | more bad sha1 | Bryan Newbold | 2020-08-11 | 1 | -0/+2 |
| | |||||
* | additional loginwall patterns | Bryan Newbold | 2020-08-11 | 1 | -0/+2 |
| | |||||
* | more SHA1 | Bryan Newbold | 2020-08-11 | 1 | -0/+2 |
| | |||||
* | Revert "ingest: reduce CDX retry_sleep to 3.0 sec (after SPN)" | Bryan Newbold | 2020-08-11 | 1 | -1/+1 |
| | | | | | | | This reverts commit 92bf9bc28ac0eacab2e06fa3b25b52f0882804c2. In practice, in prod, this resulted in much larger spn2-cdx-lookup-failure error rates. | ||||
* | ingest: reduce CDX retry_sleep to 3.0 sec (after SPN) | Bryan Newbold | 2020-08-11 | 1 | -1/+1 |
| | | | | | | | | As we are moving towards just retrying entire ingest requests, we should probably just make this zero. But until then we should give SPN CDX a small chance to sync before giving up. This change expected to improve overall throughput. | ||||
* | ingest: actually use force_get flag with SPN | Bryan Newbold | 2020-08-11 | 1 | -0/+13 |
| | | | | | | The code path was there, but wasn't actually flagging in our most popular daily domains yet. Hopefully will make a big difference in SPN throughput. | ||||
* | check for simple URL patterns that are usually paywalls or loginwalls | Bryan Newbold | 2020-08-11 | 2 | -0/+29 |
| | |||||
* | ingest: check for URL blocklist and cookie URL patterns on every hop | Bryan Newbold | 2020-08-11 | 1 | -0/+13 |
| | |||||
* | refactor: force_get -> force_simple_get | Bryan Newbold | 2020-08-11 | 2 | -8/+8 |
| | | | | | For clarity. The SPNv2 API hasn't changed, just changing the variable/parameter name. |