| Commit message (Collapse) | Author | Age | Files | Lines | ||
|---|---|---|---|---|---|---|
| ... | ||||||
| * | | more bad PDF hashes | Bryan Newbold | 2021-07-26 | 1 | -0/+2 | |
| | | | ||||||
| * | | ingest: fix postgrest lookup bug (double get of GROBID) | Bryan Newbold | 2021-07-26 | 1 | -1/+1 | |
| | | | ||||||
| * | | reingest: skip spn2 'unknown' errors | Bryan Newbold | 2021-07-21 | 2 | -0/+2 | |
| | | | ||||||
| * | | more blocked-cookie patterns; fix old typo | Bryan Newbold | 2021-07-14 | 1 | -2/+2 | |
| | | | ||||||
| * | | unpaywall 2021-07 crawl partial notes | Bryan Newbold | 2021-07-14 | 1 | -0/+224 | |
| | | | ||||||
| * | | CI: wget used in pig CI scripts | Bryan Newbold | 2021-07-13 | 1 | -1/+1 | |
| | | | ||||||
| * | | CI: new sbt debian repository | Bryan Newbold | 2021-07-13 | 1 | -4/+3 | |
| | | | ||||||
| * | | Revert "CI: sbt bintray is gone, but ubuntu focal version should work" | Bryan Newbold | 2021-07-13 | 1 | -0/+5 | |
| | | | | | | | | | This reverts commit 9aebb88aac2b620e15756b4e1531989d9d32cd43. | |||||
| * | | CI: sbt bintray is gone, but ubuntu focal version should work | Bryan Newbold | 2021-07-13 | 1 | -5/+0 | |
| | | | ||||||
| * | | another bad PDF sha1 | Bryan Newbold | 2021-07-13 | 1 | -0/+1 | |
| | | | ||||||
| * | | crawl: SPN2 non-200 success code path | Bryan Newbold | 2021-07-13 | 1 | -11/+25 | |
| | | | ||||||
| * | | crawl: SPN self-redirect hack | Bryan Newbold | 2021-07-13 | 1 | -0/+9 | |
| | | | ||||||
| * | | crawl: small comment updates | Bryan Newbold | 2021-07-13 | 1 | -3/+6 | |
| | | | ||||||
| * | | another lowercase DOI in an (unused?) script | Bryan Newbold | 2021-07-13 | 1 | -1/+1 | |
| | | | ||||||
| * | | gitignore: samples/ | Bryan Newbold | 2021-07-13 | 1 | -0/+1 | |
| | | | ||||||
| * | | add crossref postgrest fetch support to python db helpers | Bryan Newbold | 2021-06-02 | 1 | -0/+9 | |
| | | | ||||||
| * | | crossref DB proposal, and include in SQL schema | Bryan Newbold | 2021-06-02 | 2 | -0/+93 | |
| | | | ||||||
| * | | notes on large-domain ingest tweaks | Bryan Newbold | 2021-05-27 | 1 | -0/+480 | |
| | | | ||||||
| * | | 2021-04 unpaywall crawl notes | Bryan Newbold | 2021-05-27 | 1 | -0/+368 | |
| | | | ||||||
| * | | python Makefile: fix test/*.py linting with newer pylint | Bryan Newbold | 2021-05-24 | 1 | -1/+1 | |
| | | | ||||||
| * | | ingest: fix html PDF extraction exception catch behavior | Bryan Newbold | 2021-05-24 | 1 | -3/+2 | |
| | | | ||||||
| * | | ingest PDF extraction updates | Bryan Newbold | 2021-05-21 | 3 | -2/+74 | |
| | | | ||||||
| * | | better OSF preprint download re-writing | Bryan Newbold | 2021-05-21 | 1 | -6/+23 | |
| | | | ||||||
| * | | html ingest: remove whitespace around relative URLs (eg, for d-lib) | Bryan Newbold | 2021-05-21 | 1 | -1/+1 | |
| | | | ||||||
| * | | add cdx_collection.py python script (from scratch repo) | Bryan Newbold | 2021-05-04 | 1 | -0/+80 | |
| | | | ||||||
| * | | sql: do periodically retry spn2-wayback-error | Bryan Newbold | 2021-04-27 | 2 | -2/+0 | |
| | | | ||||||
| * | | ingest: cap max body size to ~128 MByte | Bryan Newbold | 2021-04-27 | 1 | -0/+6 | |
| | | | | | | | | | Should resolve 'magic' OOM errors in production. | |||||
| * | | persist: skip very long URLs | Bryan Newbold | 2021-04-12 | 1 | -0/+4 | |
| | | | ||||||
| * | | reingest scripts to run as sandcrawler | Bryan Newbold | 2021-04-09 | 2 | -12/+12 | |
| | | | ||||||
| * | | sql: notes on sql restore | Bryan Newbold | 2021-04-09 | 1 | -0/+9 | |
| | | | ||||||
| * | | sql: update paths to work with svc506 machine | Bryan Newbold | 2021-04-09 | 12 | -49/+49 | |
| | | | ||||||
| * | | sql: before/after pg13 table size stats | Bryan Newbold | 2021-04-09 | 2 | -1/+43 | |
| | | | ||||||
| * | | sql: update periodic retry/reingest scripts | Bryan Newbold | 2021-04-09 | 4 | -6/+14 | |
| | | | ||||||
| * | | update default postgrest ('db') API endpoint | Bryan Newbold | 2021-04-09 | 1 | -1/+1 | |
| | | | ||||||
| * | | SQL snapshot doc update | Bryan Newbold | 2021-04-07 | 1 | -2/+5 | |
| | | | ||||||
| * | | gitlab CI: make python_hadoop tests manual | Bryan Newbold | 2021-04-07 | 1 | -0/+1 | |
| | | | | | | | | | | | | | These require python3.5, which isn't the system version for focal. Will need to run these jobs on machines compatible with the hadoop cluster (eg, running xenial) | |||||
| * | | gitlab-ci: gpg-agent | Bryan Newbold | 2021-04-07 | 1 | -1/+1 | |
| | | | ||||||
| * | | gitlab CI: switch to ubuntu:focal | Bryan Newbold | 2021-04-07 | 1 | -14/+10 | |
| | | | ||||||
| * | | 2021-04-07 sandcrawler DB stats | Bryan Newbold | 2021-04-07 | 1 | -0/+428 | |
| | | | ||||||
| * | | grobid: disable biblio-glutton consolidation | Bryan Newbold | 2021-04-07 | 1 | -3/+3 | |
| | | | ||||||
| * | | ingest: handle current degruyter PDF link pattern | Bryan Newbold | 2021-03-26 | 1 | -0/+8 | |
| | | | ||||||
| * | | add missing dotfiles (due to gitignore oops) | Bryan Newbold | 2021-01-18 | 2 | -0/+12 | |
| | | | ||||||
| * | | pipenv: lock minio S3 library to <7.0.0 | Bryan Newbold | 2021-01-14 | 2 | -242/+196 | |
| | | | | | | | | | | | | | | | | | | | | | | | In this upstream commit: https://github.com/minio/minio-py/commit/b81883a98e6f8a09e2903609caabbf0956dd0ec9 The API for errors changes, which makes it harder for use to catch specific exceptions (such as "NoSuchKey" as a Not Found / 404 error). Instead of refactoring, just going to pin the library. We should probably remove this library for a non-implementation-specific S3 client at some point; minio seems simpler than, eg, boto3, but there is probably something ever simpler out there. | |||||
| * | | more expansive python/.gitignore rules (all .gz) | Bryan Newbold | 2021-01-05 | 1 | -1/+1 | |
| | | | ||||||
| * | | doaj ingest request updates (from prod) | Bryan Newbold | 2021-01-05 | 1 | -1/+5 | |
| | | | ||||||
| * | | gitlab CI: update kitware (updated cmake on xenial) signing key | Bryan Newbold | 2021-01-05 | 1 | -2/+2 | |
| | | | ||||||
| * | | gitlab CI: add libmagic1 dep (was a recommends) | Bryan Newbold | 2021-01-05 | 1 | -1/+1 | |
| | | | ||||||
| * | | gitlab CI: add build deps | Bryan Newbold | 2021-01-05 | 1 | -1/+1 | |
| | | | ||||||
| * | | gitlab CI: install fewer packages (no recommends) | Bryan Newbold | 2021-01-05 | 1 | -2/+2 | |
| | | | ||||||
| * | | python makefile: don't duplicate 'lint' commands during 'test' | Bryan Newbold | 2021-01-05 | 1 | -2/+0 | |
| | | | ||||||
 |
index : sandcrawler | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |