| Commit message (Collapse) | Author | Age | Files | Lines | ||
|---|---|---|---|---|---|---|
| ... | ||||||
| * | | allow through unknown-scope HTML ingests, for possible SPN import | Bryan Newbold | 2021-10-01 | 1 | -11/+5 | |
| | | | ||||||
| * | | html: fix logging of broken CDX URL | Bryan Newbold | 2021-10-01 | 1 | -1/+1 | |
| | | | ||||||
| * | | ingest CDX lookup: weigh year+month of capture against in-petabox-or-not | Bryan Newbold | 2021-09-30 | 1 | -0/+1 | |
| | | | | | | | | | | | | | | | | | This is to try working around an issue where ingests fail because an SPN capture is much newer, but the old sorting preference ignored that. Note that the sorting logic is pretty busted anyways, and we should probably allow returning multiple matching files to try. | |||||
| * | | fix typo with spn_cdx_retry_sec arg | Bryan Newbold | 2021-09-30 | 1 | -1/+1 | |
| | | | ||||||
| * | | tune SPN CDX retry/wait depending on mode (priority vs daily) | Bryan Newbold | 2021-09-30 | 3 | -3/+9 | |
| | | | ||||||
| * | | refactor reingest scripts | Bryan Newbold | 2021-09-30 | 6 | -150/+90 | |
| | | | ||||||
| * | | yet another bad PDF sha1 | Bryan Newbold | 2021-09-30 | 1 | -0/+1 | |
| | | | ||||||
| * | | new 'daily' and 'priority' ingest request topics | Bryan Newbold | 2021-09-30 | 4 | -5/+17 | |
| | | | | | | | | | | | | | | | | | | | The old ingest request queue was always getting lopsided, suspect because it was scaled up (additional partitions) at some point in the past, hoping new topics will fix this. New '-priority' queue is like '-bulk', but for smaller-volume SPN-like requests. Eg, interactive mode. | |||||
| * | | kafka: delete unused work-updates topic | Bryan Newbold | 2021-09-13 | 1 | -4/+0 | |
| | | | ||||||
| * | | old HTML extractors: handle null tag | Bryan Newbold | 2021-09-08 | 1 | -8/+9 | |
| | | | ||||||
| * | | ingest: more block patterns, for huge databases | Bryan Newbold | 2021-09-08 | 1 | -1/+4 | |
| | | | ||||||
| * | | daily OA crawl improvements/notes | Bryan Newbold | 2021-09-08 | 1 | -0/+1021 | |
| | | | ||||||
| * | | yet more PDF sha1 to skip | Bryan Newbold | 2021-09-03 | 1 | -0/+5 | |
| | | | ||||||
| * | | yet more PDF URL patterns | Bryan Newbold | 2021-09-03 | 1 | -0/+48 | |
| | | | ||||||
| * | | ingest: check URL blocklist again after redirects | Bryan Newbold | 2021-09-03 | 1 | -0/+7 | |
| | | | ||||||
| * | | OAI-PMH patch and ingest improvement notes | Bryan Newbold | 2021-09-03 | 2 | -204/+1578 | |
| | | | ||||||
| * | | commit old patch crawl notes (dec 2020) | Bryan Newbold | 2021-09-03 | 1 | -0/+1 | |
| | | | ||||||
| * | | commit old arxiv ingest notes | Bryan Newbold | 2021-09-03 | 1 | -0/+12 | |
| | | | ||||||
| * | | kafka re-balancing tweaks | Bryan Newbold | 2021-09-03 | 1 | -1/+2 | |
| | | | ||||||
| * | | refactor and expand wall/block/cookie URL patterns | Bryan Newbold | 2021-09-03 | 2 | -6/+39 | |
| | | | ||||||
| * | | HTML ingest: several more PDF fulltext URL patterns | Bryan Newbold | 2021-09-03 | 1 | -0/+87 | |
| | | | ||||||
| * | | HTML ingest: skip noisy print() statement | Bryan Newbold | 2021-09-03 | 1 | -1/+1 | |
| | | | ||||||
| * | | commit old patch notes (will rework) | Bryan Newbold | 2021-09-03 | 1 | -0/+110 | |
| | | | ||||||
| * | | MAG post-crawl stats (5m+ new PDFs crawled successfully) | Bryan Newbold | 2021-09-02 | 1 | -0/+124 | |
| | | | ||||||
| * | | HTML ingest: more meta-URI prefixes | Bryan Newbold | 2021-08-24 | 1 | -2/+8 | |
| | | | ||||||
| * | | html ingest: detect some blog platforms, and allow lower wordcount threshold | Bryan Newbold | 2021-08-16 | 1 | -0/+6 | |
| | | | ||||||
| * | | html ingest: detect domain homepage (no path) as special case | Bryan Newbold | 2021-08-16 | 1 | -0/+8 | |
| | | | ||||||
| * | | html ingest: skip 'about:blank' | Bryan Newbold | 2021-08-16 | 1 | -0/+3 | |
| | | | | | | | | | | | Couldn't get adblock rule matcher to match this, for some reason. maybe a special case? | |||||
| * | | MAG and OAI-PMH crawl/processing notes | Bryan Newbold | 2021-08-13 | 2 | -0/+480 | |
| | | | ||||||
| * | | 2021-07 unpaywall crawl wrap-up notes | Bryan Newbold | 2021-07-30 | 1 | -12/+108 | |
| | | | ||||||
| * | | more bad PDF hashes | Bryan Newbold | 2021-07-26 | 1 | -0/+2 | |
| | | | ||||||
| * | | ingest: fix postgrest lookup bug (double get of GROBID) | Bryan Newbold | 2021-07-26 | 1 | -1/+1 | |
| | | | ||||||
| * | | reingest: skip spn2 'unknown' errors | Bryan Newbold | 2021-07-21 | 2 | -0/+2 | |
| | | | ||||||
| * | | more blocked-cookie patterns; fix old typo | Bryan Newbold | 2021-07-14 | 1 | -2/+2 | |
| | | | ||||||
| * | | unpaywall 2021-07 crawl partial notes | Bryan Newbold | 2021-07-14 | 1 | -0/+224 | |
| | | | ||||||
| * | | CI: wget used in pig CI scripts | Bryan Newbold | 2021-07-13 | 1 | -1/+1 | |
| | | | ||||||
| * | | CI: new sbt debian repository | Bryan Newbold | 2021-07-13 | 1 | -4/+3 | |
| | | | ||||||
| * | | Revert "CI: sbt bintray is gone, but ubuntu focal version should work" | Bryan Newbold | 2021-07-13 | 1 | -0/+5 | |
| | | | | | | | | | This reverts commit 9aebb88aac2b620e15756b4e1531989d9d32cd43. | |||||
| * | | CI: sbt bintray is gone, but ubuntu focal version should work | Bryan Newbold | 2021-07-13 | 1 | -5/+0 | |
| | | | ||||||
| * | | another bad PDF sha1 | Bryan Newbold | 2021-07-13 | 1 | -0/+1 | |
| | | | ||||||
| * | | crawl: SPN2 non-200 success code path | Bryan Newbold | 2021-07-13 | 1 | -11/+25 | |
| | | | ||||||
| * | | crawl: SPN self-redirect hack | Bryan Newbold | 2021-07-13 | 1 | -0/+9 | |
| | | | ||||||
| * | | crawl: small comment updates | Bryan Newbold | 2021-07-13 | 1 | -3/+6 | |
| | | | ||||||
| * | | another lowercase DOI in an (unused?) script | Bryan Newbold | 2021-07-13 | 1 | -1/+1 | |
| | | | ||||||
| * | | gitignore: samples/ | Bryan Newbold | 2021-07-13 | 1 | -0/+1 | |
| | | | ||||||
| * | | add crossref postgrest fetch support to python db helpers | Bryan Newbold | 2021-06-02 | 1 | -0/+9 | |
| | | | ||||||
| * | | crossref DB proposal, and include in SQL schema | Bryan Newbold | 2021-06-02 | 2 | -0/+93 | |
| | | | ||||||
| * | | notes on large-domain ingest tweaks | Bryan Newbold | 2021-05-27 | 1 | -0/+480 | |
| | | | ||||||
| * | | 2021-04 unpaywall crawl notes | Bryan Newbold | 2021-05-27 | 1 | -0/+368 | |
| | | | ||||||
| * | | python Makefile: fix test/*.py linting with newer pylint | Bryan Newbold | 2021-05-24 | 1 | -1/+1 | |
| | | | ||||||
 |
index : sandcrawler | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |