| Commit message (Collapse) | Author | Age | Files | Lines | |
|---|---|---|---|---|---|
| * | new 'daily' and 'priority' ingest request topics | Bryan Newbold | 2021-09-30 | 4 | -5/+17 |
| | | | | | | | | | | The old ingest request queue was always getting lopsided, suspect because it was scaled up (additional partitions) at some point in the past, hoping new topics will fix this. New '-priority' queue is like '-bulk', but for smaller-volume SPN-like requests. Eg, interactive mode. | ||||
| * | kafka: delete unused work-updates topic | Bryan Newbold | 2021-09-13 | 1 | -4/+0 |
| | | |||||
| * | old HTML extractors: handle null tag | Bryan Newbold | 2021-09-08 | 1 | -8/+9 |
| | | |||||
| * | ingest: more block patterns, for huge databases | Bryan Newbold | 2021-09-08 | 1 | -1/+4 |
| | | |||||
| * | daily OA crawl improvements/notes | Bryan Newbold | 2021-09-08 | 1 | -0/+1021 |
| | | |||||
| * | yet more PDF sha1 to skip | Bryan Newbold | 2021-09-03 | 1 | -0/+5 |
| | | |||||
| * | yet more PDF URL patterns | Bryan Newbold | 2021-09-03 | 1 | -0/+48 |
| | | |||||
| * | ingest: check URL blocklist again after redirects | Bryan Newbold | 2021-09-03 | 1 | -0/+7 |
| | | |||||
| * | OAI-PMH patch and ingest improvement notes | Bryan Newbold | 2021-09-03 | 2 | -204/+1578 |
| | | |||||
| * | commit old patch crawl notes (dec 2020) | Bryan Newbold | 2021-09-03 | 1 | -0/+1 |
| | | |||||
| * | commit old arxiv ingest notes | Bryan Newbold | 2021-09-03 | 1 | -0/+12 |
| | | |||||
| * | kafka re-balancing tweaks | Bryan Newbold | 2021-09-03 | 1 | -1/+2 |
| | | |||||
| * | refactor and expand wall/block/cookie URL patterns | Bryan Newbold | 2021-09-03 | 2 | -6/+39 |
| | | |||||
| * | HTML ingest: several more PDF fulltext URL patterns | Bryan Newbold | 2021-09-03 | 1 | -0/+87 |
| | | |||||
| * | HTML ingest: skip noisy print() statement | Bryan Newbold | 2021-09-03 | 1 | -1/+1 |
| | | |||||
| * | commit old patch notes (will rework) | Bryan Newbold | 2021-09-03 | 1 | -0/+110 |
| | | |||||
| * | MAG post-crawl stats (5m+ new PDFs crawled successfully) | Bryan Newbold | 2021-09-02 | 1 | -0/+124 |
| | | |||||
| * | HTML ingest: more meta-URI prefixes | Bryan Newbold | 2021-08-24 | 1 | -2/+8 |
| | | |||||
| * | html ingest: detect some blog platforms, and allow lower wordcount threshold | Bryan Newbold | 2021-08-16 | 1 | -0/+6 |
| | | |||||
| * | html ingest: detect domain homepage (no path) as special case | Bryan Newbold | 2021-08-16 | 1 | -0/+8 |
| | | |||||
| * | html ingest: skip 'about:blank' | Bryan Newbold | 2021-08-16 | 1 | -0/+3 |
| | | | | | | Couldn't get adblock rule matcher to match this, for some reason. maybe a special case? | ||||
| * | MAG and OAI-PMH crawl/processing notes | Bryan Newbold | 2021-08-13 | 2 | -0/+480 |
| | | |||||
| * | 2021-07 unpaywall crawl wrap-up notes | Bryan Newbold | 2021-07-30 | 1 | -12/+108 |
| | | |||||
| * | more bad PDF hashes | Bryan Newbold | 2021-07-26 | 1 | -0/+2 |
| | | |||||
| * | ingest: fix postgrest lookup bug (double get of GROBID) | Bryan Newbold | 2021-07-26 | 1 | -1/+1 |
| | | |||||
| * | reingest: skip spn2 'unknown' errors | Bryan Newbold | 2021-07-21 | 2 | -0/+2 |
| | | |||||
| * | more blocked-cookie patterns; fix old typo | Bryan Newbold | 2021-07-14 | 1 | -2/+2 |
| | | |||||
| * | unpaywall 2021-07 crawl partial notes | Bryan Newbold | 2021-07-14 | 1 | -0/+224 |
| | | |||||
| * | CI: wget used in pig CI scripts | Bryan Newbold | 2021-07-13 | 1 | -1/+1 |
| | | |||||
| * | CI: new sbt debian repository | Bryan Newbold | 2021-07-13 | 1 | -4/+3 |
| | | |||||
| * | Revert "CI: sbt bintray is gone, but ubuntu focal version should work" | Bryan Newbold | 2021-07-13 | 1 | -0/+5 |
| | | | | | This reverts commit 9aebb88aac2b620e15756b4e1531989d9d32cd43. | ||||
| * | CI: sbt bintray is gone, but ubuntu focal version should work | Bryan Newbold | 2021-07-13 | 1 | -5/+0 |
| | | |||||
| * | another bad PDF sha1 | Bryan Newbold | 2021-07-13 | 1 | -0/+1 |
| | | |||||
| * | crawl: SPN2 non-200 success code path | Bryan Newbold | 2021-07-13 | 1 | -11/+25 |
| | | |||||
| * | crawl: SPN self-redirect hack | Bryan Newbold | 2021-07-13 | 1 | -0/+9 |
| | | |||||
| * | crawl: small comment updates | Bryan Newbold | 2021-07-13 | 1 | -3/+6 |
| | | |||||
| * | another lowercase DOI in an (unused?) script | Bryan Newbold | 2021-07-13 | 1 | -1/+1 |
| | | |||||
| * | gitignore: samples/ | Bryan Newbold | 2021-07-13 | 1 | -0/+1 |
| | | |||||
| * | add crossref postgrest fetch support to python db helpers | Bryan Newbold | 2021-06-02 | 1 | -0/+9 |
| | | |||||
| * | crossref DB proposal, and include in SQL schema | Bryan Newbold | 2021-06-02 | 2 | -0/+93 |
| | | |||||
| * | notes on large-domain ingest tweaks | Bryan Newbold | 2021-05-27 | 1 | -0/+480 |
| | | |||||
| * | 2021-04 unpaywall crawl notes | Bryan Newbold | 2021-05-27 | 1 | -0/+368 |
| | | |||||
| * | python Makefile: fix test/*.py linting with newer pylint | Bryan Newbold | 2021-05-24 | 1 | -1/+1 |
| | | |||||
| * | ingest: fix html PDF extraction exception catch behavior | Bryan Newbold | 2021-05-24 | 1 | -3/+2 |
| | | |||||
| * | ingest PDF extraction updates | Bryan Newbold | 2021-05-21 | 3 | -2/+74 |
| | | |||||
| * | better OSF preprint download re-writing | Bryan Newbold | 2021-05-21 | 1 | -6/+23 |
| | | |||||
| * | html ingest: remove whitespace around relative URLs (eg, for d-lib) | Bryan Newbold | 2021-05-21 | 1 | -1/+1 |
| | | |||||
| * | add cdx_collection.py python script (from scratch repo) | Bryan Newbold | 2021-05-04 | 1 | -0/+80 |
| | | |||||
| * | sql: do periodically retry spn2-wayback-error | Bryan Newbold | 2021-04-27 | 2 | -2/+0 |
| | | |||||
| * | ingest: cap max body size to ~128 MByte | Bryan Newbold | 2021-04-27 | 1 | -0/+6 |
| | | | | | Should resolve 'magic' OOM errors in production. | ||||
 |
index : sandcrawler | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |