aboutsummaryrefslogtreecommitdiffstats
Commit message (Expand)AuthorAgeFilesLines
* WIP: refactor logging calls in ingest pipelinesbnewbold-refactor-logggingBryan Newbold2022-07-126-114/+89
* ingest: targeted 2022-04 notesBryan Newbold2022-07-071-1/+3
* stats: may 2022 ingest-by-domain statsBryan Newbold2022-07-071-0/+410
* ingest: IEEE domain is blocking usBryan Newbold2022-07-071-1/+2
* ingest: catch more ConnectionErrors (SPN, replay fetch, GROBID)Bryan Newbold2022-05-162-4/+19
* ingest: skip arxiv.org DOIs, we already direct-ingestBryan Newbold2022-05-111-0/+1
* python make fmtBryan Newbold2022-05-051-3/+1
* ingest spn2: fix testsBryan Newbold2022-05-054-6/+108
* ingest: more loginwall patternsBryan Newbold2022-05-051-0/+3
* ingest_tool: fix arg parsingBryan Newbold2022-05-031-8/+8
* finished re-GROBID-ingBryan Newbold2022-05-031-5/+7
* PDF URL lists updateBryan Newbold2022-05-032-0/+76
* some weekly crawl numbers (not very helpful)Bryan Newbold2022-05-031-0/+191
* switch default kafka-broker host from wbgrp-svc263 to wbgrp-svc350Bryan Newbold2022-05-039-14/+14
* April 2022 sandcrawler DB statsBryan Newbold2022-04-271-0/+432
* more dataset crawl notesBryan Newbold2022-04-261-0/+53
* .ua crawling follow-up statsBryan Newbold2022-04-261-2/+2
* update HBase Thrift gateway hostBryan Newbold2022-04-261-1/+1
* SPNv2: several fixes for prod throughputBryan Newbold2022-04-261-11/+34
* make fmtBryan Newbold2022-04-261-2/+5
* ingest_tool: spn-status command to check user's quotaBryan Newbold2022-04-261-0/+19
* flake8: allow 'Any' typesBryan Newbold2022-04-261-1/+2
* start notes on unpaywall and targeted/patch crawlsBryan Newbold2022-04-202-0/+277
* block isiarticles.com from future PDF crawlsBryan Newbold2022-04-201-0/+2
* pipenv: update; newer devpi hostnameBryan Newbold2022-04-062-781/+850
* ingest: drive.google.com ingest supportBryan Newbold2022-04-041-0/+8
* .ua ingest notesBryan Newbold2022-04-041-0/+29
* sql: add source/created index on ingest_request tableBryan Newbold2022-04-041-0/+1
* sql: fix reingest query missing type on LEFT JOIN; wrap in read-only transactionBryan Newbold2022-04-045-5/+27
* filesets: fix archive.org path namingBryan Newbold2022-03-291-7/+8
* bugfix: sha1/md5 typoBryan Newbold2022-03-231-1/+1
* various ingest/task notesBryan Newbold2022-03-224-5/+97
* file ingest: don't 'backoff' on spn2 backoff errorBryan Newbold2022-03-222-0/+8
* DOAJ ingest/crawl notesBryan Newbold2022-03-111-0/+266
* partial notes on .ua urgent crawlingBryan Newbold2022-03-111-0/+196
* 2022 patch crawl bulk ingest notesBryan Newbold2022-03-021-0/+106
* update old OAI-PMH patch crawl notesBryan Newbold2022-02-281-1/+36
* more sentry config changesBryan Newbold2022-02-255-5/+5
* small lint/typo/fmt fixesBryan Newbold2022-02-243-5/+5
* switch from 'raven' to 'sentry-sdk'Bryan Newbold2022-02-245-37/+41
* another bad PDF sha1Bryan Newbold2022-02-231-0/+1
* ingest: fix mistakenly commented except block (?)Bryan Newbold2022-02-181-4/+3
* ingest: handle more fileset failure modesBryan Newbold2022-02-182-3/+30
* sandcrawler_worker: add --skip-spn flagBryan Newbold2022-02-081-2/+7
* yet another bad PDF sha1Bryan Newbold2022-02-081-0/+1
* more patch crawlingBryan Newbold2022-02-082-9/+209
* OAI-PMH patch crawl more updatesBryan Newbold2022-02-081-2/+71
* sql: script to reingest recent spn2 lookup failure in bulk modeBryan Newbold2022-02-085-18/+71
* pipenv: update lock fileBryan Newbold2022-02-031-592/+614
* pipenv: black (code style tool) has a stable releaseBryan Newbold2022-02-031-4/+1
generated by cgit v1.2.3 (git 2.39.1) at 2024-04-27 09:07:45 +0000