aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* update grobid2json to include given_name/surnameBryan Newbold2019-05-132-6/+10
|
* deliver_dumpgrobid_to_s3: allow heritrix-style SHA-1 formatBryan Newbold2019-05-101-0/+12
|
* clearer CDX munge notesBryan Newbold2019-05-091-1/+1
|
* deliver_dumpgrobid_to_s3: storage class configBryan Newbold2019-05-091-1/+7
|
* deliver_dumpgrobid_to_s3.pyBryan Newbold2019-04-151-0/+106
|
* schema notes on deeper file metadataBryan Newbold2019-04-121-0/+8
|
* update TODOBryan Newbold2019-04-121-1/+22
|
* scalding dump-grobid-status-code jobBryan Newbold2019-04-122-0/+58
|
* add ojs and dspace as in-domain patterns to look for in heuristic CDX PDF filterBryan Newbold2019-04-121-1/+1
|
* more fatcat update topicsBryan Newbold2019-04-121-0/+3
|
* set long timeout on HBaseStatusCountJobBryan Newbold2019-02-261-1/+3
|
* python test fixesBryan Newbold2019-02-214-5/+8
|
* backport GWB fetch improvements to extraction/kafka workersBryan Newbold2019-02-213-18/+50
| | | | *Really* need to refactor out these common methods into a base class.
* don't print secret, and MRO pylint skipBryan Newbold2019-02-211-4/+6
|
* update PipefileBryan Newbold2019-02-212-266/+220
|
* include file size in S3 uploadsBryan Newbold2019-02-201-3/+3
|
* delivery gwb counter tweaksBryan Newbold2019-02-201-2/+8
|
* silly typoBryan Newbold2019-02-191-1/+1
|
* fix empty blob errorsBryan Newbold2019-02-191-1/+5
|
* make PETABOX_WEBDATA_SECRET explicitBryan Newbold2019-02-191-1/+9
| | | | | TODO: port this change to other workers; or better yet make GWB access a mixin or something
* deliver python tweaksBryan Newbold2019-02-191-5/+8
|
* add GWB-to-S3 delivery pipeline scriptBryan Newbold2019-02-192-0/+162
|
* give sort way more RAM by defaultBryan Newbold2019-02-013-6/+6
|
* update (internal) journal-infra linkBryan Newbold2019-01-031-1/+1
|
* match_filter_enrich notesBryan Newbold2019-01-031-0/+12
|
* remove old/redundant python CDX directoryBryan Newbold2019-01-033-103/+0
| | | | This was code from Vinay; it lives on in git history.
* use ipv4 localhost with kafkacatBryan Newbold2018-12-191-1/+1
|
* notes on file-level metadata dumpBryan Newbold2018-12-191-0/+31
|
* longer match-crossref timeoutBryan Newbold2018-12-181-2/+3
|
* update notesBryan Newbold2018-12-103-1/+59
|
* crank hbase GROBID worker memory usage downBryan Newbold2018-12-101-1/+1
|
* increase message size (kafka-grobid-hbase)Bryan Newbold2018-12-101-0/+2
|
* add python-snappy depBryan Newbold2018-12-102-84/+96
|
* ah, right, it's more like extract/3sec, not 30secBryan Newbold2018-12-031-4/+4
|
* tweak grobid worker producer settingsBryan Newbold2018-12-031-2/+2
| | | | | Python CPU utilization shot way up; this is an attempt to bring it back down.
* tweak kafka config significantlyBryan Newbold2018-12-032-3/+18
|
* more sentry tags when extractingBryan Newbold2018-12-031-1/+6
|
* improvements to Kafka GROBID worker loggingBryan Newbold2018-12-032-11/+22
|
* work around kafka topic/group mistakesBryan Newbold2018-12-011-1/+1
|
* fix error var typoBryan Newbold2018-11-271-1/+1
|
* catch more wayback error typesBryan Newbold2018-11-261-1/+11
|
* only pylint fail on errorsBryan Newbold2018-11-251-1/+1
|
* fix ungrobid extraction testsBryan Newbold2018-11-221-2/+4
|
* more kafka/grobid notesBryan Newbold2018-11-211-0/+12
|
* better default consumergroup nameBryan Newbold2018-11-211-1/+1
|
* many improvements to kafka HBase inserterBryan Newbold2018-11-211-29/+29
|
* cherry-pick: correct HBase column filteringBryan Newbold2018-11-211-1/+1
|
* fixes to hbase workerBryan Newbold2018-11-211-1/+13
|
* fix kafka grobid command line topic parsingBryan Newbold2018-11-212-3/+9
|
* kafka_grobid_hbase (not 'ed')Bryan Newbold2018-11-211-0/+0
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-06 21:36:21 +0000