| Commit message (Collapse) | Author | Age | Files | Lines | ||
|---|---|---|---|---|---|---|
| ... | ||||||
| * | | | deliver_dumpgrobid_to_s3: storage class config | Bryan Newbold | 2019-05-09 | 1 | -1/+7 | |
| | | | | ||||||
| * | | | deliver_dumpgrobid_to_s3.py | Bryan Newbold | 2019-04-15 | 1 | -0/+106 | |
| | | | | ||||||
| * | | | schema notes on deeper file metadata | Bryan Newbold | 2019-04-12 | 1 | -0/+8 | |
| | | | | ||||||
| * | | | update TODO | Bryan Newbold | 2019-04-12 | 1 | -1/+22 | |
| | | | | ||||||
| * | | | scalding dump-grobid-status-code job | Bryan Newbold | 2019-04-12 | 2 | -0/+58 | |
| | | | | ||||||
| * | | | add ojs and dspace as in-domain patterns to look for in heuristic CDX PDF filter | Bryan Newbold | 2019-04-12 | 1 | -1/+1 | |
| | | | | ||||||
| * | | | more fatcat update topics | Bryan Newbold | 2019-04-12 | 1 | -0/+3 | |
| | | | | ||||||
| * | | | set long timeout on HBaseStatusCountJob | Bryan Newbold | 2019-02-26 | 1 | -1/+3 | |
| |/ / | ||||||
| * | | python test fixes | Bryan Newbold | 2019-02-21 | 4 | -5/+8 | |
| | | | ||||||
| * | | backport GWB fetch improvements to extraction/kafka workers | Bryan Newbold | 2019-02-21 | 3 | -18/+50 | |
| | | | | | | | | | *Really* need to refactor out these common methods into a base class. | |||||
| * | | don't print secret, and MRO pylint skip | Bryan Newbold | 2019-02-21 | 1 | -4/+6 | |
| | | | ||||||
| * | | update Pipefile | Bryan Newbold | 2019-02-21 | 2 | -266/+220 | |
| | | | ||||||
| * | | include file size in S3 uploads | Bryan Newbold | 2019-02-20 | 1 | -3/+3 | |
| | | | ||||||
| * | | delivery gwb counter tweaks | Bryan Newbold | 2019-02-20 | 1 | -2/+8 | |
| | | | ||||||
| * | | silly typo | Bryan Newbold | 2019-02-19 | 1 | -1/+1 | |
| | | | ||||||
| * | | fix empty blob errors | Bryan Newbold | 2019-02-19 | 1 | -1/+5 | |
| | | | ||||||
| * | | make PETABOX_WEBDATA_SECRET explicit | Bryan Newbold | 2019-02-19 | 1 | -1/+9 | |
| | | | | | | | | | | | TODO: port this change to other workers; or better yet make GWB access a mixin or something | |||||
| * | | deliver python tweaks | Bryan Newbold | 2019-02-19 | 1 | -5/+8 | |
| | | | ||||||
| * | | add GWB-to-S3 delivery pipeline script | Bryan Newbold | 2019-02-19 | 2 | -0/+162 | |
| | | | ||||||
| * | | give sort way more RAM by default | Bryan Newbold | 2019-02-01 | 3 | -6/+6 | |
| | | | ||||||
| * | | update (internal) journal-infra link | Bryan Newbold | 2019-01-03 | 1 | -1/+1 | |
| | | | ||||||
| * | | match_filter_enrich notes | Bryan Newbold | 2019-01-03 | 1 | -0/+12 | |
| | | | ||||||
| * | | remove old/redundant python CDX directory | Bryan Newbold | 2019-01-03 | 3 | -103/+0 | |
| | | | | | | | | | This was code from Vinay; it lives on in git history. | |||||
| * | | use ipv4 localhost with kafkacat | Bryan Newbold | 2018-12-19 | 1 | -1/+1 | |
| | | | ||||||
| * | | notes on file-level metadata dump | Bryan Newbold | 2018-12-19 | 1 | -0/+31 | |
| | | | ||||||
| * | | longer match-crossref timeout | Bryan Newbold | 2018-12-18 | 1 | -2/+3 | |
| | | | ||||||
| * | | update notes | Bryan Newbold | 2018-12-10 | 3 | -1/+59 | |
| | | | ||||||
| * | | crank hbase GROBID worker memory usage down | Bryan Newbold | 2018-12-10 | 1 | -1/+1 | |
| | | | ||||||
| * | | increase message size (kafka-grobid-hbase) | Bryan Newbold | 2018-12-10 | 1 | -0/+2 | |
| | | | ||||||
| * | | add python-snappy dep | Bryan Newbold | 2018-12-10 | 2 | -84/+96 | |
| | | | ||||||
| * | | ah, right, it's more like extract/3sec, not 30sec | Bryan Newbold | 2018-12-03 | 1 | -4/+4 | |
| | | | ||||||
| * | | tweak grobid worker producer settings | Bryan Newbold | 2018-12-03 | 1 | -2/+2 | |
| | | | | | | | | | | | Python CPU utilization shot way up; this is an attempt to bring it back down. | |||||
| * | | tweak kafka config significantly | Bryan Newbold | 2018-12-03 | 2 | -3/+18 | |
| | | | ||||||
| * | | more sentry tags when extracting | Bryan Newbold | 2018-12-03 | 1 | -1/+6 | |
| | | | ||||||
| * | | improvements to Kafka GROBID worker logging | Bryan Newbold | 2018-12-03 | 2 | -11/+22 | |
| | | | ||||||
| * | | work around kafka topic/group mistakes | Bryan Newbold | 2018-12-01 | 1 | -1/+1 | |
| | | | ||||||
| * | | fix error var typo | Bryan Newbold | 2018-11-27 | 1 | -1/+1 | |
| | | | ||||||
| * | | catch more wayback error types | Bryan Newbold | 2018-11-26 | 1 | -1/+11 | |
| | | | ||||||
| * | | only pylint fail on errors | Bryan Newbold | 2018-11-25 | 1 | -1/+1 | |
| | | | ||||||
| * | | fix ungrobid extraction tests | Bryan Newbold | 2018-11-22 | 1 | -2/+4 | |
| | | | ||||||
| * | | more kafka/grobid notes | Bryan Newbold | 2018-11-21 | 1 | -0/+12 | |
| | | | ||||||
| * | | better default consumergroup name | Bryan Newbold | 2018-11-21 | 1 | -1/+1 | |
| | | | ||||||
| * | | many improvements to kafka HBase inserter | Bryan Newbold | 2018-11-21 | 1 | -29/+29 | |
| | | | ||||||
| * | | cherry-pick: correct HBase column filtering | Bryan Newbold | 2018-11-21 | 1 | -1/+1 | |
| | | | ||||||
| * | | fixes to hbase worker | Bryan Newbold | 2018-11-21 | 1 | -1/+13 | |
| | | | ||||||
| * | | fix kafka grobid command line topic parsing | Bryan Newbold | 2018-11-21 | 2 | -3/+9 | |
| | | | ||||||
| * | | kafka_grobid_hbase (not 'ed') | Bryan Newbold | 2018-11-21 | 1 | -0/+0 | |
| | | | ||||||
| * | | kafka_grobid fixes and hbase WIP | Bryan Newbold | 2018-11-21 | 2 | -2/+179 | |
| | | | ||||||
| * | | small kafka_grobid tweaks | Bryan Newbold | 2018-11-21 | 1 | -1/+2 | |
| | | | ||||||
| * | | updated Pipfile.lock (VERY SLOW) | Bryan Newbold | 2018-11-21 | 1 | -548/+431 | |
| | | | ||||||
 |
index : sandcrawler | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |