index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
sandcrawler
/
workers.py
Commit message (
Expand
)
Author
Age
Files
Lines
*
customize timeout per worker; 120sec for pdf-extract
Bryan Newbold
2020-06-29
1
-1
/
+2
*
handle empty fetched blob
Bryan Newbold
2020-06-27
1
-1
/
+6
*
CDX KeyError as WaybackError from fetch worker
Bryan Newbold
2020-06-26
1
-1
/
+1
*
don't nest generic fetch errors under pdf_trio
Bryan Newbold
2020-06-25
1
-12
/
+6
*
fixes and tweaks from testing locally
Bryan Newbold
2020-06-17
1
-2
/
+2
*
workers: refactor to pass key to process()
Bryan Newbold
2020-06-17
1
-7
/
+15
*
refactor worker fetch code into wrapper class
Bryan Newbold
2020-06-16
1
-1
/
+88
*
rename KafkaGrobidSink -> KafkaCompressSink
Bryan Newbold
2020-06-16
1
-1
/
+1
*
workers: add missing want() dataflow path
Bryan Newbold
2020-04-30
1
-0
/
+9
*
timeouts: don't push through None error messages
Bryan Newbold
2020-04-29
1
-2
/
+2
*
worker timeout wrapper, and use for kafka
Bryan Newbold
2020-04-27
1
-2
/
+40
*
batch/multiprocess for ZipfilePusher
Bryan Newbold
2020-04-16
1
-3
/
+18
*
workers: add explicit process to base class
Martin Czygan
2020-03-12
1
-0
/
+6
*
improvements to reliability from prod testing
Bryan Newbold
2020-02-03
1
-2
/
+9
*
hack-y backoff ingest attempt
Bryan Newbold
2020-02-03
1
-1
/
+15
*
worker kafka setting tweaks
Bryan Newbold
2020-01-28
1
-2
/
+4
*
workers: yes, poll is necessary
Bryan Newbold
2020-01-28
1
-1
/
+1
*
fix kafka worker partition-specific error
Bryan Newbold
2020-01-28
1
-1
/
+1
*
have JsonLinePusher continue on JSON decode errors (but count)
Bryan Newbold
2020-01-02
1
-1
/
+5
*
refactor: use print(..., file=sys.stderr)
Bryan Newbold
2019-12-18
1
-20
/
+22
*
CI: make some jobs manual
Bryan Newbold
2019-11-15
1
-0
/
+2
*
bump kafka max poll interval for consumers
Bryan Newbold
2019-11-14
1
-2
/
+2
*
update ingest-file batch size to 1
Bryan Newbold
2019-11-14
1
-3
/
+3
*
refactor consume_topic name out of make_kafka_consumer()
Bryan Newbold
2019-11-13
1
-5
/
+5
*
workers: better generic batch-size arg handling
Bryan Newbold
2019-10-03
1
-0
/
+6
*
more counts and bugfixes in grobid_tool
Bryan Newbold
2019-09-26
1
-0
/
+6
*
off-by-one error in batch sizes
Bryan Newbold
2019-09-26
1
-1
/
+1
*
lots of grobid tool implementation (still WIP)
Bryan Newbold
2019-09-26
1
-0
/
+419