index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
refactor worker fetch code into wrapper class
Bryan Newbold
2020-06-16
3
-141
/
+111
*
rename KafkaGrobidSink -> KafkaCompressSink
Bryan Newbold
2020-06-16
3
-3
/
+3
*
remove deprecated kafka_grobid.py worker
Bryan Newbold
2020-05-26
1
-331
/
+0
*
pipenv: remove old python3.5 cruft; add mypy
Bryan Newbold
2020-05-26
2
-185
/
+196
*
start a python Makefile
Bryan Newbold
2020-05-19
1
-0
/
+15
*
handle UnboundLocalError in HTML parsing
Bryan Newbold
2020-05-19
1
-1
/
+4
*
first iteration of oai2ingestrequest script
Bryan Newbold
2020-05-05
1
-0
/
+137
*
hotfix for html meta extract codepath
Bryan Newbold
2020-05-03
1
-1
/
+1
*
ingest: handle partial citation_pdf_url tag
Bryan Newbold
2020-05-03
1
-0
/
+3
*
workers: add missing want() dataflow path
Bryan Newbold
2020-04-30
1
-0
/
+9
*
ingest: don't 'want' non-PDF ingest
Bryan Newbold
2020-04-30
1
-0
/
+5
*
timeouts: don't push through None error messages
Bryan Newbold
2020-04-29
1
-2
/
+2
*
timeout message implementation for GROBID and ingest workers
Bryan Newbold
2020-04-27
2
-0
/
+18
*
worker timeout wrapper, and use for kafka
Bryan Newbold
2020-04-27
1
-2
/
+40
*
fix KeyError in HTML PDF URL extraction
Bryan Newbold
2020-04-17
1
-1
/
+1
*
persist: only GROBID updates file_meta, not file-result
Bryan Newbold
2020-04-16
1
-1
/
+1
*
batch/multiprocess for ZipfilePusher
Bryan Newbold
2020-04-16
2
-5
/
+26
*
pipenv: update to python3.7
Bryan Newbold
2020-04-15
2
-197
/
+202
*
COVID-19 chinese paper ingest
Bryan Newbold
2020-04-15
1
-0
/
+83
*
ingest: quick hack to capture CNKI outlinks
Bryan Newbold
2020-04-13
1
-2
/
+9
*
html: attempt at CNKI href extraction
Bryan Newbold
2020-04-13
1
-0
/
+11
*
unpaywall2ingestrequest: canonicalize URL
Bryan Newbold
2020-04-07
1
-1
/
+9
*
ia: set User-Agent for replay fetch from wayback
Bryan Newbold
2020-03-29
1
-0
/
+5
*
ingest: block another large domain (and DOI prefix)
Bryan Newbold
2020-03-27
1
-0
/
+2
*
ingest: better spn2 pending error code
Bryan Newbold
2020-03-27
1
-0
/
+2
*
ingest: eurosurveillance PDF parser
Bryan Newbold
2020-03-25
1
-0
/
+11
*
ia: more conservative use of clean_url()
Bryan Newbold
2020-03-24
1
-3
/
+5
*
ingest: clean_url() in more places
Bryan Newbold
2020-03-23
3
-1
/
+6
*
skip-db option also for worker
Bryan Newbold
2020-03-19
1
-0
/
+4
*
persist grobid: add option to skip S3 upload
Bryan Newbold
2020-03-19
2
-7
/
+14
*
ingest: log every URL (from ia code side)
Bryan Newbold
2020-03-18
1
-0
/
+1
*
implement (unused) force_get flag for SPN2
Bryan Newbold
2020-03-18
2
-4
/
+19
*
work around local redirect (resource.location)
Bryan Newbold
2020-03-17
1
-1
/
+6
*
Merge branch 'martin-abstract-class-process' into 'master'
bnewbold
2020-03-12
1
-0
/
+6
|
\
|
*
workers: add explicit process to base class
Martin Czygan
2020-03-12
1
-0
/
+6
*
|
pipenv: work around zipp issue
Bryan Newbold
2020-03-10
2
-4
/
+16
*
|
pipenv: add urlcanon; update pipefile.lock
Bryan Newbold
2020-03-10
2
-209
/
+221
*
|
use local env in python scripts
Bryan Newbold
2020-03-10
3
-3
/
+3
*
|
url cleaning (canonicalization) for ingest base_url
Bryan Newbold
2020-03-10
4
-4
/
+21
*
|
ingest_file: --no-spn2 flag for single command
Bryan Newbold
2020-03-10
1
-1
/
+6
|
/
*
ingestrequest_row2json: skip on unicode errors
Bryan Newbold
2020-03-05
1
-1
/
+4
*
fixes to ingest-request persist
Bryan Newbold
2020-03-05
2
-4
/
+2
*
persist: ingest_request tool (with no ingest_file_result)
Bryan Newbold
2020-03-05
3
-1
/
+48
*
ingest_tool: force-recrawl arg
Bryan Newbold
2020-03-05
1
-0
/
+5
*
ia: catch wayback ChunkedEncodingError
Bryan Newbold
2020-03-05
1
-0
/
+3
*
ingest: make content-decoding more robust
Bryan Newbold
2020-03-03
1
-1
/
+2
*
make gzip content-encoding path more robust
Bryan Newbold
2020-03-03
1
-1
/
+10
*
ingest: crude content-encoding support
Bryan Newbold
2020-03-02
1
-1
/
+19
*
ingest: add force_recrawl flag to skip historical wayback lookup
Bryan Newbold
2020-03-02
1
-3
/
+5
*
remove protocols.io octet-stream hack
Bryan Newbold
2020-03-02
1
-6
/
+2
[prev]
[next]