index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
sandcrawler
Commit message (
Expand
)
Author
Age
Files
Lines
*
changes from prod
Bryan Newbold
2020-06-25
2
-4
/
+18
*
fixes and tweaks from testing locally
Bryan Newbold
2020-06-17
5
-17
/
+132
*
tweak kafka topic names and seaweedfs layout
Bryan Newbold
2020-06-17
1
-1
/
+2
*
make process_pdf() more robust to parse errors
Bryan Newbold
2020-06-17
1
-5
/
+29
*
note about text layout with pdf extraction
Bryan Newbold
2020-06-17
1
-0
/
+8
*
lint fixes
Bryan Newbold
2020-06-17
1
-1
/
+1
*
rename pdf tools to pdfextract
Bryan Newbold
2020-06-17
1
-0
/
+0
*
partial test coverage of pdf extract worker
Bryan Newbold
2020-06-17
1
-6
/
+9
*
add new pdf workers/persisters
Bryan Newbold
2020-06-17
2
-2
/
+101
*
pdf: mypy and typo fixes
Bryan Newbold
2020-06-17
2
-15
/
+22
*
workers: refactor to pass key to process()
Bryan Newbold
2020-06-17
6
-20
/
+28
*
initial work on PDF extraction worker
Bryan Newbold
2020-06-16
2
-1
/
+158
*
refactor worker fetch code into wrapper class
Bryan Newbold
2020-06-16
3
-141
/
+111
*
rename KafkaGrobidSink -> KafkaCompressSink
Bryan Newbold
2020-06-16
2
-2
/
+2
*
handle UnboundLocalError in HTML parsing
Bryan Newbold
2020-05-19
1
-1
/
+4
*
hotfix for html meta extract codepath
Bryan Newbold
2020-05-03
1
-1
/
+1
*
ingest: handle partial citation_pdf_url tag
Bryan Newbold
2020-05-03
1
-0
/
+3
*
workers: add missing want() dataflow path
Bryan Newbold
2020-04-30
1
-0
/
+9
*
ingest: don't 'want' non-PDF ingest
Bryan Newbold
2020-04-30
1
-0
/
+5
*
timeouts: don't push through None error messages
Bryan Newbold
2020-04-29
1
-2
/
+2
*
timeout message implementation for GROBID and ingest workers
Bryan Newbold
2020-04-27
2
-0
/
+18
*
worker timeout wrapper, and use for kafka
Bryan Newbold
2020-04-27
1
-2
/
+40
*
fix KeyError in HTML PDF URL extraction
Bryan Newbold
2020-04-17
1
-1
/
+1
*
persist: only GROBID updates file_meta, not file-result
Bryan Newbold
2020-04-16
1
-1
/
+1
*
batch/multiprocess for ZipfilePusher
Bryan Newbold
2020-04-16
1
-3
/
+18
*
ingest: quick hack to capture CNKI outlinks
Bryan Newbold
2020-04-13
1
-2
/
+9
*
html: attempt at CNKI href extraction
Bryan Newbold
2020-04-13
1
-0
/
+11
*
ia: set User-Agent for replay fetch from wayback
Bryan Newbold
2020-03-29
1
-0
/
+5
*
ingest: block another large domain (and DOI prefix)
Bryan Newbold
2020-03-27
1
-0
/
+2
*
ingest: better spn2 pending error code
Bryan Newbold
2020-03-27
1
-0
/
+2
*
ingest: eurosurveillance PDF parser
Bryan Newbold
2020-03-25
1
-0
/
+11
*
ia: more conservative use of clean_url()
Bryan Newbold
2020-03-24
1
-3
/
+5
*
ingest: clean_url() in more places
Bryan Newbold
2020-03-23
3
-1
/
+6
*
persist grobid: add option to skip S3 upload
Bryan Newbold
2020-03-19
1
-7
/
+10
*
ingest: log every URL (from ia code side)
Bryan Newbold
2020-03-18
1
-0
/
+1
*
implement (unused) force_get flag for SPN2
Bryan Newbold
2020-03-18
2
-4
/
+19
*
work around local redirect (resource.location)
Bryan Newbold
2020-03-17
1
-1
/
+6
*
Merge branch 'martin-abstract-class-process' into 'master'
bnewbold
2020-03-12
1
-0
/
+6
|
\
|
*
workers: add explicit process to base class
Martin Czygan
2020-03-12
1
-0
/
+6
*
|
url cleaning (canonicalization) for ingest base_url
Bryan Newbold
2020-03-10
3
-3
/
+14
|
/
*
fixes to ingest-request persist
Bryan Newbold
2020-03-05
1
-3
/
+1
*
persist: ingest_request tool (with no ingest_file_result)
Bryan Newbold
2020-03-05
2
-1
/
+30
*
ia: catch wayback ChunkedEncodingError
Bryan Newbold
2020-03-05
1
-0
/
+3
*
ingest: make content-decoding more robust
Bryan Newbold
2020-03-03
1
-1
/
+2
*
make gzip content-encoding path more robust
Bryan Newbold
2020-03-03
1
-1
/
+10
*
ingest: crude content-encoding support
Bryan Newbold
2020-03-02
1
-1
/
+19
*
ingest: add force_recrawl flag to skip historical wayback lookup
Bryan Newbold
2020-03-02
1
-3
/
+5
*
remove protocols.io octet-stream hack
Bryan Newbold
2020-03-02
1
-6
/
+2
*
more mime normalization
Bryan Newbold
2020-02-27
1
-1
/
+18
*
ingest: narrow xhtml filter
Bryan Newbold
2020-02-25
1
-1
/
+1
[next]