index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
sandcrawler
Commit message (
Expand
)
Author
Age
Files
Lines
*
more bad PDF sha1; print sha1 before poppler extract
Bryan Newbold
2020-08-05
1
-0
/
+7
*
spn2: skip js behavior (experiment)
Bryan Newbold
2020-08-05
1
-0
/
+1
*
SPN2: ensure not fetching outlinks
Bryan Newbold
2020-08-05
1
-0
/
+1
*
another bad PDF sha1
Bryan Newbold
2020-08-04
1
-0
/
+1
*
another PDF sha1hex
Bryan Newbold
2020-07-27
1
-0
/
+1
*
yet another 'bad' PDF sha1hex
Bryan Newbold
2020-07-27
1
-0
/
+1
*
use new SPNv2 'skip_first_archive' param
Bryan Newbold
2020-07-22
1
-0
/
+1
*
add more slow PDF hashes
Bryan Newbold
2020-07-05
1
-0
/
+2
*
add another bad PDF sha1hex
Bryan Newbold
2020-07-02
1
-0
/
+1
*
another bad PDF SHA-1
Bryan Newbold
2020-06-30
1
-0
/
+1
*
hack to unblock thumbnail processing pipeline
Bryan Newbold
2020-06-29
1
-0
/
+16
*
customize timeout per worker; 120sec for pdf-extract
Bryan Newbold
2020-06-29
2
-2
/
+3
*
handle empty fetched blob
Bryan Newbold
2020-06-27
1
-1
/
+6
*
CDX KeyError as WaybackError from fetch worker
Bryan Newbold
2020-06-26
1
-1
/
+1
*
handle None 'metadata' field correctly
Bryan Newbold
2020-06-26
1
-1
/
+1
*
handle non-success case of parsing extract from JSON/dict
Bryan Newbold
2020-06-26
1
-1
/
+1
*
report revisit non-200 as a WaybackError
Bryan Newbold
2020-06-26
1
-7
/
+7
*
Revert "simpler handling of null PDF text pages"
Bryan Newbold
2020-06-25
1
-4
/
+11
*
simpler handling of null PDF text pages
Bryan Newbold
2020-06-25
1
-11
/
+4
*
pdfextract: attributerror with text extraction
Bryan Newbold
2020-06-25
1
-4
/
+12
*
catch UnicodeDecodeError in pdfextract
Bryan Newbold
2020-06-25
1
-1
/
+10
*
don't nest generic fetch errors under pdf_trio
Bryan Newbold
2020-06-25
1
-12
/
+6
*
pdfextract: handle too-large fulltext
Bryan Newbold
2020-06-25
1
-0
/
+17
*
another bad/non PDF test; catch correct error
Bryan Newbold
2020-06-25
1
-1
/
+1
*
pdfextract: catch poppler.LockedDocumentError
Bryan Newbold
2020-06-25
1
-1
/
+1
*
pdfextract support in ingest worker
Bryan Newbold
2020-06-25
2
-1
/
+59
*
poppler: correct RGBA buffer endian-ness
Bryan Newbold
2020-06-25
1
-1
/
+1
*
pdfextract_tool fixes from prod usage
Bryan Newbold
2020-06-25
1
-1
/
+1
*
pdfextract: fix pdf_extra key names
Bryan Newbold
2020-06-25
1
-2
/
+2
*
ensure pdf_meta isn't passed an empty dict()
Bryan Newbold
2020-06-25
1
-1
/
+4
*
changes from prod
Bryan Newbold
2020-06-25
2
-4
/
+18
*
fixes and tweaks from testing locally
Bryan Newbold
2020-06-17
5
-17
/
+132
*
tweak kafka topic names and seaweedfs layout
Bryan Newbold
2020-06-17
1
-1
/
+2
*
make process_pdf() more robust to parse errors
Bryan Newbold
2020-06-17
1
-5
/
+29
*
note about text layout with pdf extraction
Bryan Newbold
2020-06-17
1
-0
/
+8
*
lint fixes
Bryan Newbold
2020-06-17
1
-1
/
+1
*
rename pdf tools to pdfextract
Bryan Newbold
2020-06-17
1
-0
/
+0
*
partial test coverage of pdf extract worker
Bryan Newbold
2020-06-17
1
-6
/
+9
*
add new pdf workers/persisters
Bryan Newbold
2020-06-17
2
-2
/
+101
*
pdf: mypy and typo fixes
Bryan Newbold
2020-06-17
2
-15
/
+22
*
workers: refactor to pass key to process()
Bryan Newbold
2020-06-17
6
-20
/
+28
*
initial work on PDF extraction worker
Bryan Newbold
2020-06-16
2
-1
/
+158
*
refactor worker fetch code into wrapper class
Bryan Newbold
2020-06-16
3
-141
/
+111
*
rename KafkaGrobidSink -> KafkaCompressSink
Bryan Newbold
2020-06-16
2
-2
/
+2
*
handle UnboundLocalError in HTML parsing
Bryan Newbold
2020-05-19
1
-1
/
+4
*
hotfix for html meta extract codepath
Bryan Newbold
2020-05-03
1
-1
/
+1
*
ingest: handle partial citation_pdf_url tag
Bryan Newbold
2020-05-03
1
-0
/
+3
*
workers: add missing want() dataflow path
Bryan Newbold
2020-04-30
1
-0
/
+9
*
ingest: don't 'want' non-PDF ingest
Bryan Newbold
2020-04-30
1
-0
/
+5
*
timeouts: don't push through None error messages
Bryan Newbold
2020-04-29
1
-2
/
+2
[next]