index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
|
more pdfextract skip sha1hex
Bryan Newbold
2020-08-06
1
-9
/
+12
*
|
grobid+pdftext missing catch-up commands
Bryan Newbold
2020-08-05
5
-10
/
+150
*
|
commit stats from a couple weeks back
Bryan Newbold
2020-08-05
1
-0
/
+347
*
|
sql stats commands updates
Bryan Newbold
2020-08-05
1
-2
/
+2
*
|
MAG ingest follow-up notes
Bryan Newbold
2020-08-05
1
-0
/
+194
*
|
more bad PDF sha1; print sha1 before poppler extract
Bryan Newbold
2020-08-05
1
-0
/
+7
*
|
spn2: skip js behavior (experiment)
Bryan Newbold
2020-08-05
1
-0
/
+1
*
|
SPN2: ensure not fetching outlinks
Bryan Newbold
2020-08-05
1
-0
/
+1
*
|
another bad PDF sha1
Bryan Newbold
2020-08-04
1
-0
/
+1
*
|
another PDF sha1hex
Bryan Newbold
2020-07-27
1
-0
/
+1
*
|
yet another 'bad' PDF sha1hex
Bryan Newbold
2020-07-27
1
-0
/
+1
*
|
use new SPNv2 'skip_first_archive' param
Bryan Newbold
2020-07-22
1
-0
/
+1
*
|
MAG 2020-07 ingest notes
Bryan Newbold
2020-07-08
1
-0
/
+159
*
|
add more slow PDF hashes
Bryan Newbold
2020-07-05
1
-0
/
+2
*
|
add another bad PDF sha1hex
Bryan Newbold
2020-07-02
1
-0
/
+1
*
|
seaweedfs proposal: fix typos and wording
Martin Czygan
2020-07-01
1
-9
/
+11
*
|
another bad PDF SHA-1
Bryan Newbold
2020-06-30
1
-0
/
+1
*
|
hack to unblock thumbnail processing pipeline
Bryan Newbold
2020-06-29
1
-0
/
+16
*
|
customize timeout per worker; 120sec for pdf-extract
Bryan Newbold
2020-06-29
3
-2
/
+4
*
|
handle empty fetched blob
Bryan Newbold
2020-06-27
1
-1
/
+6
*
|
CDX KeyError as WaybackError from fetch worker
Bryan Newbold
2020-06-26
1
-1
/
+1
*
|
handle None 'metadata' field correctly
Bryan Newbold
2020-06-26
1
-1
/
+1
*
|
handle non-success case of parsing extract from JSON/dict
Bryan Newbold
2020-06-26
1
-1
/
+1
*
|
report revisit non-200 as a WaybackError
Bryan Newbold
2020-06-26
1
-7
/
+7
*
|
Revert "simpler handling of null PDF text pages"
Bryan Newbold
2020-06-25
1
-4
/
+11
*
|
simpler handling of null PDF text pages
Bryan Newbold
2020-06-25
1
-11
/
+4
*
|
pdfextract: attributerror with text extraction
Bryan Newbold
2020-06-25
1
-4
/
+12
*
|
catch UnicodeDecodeError in pdfextract
Bryan Newbold
2020-06-25
1
-1
/
+10
*
|
don't nest generic fetch errors under pdf_trio
Bryan Newbold
2020-06-25
1
-12
/
+6
*
|
pdfextract: handle too-large fulltext
Bryan Newbold
2020-06-25
1
-0
/
+17
*
|
another bad/non PDF test; catch correct error
Bryan Newbold
2020-06-25
2
-1
/
+6
*
|
pdfextract: catch poppler.LockedDocumentError
Bryan Newbold
2020-06-25
1
-1
/
+1
*
|
commented special modes for dump_unextracted_pdf.sql
Bryan Newbold
2020-06-25
1
-1
/
+4
*
|
pdfextract support in ingest worker
Bryan Newbold
2020-06-25
3
-1
/
+66
*
|
poppler: correct RGBA buffer endian-ness
Bryan Newbold
2020-06-25
2
-2
/
+2
*
|
2020-05_pubmed ingest notes (short)
Bryan Newbold
2020-06-25
1
-0
/
+10
*
|
commit old notes on a one-off CDX table cleanup
Bryan Newbold
2020-06-25
1
-0
/
+34
*
|
commit old (2020-02) pdftrio commands
Bryan Newbold
2020-06-25
1
-0
/
+162
*
|
pdftrio SQL queries
Bryan Newbold
2020-06-25
1
-0
/
+65
*
|
SQL commands for re-trying PDF ingests
Bryan Newbold
2020-06-25
1
-0
/
+158
*
|
start of RUNBOOK commands
Bryan Newbold
2020-06-25
1
-0
/
+44
*
|
unextracted PDF job dump command
Bryan Newbold
2020-06-25
1
-0
/
+16
*
|
pdfextract_tool fixes from prod usage
Bryan Newbold
2020-06-25
2
-3
/
+6
*
|
fix tests for page0_height/width
Bryan Newbold
2020-06-25
1
-2
/
+2
*
|
pdfextract: fix pdf_extra key names
Bryan Newbold
2020-06-25
1
-2
/
+2
*
|
ensure pdf_meta isn't passed an empty dict()
Bryan Newbold
2020-06-25
1
-1
/
+4
*
|
args.kafka_env refactor didn't happen (yet)
Bryan Newbold
2020-06-25
1
-2
/
+2
*
|
s3-only mode persist workers use different consumer group
Bryan Newbold
2020-06-25
1
-2
/
+8
*
|
changes from prod
Bryan Newbold
2020-06-25
2
-4
/
+18
*
|
PDF extraction kafka topics
Bryan Newbold
2020-06-25
1
-0
/
+24
[prev]
[next]