index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
|
seaweedfs proposal: fix typos and wording
Martin Czygan
2020-07-01
1
-9
/
+11
*
|
another bad PDF SHA-1
Bryan Newbold
2020-06-30
1
-0
/
+1
*
|
hack to unblock thumbnail processing pipeline
Bryan Newbold
2020-06-29
1
-0
/
+16
*
|
customize timeout per worker; 120sec for pdf-extract
Bryan Newbold
2020-06-29
3
-2
/
+4
*
|
handle empty fetched blob
Bryan Newbold
2020-06-27
1
-1
/
+6
*
|
CDX KeyError as WaybackError from fetch worker
Bryan Newbold
2020-06-26
1
-1
/
+1
*
|
handle None 'metadata' field correctly
Bryan Newbold
2020-06-26
1
-1
/
+1
*
|
handle non-success case of parsing extract from JSON/dict
Bryan Newbold
2020-06-26
1
-1
/
+1
*
|
report revisit non-200 as a WaybackError
Bryan Newbold
2020-06-26
1
-7
/
+7
*
|
Revert "simpler handling of null PDF text pages"
Bryan Newbold
2020-06-25
1
-4
/
+11
*
|
simpler handling of null PDF text pages
Bryan Newbold
2020-06-25
1
-11
/
+4
*
|
pdfextract: attributerror with text extraction
Bryan Newbold
2020-06-25
1
-4
/
+12
*
|
catch UnicodeDecodeError in pdfextract
Bryan Newbold
2020-06-25
1
-1
/
+10
*
|
don't nest generic fetch errors under pdf_trio
Bryan Newbold
2020-06-25
1
-12
/
+6
*
|
pdfextract: handle too-large fulltext
Bryan Newbold
2020-06-25
1
-0
/
+17
*
|
another bad/non PDF test; catch correct error
Bryan Newbold
2020-06-25
2
-1
/
+6
*
|
pdfextract: catch poppler.LockedDocumentError
Bryan Newbold
2020-06-25
1
-1
/
+1
*
|
commented special modes for dump_unextracted_pdf.sql
Bryan Newbold
2020-06-25
1
-1
/
+4
*
|
pdfextract support in ingest worker
Bryan Newbold
2020-06-25
3
-1
/
+66
*
|
poppler: correct RGBA buffer endian-ness
Bryan Newbold
2020-06-25
2
-2
/
+2
*
|
2020-05_pubmed ingest notes (short)
Bryan Newbold
2020-06-25
1
-0
/
+10
*
|
commit old notes on a one-off CDX table cleanup
Bryan Newbold
2020-06-25
1
-0
/
+34
*
|
commit old (2020-02) pdftrio commands
Bryan Newbold
2020-06-25
1
-0
/
+162
*
|
pdftrio SQL queries
Bryan Newbold
2020-06-25
1
-0
/
+65
*
|
SQL commands for re-trying PDF ingests
Bryan Newbold
2020-06-25
1
-0
/
+158
*
|
start of RUNBOOK commands
Bryan Newbold
2020-06-25
1
-0
/
+44
*
|
unextracted PDF job dump command
Bryan Newbold
2020-06-25
1
-0
/
+16
*
|
pdfextract_tool fixes from prod usage
Bryan Newbold
2020-06-25
2
-3
/
+6
*
|
fix tests for page0_height/width
Bryan Newbold
2020-06-25
1
-2
/
+2
*
|
pdfextract: fix pdf_extra key names
Bryan Newbold
2020-06-25
1
-2
/
+2
*
|
ensure pdf_meta isn't passed an empty dict()
Bryan Newbold
2020-06-25
1
-1
/
+4
*
|
args.kafka_env refactor didn't happen (yet)
Bryan Newbold
2020-06-25
1
-2
/
+2
*
|
s3-only mode persist workers use different consumer group
Bryan Newbold
2020-06-25
1
-2
/
+8
*
|
changes from prod
Bryan Newbold
2020-06-25
2
-4
/
+18
*
|
PDF extraction kafka topics
Bryan Newbold
2020-06-25
1
-0
/
+24
*
|
sandcrawler_worker: remove duplicate run_pdf_extract()
Bryan Newbold
2020-06-25
1
-29
/
+0
*
|
gitlab-ci: install most recent cmake via PPA
Bryan Newbold
2020-06-25
1
-3
/
+7
*
|
pdfextract worker
Bryan Newbold
2020-06-25
1
-1
/
+34
*
|
pdfextract: don't compress thumbnail output
Bryan Newbold
2020-06-25
1
-1
/
+1
*
|
pipenv: python-poppler 0.2.1
Bryan Newbold
2020-06-25
2
-49
/
+51
*
|
add xenial poppler PPA (PDF parsing)
Bryan Newbold
2020-06-18
1
-0
/
+1
*
|
fixes and tweaks from testing locally
Bryan Newbold
2020-06-17
6
-18
/
+134
*
|
fixes to pdfextract_tool
Bryan Newbold
2020-06-17
1
-12
/
+8
*
|
tweak pdf_meta SQL schema
Bryan Newbold
2020-06-17
2
-5
/
+31
*
|
add a docker-compose file for local development
Bryan Newbold
2020-06-17
2
-0
/
+50
*
|
write up blob storage (w/ seaweedfs)
Bryan Newbold
2020-06-17
2
-0
/
+95
*
|
tweak kafka topic names and seaweedfs layout
Bryan Newbold
2020-06-17
3
-13
/
+16
*
|
make process_pdf() more robust to parse errors
Bryan Newbold
2020-06-17
1
-5
/
+29
*
|
note about text layout with pdf extraction
Bryan Newbold
2020-06-17
1
-0
/
+8
*
|
lint fixes
Bryan Newbold
2020-06-17
4
-7
/
+9
[prev]
[next]