aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* unextracted PDF job dump commandBryan Newbold2020-06-251-0/+16
|
* pdfextract_tool fixes from prod usageBryan Newbold2020-06-252-3/+6
|
* fix tests for page0_height/widthBryan Newbold2020-06-251-2/+2
|
* pdfextract: fix pdf_extra key namesBryan Newbold2020-06-251-2/+2
|
* ensure pdf_meta isn't passed an empty dict()Bryan Newbold2020-06-251-1/+4
|
* args.kafka_env refactor didn't happen (yet)Bryan Newbold2020-06-251-2/+2
|
* s3-only mode persist workers use different consumer groupBryan Newbold2020-06-251-2/+8
|
* changes from prodBryan Newbold2020-06-252-4/+18
|
* PDF extraction kafka topicsBryan Newbold2020-06-251-0/+24
|
* sandcrawler_worker: remove duplicate run_pdf_extract()Bryan Newbold2020-06-251-29/+0
|
* gitlab-ci: install most recent cmake via PPABryan Newbold2020-06-251-3/+7
|
* pdfextract workerBryan Newbold2020-06-251-1/+34
|
* pdfextract: don't compress thumbnail outputBryan Newbold2020-06-251-1/+1
|
* pipenv: python-poppler 0.2.1Bryan Newbold2020-06-252-49/+51
|
* add xenial poppler PPA (PDF parsing)Bryan Newbold2020-06-181-0/+1
|
* fixes and tweaks from testing locallyBryan Newbold2020-06-176-18/+134
|
* fixes to pdfextract_toolBryan Newbold2020-06-171-12/+8
|
* tweak pdf_meta SQL schemaBryan Newbold2020-06-172-5/+31
|
* add a docker-compose file for local developmentBryan Newbold2020-06-172-0/+50
|
* write up blob storage (w/ seaweedfs)Bryan Newbold2020-06-172-0/+95
| | | | This follows-on to Martin's earlier proposal.
* tweak kafka topic names and seaweedfs layoutBryan Newbold2020-06-173-13/+16
|
* make process_pdf() more robust to parse errorsBryan Newbold2020-06-171-5/+29
|
* note about text layout with pdf extractionBryan Newbold2020-06-171-0/+8
|
* lint fixesBryan Newbold2020-06-174-7/+9
|
* rename pdf tools to pdfextractBryan Newbold2020-06-173-0/+0
|
* coverage-html makefile targetBryan Newbold2020-06-171-0/+3
|
* add debian dependencies for python-poppler to CI setupBryan Newbold2020-06-171-1/+1
|
* pdf thumbnail+text+meta proposalBryan Newbold2020-06-171-0/+327
|
* ignore PIL deprecation warningsBryan Newbold2020-06-171-0/+1
|
* partial test coverage of pdf extract workerBryan Newbold2020-06-172-6/+70
|
* fix coverage commandBryan Newbold2020-06-172-2/+4
|
* update grobid2json with type annotationsBryan Newbold2020-06-171-94/+110
|
* remove unused common.pyBryan Newbold2020-06-172-139/+0
|
* better DeprecationWarning filtersBryan Newbold2020-06-171-3/+4
|
* update Makefile from fatcat-scholar tweaks/commandsBryan Newbold2020-06-171-3/+21
|
* WIP on pdf_tool.pyBryan Newbold2020-06-171-0/+137
|
* add new pdf workers/persistersBryan Newbold2020-06-174-2/+214
|
* pdf: mypy and typo fixesBryan Newbold2020-06-172-15/+22
|
* workers: refactor to pass key to process()Bryan Newbold2020-06-176-20/+28
|
* pipenv: correct poppler; update lockfileBryan Newbold2020-06-162-76/+255
|
* pipenv: flake8, pytype, blackBryan Newbold2020-06-161-0/+7
|
* pipenv: pillow and poppler (for PDF extraction)Bryan Newbold2020-06-161-0/+2
|
* initial work on PDF extraction workerBryan Newbold2020-06-162-1/+158
| | | | | This worker fetches full PDFs, then extracts thumbnails, raw text, and PDF metadata. Similar to GROBID worker.
* pdf_thumbnail script: demonstrate PDF thumbnail generationBryan Newbold2020-06-161-0/+35
|
* refactor worker fetch code into wrapper classBryan Newbold2020-06-163-141/+111
|
* rename KafkaGrobidSink -> KafkaCompressSinkBryan Newbold2020-06-163-3/+3
|
* ingest: OAI-PMH count tableBryan Newbold2020-05-281-0/+24
|
* blobs: start documenting seaweedfs backfillBryan Newbold2020-05-281-0/+53
|
* move minio directory to 'blobs'Bryan Newbold2020-05-282-0/+0
| | | | | Part of migration from minio to seaweedfs, should be agnostic about what our actual blobstore (S3 API) is.
* remove deprecated kafka_grobid.py workerBryan Newbold2020-05-261-331/+0
| | | | | | All use of pykafka was refactored to use the confluent library some time ago. And all kafka workers have been using the newer sandcrawler style worker for some time.
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-27 13:19:06 +0000