| Commit message (Collapse) | Author | Age | Files | Lines | |
|---|---|---|---|---|---|
| * | sandcrawler_worker: remove duplicate run_pdf_extract() | Bryan Newbold | 2020-06-25 | 1 | -29/+0 |
| | | |||||
| * | gitlab-ci: install most recent cmake via PPA | Bryan Newbold | 2020-06-25 | 1 | -3/+7 |
| | | |||||
| * | pdfextract worker | Bryan Newbold | 2020-06-25 | 1 | -1/+34 |
| | | |||||
| * | pdfextract: don't compress thumbnail output | Bryan Newbold | 2020-06-25 | 1 | -1/+1 |
| | | |||||
| * | pipenv: python-poppler 0.2.1 | Bryan Newbold | 2020-06-25 | 2 | -49/+51 |
| | | |||||
| * | add xenial poppler PPA (PDF parsing) | Bryan Newbold | 2020-06-18 | 1 | -0/+1 |
| | | |||||
| * | fixes and tweaks from testing locally | Bryan Newbold | 2020-06-17 | 6 | -18/+134 |
| | | |||||
| * | fixes to pdfextract_tool | Bryan Newbold | 2020-06-17 | 1 | -12/+8 |
| | | |||||
| * | tweak pdf_meta SQL schema | Bryan Newbold | 2020-06-17 | 2 | -5/+31 |
| | | |||||
| * | add a docker-compose file for local development | Bryan Newbold | 2020-06-17 | 2 | -0/+50 |
| | | |||||
| * | write up blob storage (w/ seaweedfs) | Bryan Newbold | 2020-06-17 | 2 | -0/+95 |
| | | | | | This follows-on to Martin's earlier proposal. | ||||
| * | tweak kafka topic names and seaweedfs layout | Bryan Newbold | 2020-06-17 | 3 | -13/+16 |
| | | |||||
| * | make process_pdf() more robust to parse errors | Bryan Newbold | 2020-06-17 | 1 | -5/+29 |
| | | |||||
| * | note about text layout with pdf extraction | Bryan Newbold | 2020-06-17 | 1 | -0/+8 |
| | | |||||
| * | lint fixes | Bryan Newbold | 2020-06-17 | 4 | -7/+9 |
| | | |||||
| * | rename pdf tools to pdfextract | Bryan Newbold | 2020-06-17 | 3 | -0/+0 |
| | | |||||
| * | coverage-html makefile target | Bryan Newbold | 2020-06-17 | 1 | -0/+3 |
| | | |||||
| * | add debian dependencies for python-poppler to CI setup | Bryan Newbold | 2020-06-17 | 1 | -1/+1 |
| | | |||||
| * | pdf thumbnail+text+meta proposal | Bryan Newbold | 2020-06-17 | 1 | -0/+327 |
| | | |||||
| * | ignore PIL deprecation warnings | Bryan Newbold | 2020-06-17 | 1 | -0/+1 |
| | | |||||
| * | partial test coverage of pdf extract worker | Bryan Newbold | 2020-06-17 | 2 | -6/+70 |
| | | |||||
| * | fix coverage command | Bryan Newbold | 2020-06-17 | 2 | -2/+4 |
| | | |||||
| * | update grobid2json with type annotations | Bryan Newbold | 2020-06-17 | 1 | -94/+110 |
| | | |||||
| * | remove unused common.py | Bryan Newbold | 2020-06-17 | 2 | -139/+0 |
| | | |||||
| * | better DeprecationWarning filters | Bryan Newbold | 2020-06-17 | 1 | -3/+4 |
| | | |||||
| * | update Makefile from fatcat-scholar tweaks/commands | Bryan Newbold | 2020-06-17 | 1 | -3/+21 |
| | | |||||
| * | WIP on pdf_tool.py | Bryan Newbold | 2020-06-17 | 1 | -0/+137 |
| | | |||||
| * | add new pdf workers/persisters | Bryan Newbold | 2020-06-17 | 4 | -2/+214 |
| | | |||||
| * | pdf: mypy and typo fixes | Bryan Newbold | 2020-06-17 | 2 | -15/+22 |
| | | |||||
| * | workers: refactor to pass key to process() | Bryan Newbold | 2020-06-17 | 6 | -20/+28 |
| | | |||||
| * | pipenv: correct poppler; update lockfile | Bryan Newbold | 2020-06-16 | 2 | -76/+255 |
| | | |||||
| * | pipenv: flake8, pytype, black | Bryan Newbold | 2020-06-16 | 1 | -0/+7 |
| | | |||||
| * | pipenv: pillow and poppler (for PDF extraction) | Bryan Newbold | 2020-06-16 | 1 | -0/+2 |
| | | |||||
| * | initial work on PDF extraction worker | Bryan Newbold | 2020-06-16 | 2 | -1/+158 |
| | | | | | | This worker fetches full PDFs, then extracts thumbnails, raw text, and PDF metadata. Similar to GROBID worker. | ||||
| * | pdf_thumbnail script: demonstrate PDF thumbnail generation | Bryan Newbold | 2020-06-16 | 1 | -0/+35 |
| | | |||||
| * | refactor worker fetch code into wrapper class | Bryan Newbold | 2020-06-16 | 3 | -141/+111 |
| | | |||||
| * | rename KafkaGrobidSink -> KafkaCompressSink | Bryan Newbold | 2020-06-16 | 3 | -3/+3 |
| | | |||||
| * | ingest: OAI-PMH count table | Bryan Newbold | 2020-05-28 | 1 | -0/+24 |
| | | |||||
| * | blobs: start documenting seaweedfs backfill | Bryan Newbold | 2020-05-28 | 1 | -0/+53 |
| | | |||||
| * | move minio directory to 'blobs' | Bryan Newbold | 2020-05-28 | 2 | -0/+0 |
| | | | | | | Part of migration from minio to seaweedfs, should be agnostic about what our actual blobstore (S3 API) is. | ||||
| * | remove deprecated kafka_grobid.py worker | Bryan Newbold | 2020-05-26 | 1 | -331/+0 |
| | | | | | | | All use of pykafka was refactored to use the confluent library some time ago. And all kafka workers have been using the newer sandcrawler style worker for some time. | ||||
| * | ingest notes | Bryan Newbold | 2020-05-26 | 2 | -6/+76 |
| | | |||||
| * | pipenv: remove old python3.5 cruft; add mypy | Bryan Newbold | 2020-05-26 | 2 | -185/+196 |
| | | |||||
| * | Merge branch 'martin-seaweed-s3' into 'master' | bnewbold | 2020-05-26 | 1 | -0/+424 |
| |\ | | | | | | | | | notes on seaweedfs (s3 backend) See merge request webgroup/sandcrawler!28 | ||||
| | * | notes on seaweedfs (s3 backend) | Martin Czygan | 2020-05-20 | 1 | -0/+424 |
| | | | | | | | | | Notes gathered during seaweedfs setup and test runs. | ||||
| * | | potential future backfill ingests | Bryan Newbold | 2020-05-26 | 1 | -0/+52 |
| | | | |||||
| * | | ingests: normalize file names; commit updates | Bryan Newbold | 2020-05-26 | 10 | -63/+279 |
| | | | |||||
| * | | start a python Makefile | Bryan Newbold | 2020-05-19 | 1 | -0/+15 |
| | | | |||||
| * | | handle UnboundLocalError in HTML parsing | Bryan Newbold | 2020-05-19 | 1 | -1/+4 |
| | | | |||||
| * | | first iteration of oai2ingestrequest script | Bryan Newbold | 2020-05-05 | 1 | -0/+137 |
| | | | |||||
 |
index : sandcrawler | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |