| Commit message (Collapse) | Author | Age | Files | Lines | |
|---|---|---|---|---|---|
| * | persist PDF extraction in ingest pipeline | Bryan Newbold | 2020-10-20 | 1 | -4/+16 |
| | | | | | | Ooof, didn't realize that this wasn't happening. Explains a lot of missing thumbnails in scholar! | ||||
| * | customize timeout per worker; 120sec for pdf-extract | Bryan Newbold | 2020-06-29 | 1 | -0/+1 |
| | | | | | | This is a stab-in-the-dark attempt to resolve long timeouts with this worker in prod. | ||||
| * | args.kafka_env refactor didn't happen (yet) | Bryan Newbold | 2020-06-25 | 1 | -2/+2 |
| | | |||||
| * | s3-only mode persist workers use different consumer group | Bryan Newbold | 2020-06-25 | 1 | -2/+8 |
| | | |||||
| * | sandcrawler_worker: remove duplicate run_pdf_extract() | Bryan Newbold | 2020-06-25 | 1 | -29/+0 |
| | | |||||
| * | pdfextract worker | Bryan Newbold | 2020-06-25 | 1 | -1/+34 |
| | | |||||
| * | fixes and tweaks from testing locally | Bryan Newbold | 2020-06-17 | 1 | -1/+2 |
| | | |||||
| * | tweak kafka topic names and seaweedfs layout | Bryan Newbold | 2020-06-17 | 1 | -9/+10 |
| | | |||||
| * | add new pdf workers/persisters | Bryan Newbold | 2020-06-17 | 1 | -0/+83 |
| | | |||||
| * | skip-db option also for worker | Bryan Newbold | 2020-03-19 | 1 | -0/+4 |
| | | |||||
| * | ingest: bulk workers don't hit SPNv2 | Bryan Newbold | 2020-02-13 | 1 | -0/+2 |
| | | |||||
| * | pdftrio basic python code | Bryan Newbold | 2020-02-12 | 1 | -0/+19 |
| | | | | | This is basically just a copy/paste of GROBID code, only simpler! | ||||
| * | sandcrawler_worker: ingest worker distinct consumer groups | Bryan Newbold | 2020-01-29 | 1 | -1/+3 |
| | | | | | | | I'm in the process of resetting these consumer groups, so might as well take the opportunity to split by topic and use the new canonical naming format. | ||||
| * | make grobid-extract worker batch size 1 | Bryan Newbold | 2020-01-28 | 1 | -0/+1 |
| | | | | | | This is part of attempts to fix Kafka errors that look like they might be timeouts. | ||||
| * | improve sentry reporting with 'release' git hash | Bryan Newbold | 2020-01-15 | 1 | -1/+5 |
| | | |||||
| * | bulk ingest file request topic support | Bryan Newbold | 2020-01-14 | 1 | -1/+7 |
| | | |||||
| * | grobid-to-kafka support in ingest worker | Bryan Newbold | 2020-01-14 | 1 | -0/+6 |
| | | |||||
| * | update persist worker invocation to use batches | Bryan Newbold | 2020-01-02 | 1 | -15/+55 |
| | | |||||
| * | fix sandcrawler persist workers | Bryan Newbold | 2020-01-02 | 1 | -8/+36 |
| | | |||||
| * | start work on persist workers and tool | Bryan Newbold | 2020-01-02 | 1 | -5/+15 |
| | | |||||
| * | refactor: improve argparse usage | Bryan Newbold | 2019-12-18 | 1 | -4/+8 |
| | | | | | | use ArgumentDefaultsHelpFormatter and add help messages to all sub-commands | ||||
| * | update ingest-file batch size to 1 | Bryan Newbold | 2019-11-14 | 1 | -1/+1 |
| | | | | | | | | | Was defaulting to 100, which I think was resulting in lots of consumer group timeouts, resulting in UNKNOWN_MEMBER_ID errors. Will probably switch back to batches of 10 or so, but multi-processing or some other concurrent dispatch/processing. | ||||
| * | fix lint errors | Bryan Newbold | 2019-11-13 | 1 | -5/+10 |
| | | |||||
| * | correct ingest-file consumer group | Bryan Newbold | 2019-11-13 | 1 | -1/+1 |
| | | |||||
| * | add basic sandcrawler worker (kafka) | Bryan Newbold | 2019-11-13 | 1 | -0/+74 |
 |
index : sandcrawler | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |