aboutsummaryrefslogtreecommitdiffstats
path: root/kafka
Commit message (Collapse)AuthorAgeFilesLines
* new 'daily' and 'priority' ingest request topicsBryan Newbold2021-09-301-2/+8
| | | | | | | | | The old ingest request queue was always getting lopsided, suspect because it was scaled up (additional partitions) at some point in the past, hoping new topics will fix this. New '-priority' queue is like '-bulk', but for smaller-volume SPN-like requests. Eg, interactive mode.
* kafka: delete unused work-updates topicBryan Newbold2021-09-131-4/+0
|
* kafka re-balancing tweaksBryan Newbold2021-09-031-1/+2
|
* kafka docs for rolling back a consumer groupBryan Newbold2020-11-201-0/+9
|
* Merge branch 'bnewbold-html-ingest'Bryan Newbold2020-11-061-0/+16
|\
| * kafka: new XML+HTML topicsBryan Newbold2020-11-041-0/+15
| |
* | kafka topics for fatcat -> scholar pipelineBryan Newbold2020-10-271-0/+19
|/
* PDF extraction kafka topicsBryan Newbold2020-06-251-0/+24
|
* kafka: more reblance notesBryan Newbold2020-04-241-1/+14
|
* kafka: how to rebalance parititions between brokersBryan Newbold2020-04-241-0/+29
|
* topics: add pubmed ftp topicMartin Czygan2020-03-121-1/+9
| | | | PubmedFTPWorker replaced OAI recently. This documents the new topic.
* pdftrio proposal and start on schema+kafkaBryan Newbold2020-02-121-0/+8
|
* kafka config: actually we do want large bulk ingest request topicBryan Newbold2020-01-151-1/+1
|
* add new bulk ingest request topicBryan Newbold2020-01-141-1/+6
|
* kafka topics: compress api-dataciteBryan Newbold2019-12-241-1/+1
|
* new kafka topics for file ingestBryan Newbold2019-11-131-0/+15
| | | | Created these for 'qa' and 'prod'
* update kafka topic listingsBryan Newbold2019-10-031-21/+35
|
* more kafka topicsBryan Newbold2019-07-071-0/+9
|
* new release schema kafka topicBryan Newbold2019-05-241-2/+3
|
* more fatcat update topicsBryan Newbold2019-04-121-0/+3
|
* use ipv4 localhost with kafkacatBryan Newbold2018-12-191-1/+1
|
* update notesBryan Newbold2018-12-102-0/+45
|
* more kafka/grobid notesBryan Newbold2018-11-211-0/+12
|
* rename grobided to grobid-outputBryan Newbold2018-11-212-3/+7
|
* initial work on kafka_grobid workerBryan Newbold2018-11-201-0/+18
|
* kafka notesBryan Newbold2018-11-202-0/+110