aboutsummaryrefslogtreecommitdiffstats
path: root/python/fatcat_worker.py
Commit message (Collapse)AuthorAgeFilesLines
* add ingest request feature to entity_updates workerBryan Newbold2019-11-151-0/+2
| | | | | | | | | | | | | Initially was going to create a new worker to consume from the release update channel, but couldn't get the edit context ("is this a new release, or update to an existing") from that context. Currently there is a flag in source code to control whether we only do OA releases or all releases. Starting with OA only to start slow, but should probably default to all, and make this a config flag. Should probably also have a config flag to control this entire feature. Tested locally in dev.
* small fixes to confluent-kafka importers/workersBryan Newbold2019-09-201-1/+1
| | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again)
* make default kafka env 'dev', not 'qa'Bryan Newbold2019-09-201-2/+2
|
* fix syntax bugs (container elastic worker)Bryan Newbold2019-05-301-1/+1
|
* add container update elastic workerBryan Newbold2019-05-301-0/+16
|
* file and container update kafka topicsBryan Newbold2019-05-301-2/+8
|
* push elastic release updates to new (schema) indexBryan Newbold2019-05-241-1/+1
|
* new kafka topic for release obj w/ v0.3 schemaBryan Newbold2019-05-231-2/+2
|
* elastic-release worker w/o APIBryan Newbold2019-03-041-1/+1
| | | | | Forgot that this worker really doesn't want/need any API connection at all; just an ApiClient to deserialize objects from Kafka.
* fix (untested) ElasticsearchReleaseWorker invocationBryan Newbold2019-03-041-1/+1
|
* bunch of lint/whitespace cleanupsBryan Newbold2019-02-221-1/+2
|
* more ES index fixesBryan Newbold2019-01-281-1/+1
|
* workers do API-passing (not URI-passing)Bryan Newbold2019-01-081-5/+5
|
* start refactoring API object passingBryan Newbold2019-01-081-2/+9
|
* bunch of pylint cleanupBryan Newbold2018-11-151-1/+1
|
* large refactor of python names/pathsBryan Newbold2018-11-151-12/+12
| | | | | | | - Add __init__.py files for fatcat_tools submodules, and use them in imports - Add a bunch of comments to files. - rename a number of classes and functions to be less verbose
* refactoring harvestersBryan Newbold2018-11-151-36/+0
|
* initial work on metadata harvest botsBryan Newbold2018-11-141-6/+43
|
* elasticsearch config for workerBryan Newbold2018-11-131-1/+8
|
* shuffle around fatcat_tools layoutBryan Newbold2018-11-131-2/+2
|
* more python module refactoringBryan Newbold2018-11-121-2/+2
|
* elastic release workerBryan Newbold2018-11-041-0/+10
|
* first-draft kafka workers (changelog, release_update)Bryan Newbold2018-11-041-0/+52