aboutsummaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/workers/elasticsearch.py
Commit message (Collapse)AuthorAgeFilesLines
* move container_status ES query code from fatcat_web to fatcat_toolsBryan Newbold2022-02-091-2/+2
| | | | | | The main motivation is to never have fatcat_tools import from fatcat_web, only vica-versa. Some code in fatcat_tools needs container stats, so starting with that code path (plus some generic helpers).
* small default config typo fixes for elasticsearch workersBryan Newbold2021-12-151-2/+2
|
* file elasticsearch index workerBryan Newbold2021-12-151-1/+34
|
* typing: add annotations to remaining fatcat_tools codeBryan Newbold2021-11-031-31/+32
| | | | | Again, these are just annotations, no changes made to get type checks to pass
* re-fix some lint issues after big 'fmt'Bryan Newbold2021-11-021-2/+3
|
* fmt (black): fatcat_tools/Bryan Newbold2021-11-021-61/+110
|
* hacks to work around new pylint false positivesBryan Newbold2021-11-021-2/+3
|
* cleanup imports after fatcat_tools.transforms changeBryan Newbold2021-11-021-5/+8
|
* es worker: ensure kafka messages get clearedBryan Newbold2021-04-121-0/+2
|
* es indexing: more 'wip' fixesBryan Newbold2021-04-121-1/+5
|
* ES indexing: skip 'wip' entities with a warningBryan Newbold2021-04-121-11/+16
|
* container ES index worker: support for querying statusBryan Newbold2021-04-061-5/+32
|
* indexing: don't use document namesBryan Newbold2021-04-061-14/+4
|
* lint (flake8) tool python filesBryan Newbold2020-07-011-2/+0
|
* more changelog ES fixesBryan Newbold2020-04-171-4/+6
|
* ES changelog worker: fixes for ident; fetch update from API if neededBryan Newbold2020-04-171-2/+9
| | | | | The API fetch update may be needed for old changelog entries in the kafka feed.
* derive changelog worker from release workerMartin Czygan2020-04-171-1/+22
| | | | | Early versions of changelog entries may not have all the fields required for the current transform.
* fix KafkaError worker reporting for partition errorsBryan Newbold2020-01-291-1/+1
|
* review/fix all confluent-kafka produce codeBryan Newbold2019-09-201-8/+14
|
* small fixes to confluent-kafka importers/workersBryan Newbold2019-09-201-3/+30
| | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again)
* convert pipeline workers from pykafka to confluent-kafkaBryan Newbold2019-09-201-29/+66
|
* refactor all python source for client lib nameBryan Newbold2019-09-051-1/+1
|
* start new ES container worker kafka groupBryan Newbold2019-07-311-0/+2
| | | | | | | | The previous group seems to have gotten corrupted; my hypothesis is that this is due to pykafka being somewhat flakey, and am planning to move to librdkafka anyways. Re-indexing all the containers is pretty small/easy, so starting a new consumer group works find in this case; release indexer would be a bigger problem.
* more links on new homepageBryan Newbold2019-06-191-1/+1
| | | | | matching produce sizes. may want to tweak this config in the future for throughput.
* fix and workaround container entities in release topicBryan Newbold2019-05-301-0/+6
|
* fix syntax bugs (container elastic worker)Bryan Newbold2019-05-301-5/+5
|
* add container update elastic workerBryan Newbold2019-05-301-5/+25
|
* elastic-release worker w/o APIBryan Newbold2019-03-041-4/+4
| | | | | Forgot that this worker really doesn't want/need any API connection at all; just an ApiClient to deserialize objects from Kafka.
* fix elastic research worker api argBryan Newbold2019-03-041-4/+3
|
* fatcat -> fatcat_release ES indexBryan Newbold2019-01-281-2/+3
|
* check request status codes idiomaticallyBryan Newbold2018-12-291-1/+1
|
* large refactor of python names/pathsBryan Newbold2018-11-151-0/+52
- Add __init__.py files for fatcat_tools submodules, and use them in imports - Add a bunch of comments to files. - rename a number of classes and functions to be less verbose