summaryrefslogtreecommitdiffstats
path: root/python/fatcat_worker.py
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'bnewbold-file-es' into 'master'bnewbold2022-01-211-0/+28
|\ | | | | | | | | File entity elasticsearch index worker See merge request webgroup/fatcat!136
| * file elasticsearch index workerBryan Newbold2021-12-151-0/+28
| |
* | move from raven to sentry_sdkMartin Czygan2021-12-141-2/+2
|/ | | | | | | | | related docs: * https://docs.sentry.io/platforms/python/guides/flask/migration/ * https://docs.sentry.io/platforms/python/guides/asgi/configuration/integrations/flask/ > `fetch_git_sha` is gone, see: https://forum.sentry.io/t/fetch-git-sha-equivalent-in-the-unified-python-sdk/5521
* typing: first batch of python bulk type annotationsBryan Newbold2021-11-031-6/+6
| | | | | | While these changes are more delicate than simple lint changes, this specific batch of edits and annotations was *relatively* simple, and resulted in few code changes other than function signature additions.
* fmt (black): *.pyBryan Newbold2021-11-021-47/+92
|
* python: isort everythingBryan Newbold2021-11-021-2/+9
|
* default ingest request topic now '-daily'; configurable for ingest_tool.pyBryan Newbold2021-09-301-1/+1
|
* container ES index worker: support for querying statusBryan Newbold2021-04-061-0/+5
|
* entity updater: new work update feed (ident and changelog metadata only)Bryan Newbold2020-10-161-0/+2
|
* lint (flake8) top-level python filesBryan Newbold2020-07-011-1/+0
|
* derive changelog worker from release workerMartin Czygan2020-04-171-1/+18
| | | | | Early versions of changelog entries may not have all the fields required for the current transform.
* improve argparse usageBryan Newbold2019-12-111-8/+10
| | | | | | | | | | | | | | Use --fatcat-api-url instead of (ambiguous) --host-url for commands that aren't deployed/running via systemd. TODO: update the other --host-url usage, and either roll-out change consistently or support the old arg as an alias during cut-over Use argparse.ArgumentDefaultsHelpFormatter (thanks Martin!) Add help messages for all sub-commands, both as documentation and as a way to get argparse to print available commands in a more readable format.
* add ingest request feature to entity_updates workerBryan Newbold2019-11-151-0/+2
| | | | | | | | | | | | | Initially was going to create a new worker to consume from the release update channel, but couldn't get the edit context ("is this a new release, or update to an existing") from that context. Currently there is a flag in source code to control whether we only do OA releases or all releases. Starting with OA only to start slow, but should probably default to all, and make this a config flag. Should probably also have a config flag to control this entire feature. Tested locally in dev.
* small fixes to confluent-kafka importers/workersBryan Newbold2019-09-201-1/+1
| | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again)
* make default kafka env 'dev', not 'qa'Bryan Newbold2019-09-201-2/+2
|
* fix syntax bugs (container elastic worker)Bryan Newbold2019-05-301-1/+1
|
* add container update elastic workerBryan Newbold2019-05-301-0/+16
|
* file and container update kafka topicsBryan Newbold2019-05-301-2/+8
|
* push elastic release updates to new (schema) indexBryan Newbold2019-05-241-1/+1
|
* new kafka topic for release obj w/ v0.3 schemaBryan Newbold2019-05-231-2/+2
|
* elastic-release worker w/o APIBryan Newbold2019-03-041-1/+1
| | | | | Forgot that this worker really doesn't want/need any API connection at all; just an ApiClient to deserialize objects from Kafka.
* fix (untested) ElasticsearchReleaseWorker invocationBryan Newbold2019-03-041-1/+1
|
* bunch of lint/whitespace cleanupsBryan Newbold2019-02-221-1/+2
|
* more ES index fixesBryan Newbold2019-01-281-1/+1
|
* workers do API-passing (not URI-passing)Bryan Newbold2019-01-081-5/+5
|
* start refactoring API object passingBryan Newbold2019-01-081-2/+9
|
* bunch of pylint cleanupBryan Newbold2018-11-151-1/+1
|
* large refactor of python names/pathsBryan Newbold2018-11-151-12/+12
| | | | | | | - Add __init__.py files for fatcat_tools submodules, and use them in imports - Add a bunch of comments to files. - rename a number of classes and functions to be less verbose
* refactoring harvestersBryan Newbold2018-11-151-36/+0
|
* initial work on metadata harvest botsBryan Newbold2018-11-141-6/+43
|
* elasticsearch config for workerBryan Newbold2018-11-131-1/+8
|
* shuffle around fatcat_tools layoutBryan Newbold2018-11-131-2/+2
|
* more python module refactoringBryan Newbold2018-11-121-2/+2
|
* elastic release workerBryan Newbold2018-11-041-0/+10
|
* first-draft kafka workers (changelog, release_update)Bryan Newbold2018-11-041-0/+52