summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/workers
Commit message (Collapse)AuthorAgeFilesLines
* start new ES container worker kafka groupBryan Newbold2019-07-311-0/+2
| | | | | | | | The previous group seems to have gotten corrupted; my hypothesis is that this is due to pykafka being somewhat flakey, and am planning to move to librdkafka anyways. Re-indexing all the containers is pretty small/easy, so starting a new consumer group works find in this case; release indexer would be a bigger problem.
* fix typo in typoBryan Newbold2019-06-241-1/+1
|
* fix typo in changelog workerBryan Newbold2019-06-241-1/+1
|
* more links on new homepageBryan Newbold2019-06-192-2/+2
| | | | | matching produce sizes. may want to tweak this config in the future for throughput.
* fix and workaround container entities in release topicBryan Newbold2019-05-302-2/+8
|
* fix syntax bugs (container elastic worker)Bryan Newbold2019-05-301-5/+5
|
* add container update elastic workerBryan Newbold2019-05-302-6/+26
|
* file and container update kafka topicsBryan Newbold2019-05-301-54/+69
|
* update elastic for releases when files addedBryan Newbold2019-05-301-1/+36
| | | | A bunch of remaining TODOs here
* 10 MByte default Kafka produce (workers)Bryan Newbold2019-03-062-2/+9
|
* elastic-release worker w/o APIBryan Newbold2019-03-041-4/+4
| | | | | Forgot that this worker really doesn't want/need any API connection at all; just an ApiClient to deserialize objects from Kafka.
* fix elastic research worker api argBryan Newbold2019-03-041-4/+3
|
* bunch of lint/whitespace cleanupsBryan Newbold2019-02-222-4/+3
|
* fatcat -> fatcat_release ES indexBryan Newbold2019-01-281-2/+3
|
* include filesets and webcaptures in exportsBryan Newbold2019-01-181-1/+1
|
* Merge branch 'bnewbold-crude-auth'Bryan Newbold2019-01-082-9/+7
|\ | | | | | | | | Fixed a conflict in: python/fatcat_export.py
| * workers do API-passing (not URI-passing)Bryan Newbold2019-01-082-9/+7
| |
* | check request status codes idiomaticallyBryan Newbold2018-12-291-1/+1
|/
* not as strong a todo (timestamps)Bryan Newbold2018-11-191-1/+1
|
* bunch of pylint cleanupBryan Newbold2018-11-151-1/+1
|
* large refactor of python names/pathsBryan Newbold2018-11-153-17/+22
| | | | | | | - Add __init__.py files for fatcat_tools submodules, and use them in imports - Add a bunch of comments to files. - rename a number of classes and functions to be less verbose
* have recent message helper cleanup consumerBryan Newbold2018-11-151-1/+5
|
* fix worker codeBryan Newbold2018-11-142-2/+5
|
* most_recent_message as reusable functionBryan Newbold2018-11-142-26/+26
|
* switch to auto consumer offset updatesBryan Newbold2018-11-132-2/+11
| | | | | | This is the classic/correct way to do consumer group updates for higher throughput, when "at least once" semantics are acceptible (as they are here; double processing should be safe/fine).
* to_elastic_dict -> release_elastic_dictBryan Newbold2018-11-131-1/+2
|
* more simple fatcat_client importsBryan Newbold2018-11-131-1/+1
|
* shuffle around fatcat_tools layoutBryan Newbold2018-11-133-0/+194