diff options
| author | Bryan Newbold <bnewbold@robocracy.org> | 2018-11-21 22:21:52 -0800 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@robocracy.org> | 2018-11-21 22:22:32 -0800 |
| commit | 3f6a7483e64c907e4af2e87427321e65e77b169c (patch) | |
| tree | dd9ea5a3ed000015cd3b36c8ea938622b27822aa /notes/performance/kafka_pipeline.txt | |
| parent | fb80eab1bdae2d21a3dda2e82230b7477ed41ebc (diff) | |
| download | fatcat-3f6a7483e64c907e4af2e87427321e65e77b169c.tar.gz fatcat-3f6a7483e64c907e4af2e87427321e65e77b169c.zip | |
update notes/TODO
Diffstat (limited to 'notes/performance/kafka_pipeline.txt')
| -rw-r--r-- | notes/performance/kafka_pipeline.txt | 16 |
1 files changed, 16 insertions, 0 deletions
diff --git a/notes/performance/kafka_pipeline.txt b/notes/performance/kafka_pipeline.txt index 0a503a18..0ff2e411 100644 --- a/notes/performance/kafka_pipeline.txt +++ b/notes/performance/kafka_pipeline.txt @@ -29,3 +29,19 @@ elastic-release python processing is at 66% (of one core) CPU! and elastic at ~30%. Huh. But, in general, "seems to be working". + +## End-To-End + +release-updates: 40/sec +api-crossref: 40/sec +api-datacite: 15/sec +changelog: 11/sec +consumer_offsets: 0.5/sec + +elastic indexing looks like only 8/sec or so. Probably need to batch. + +Tried running additional fatcat-elasticsearch-release-worker processes, and +throughput goes linearly. + +Are consumer group names not actually topic-dependent? Hrm, might need to +rename them all for prod/qa split. |