summaryrefslogtreecommitdiffstats
path: root/notes/performance
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@robocracy.org>2018-11-21 22:21:52 -0800
committerBryan Newbold <bnewbold@robocracy.org>2018-11-21 22:22:32 -0800
commit3f6a7483e64c907e4af2e87427321e65e77b169c (patch)
treedd9ea5a3ed000015cd3b36c8ea938622b27822aa /notes/performance
parentfb80eab1bdae2d21a3dda2e82230b7477ed41ebc (diff)
downloadfatcat-3f6a7483e64c907e4af2e87427321e65e77b169c.tar.gz
fatcat-3f6a7483e64c907e4af2e87427321e65e77b169c.zip
update notes/TODO
Diffstat (limited to 'notes/performance')
-rw-r--r--notes/performance/kafka_pipeline.txt16
1 files changed, 16 insertions, 0 deletions
diff --git a/notes/performance/kafka_pipeline.txt b/notes/performance/kafka_pipeline.txt
index 0a503a18..0ff2e411 100644
--- a/notes/performance/kafka_pipeline.txt
+++ b/notes/performance/kafka_pipeline.txt
@@ -29,3 +29,19 @@ elastic-release python processing is at 66% (of one core) CPU! and elastic at
~30%. Huh.
But, in general, "seems to be working".
+
+## End-To-End
+
+release-updates: 40/sec
+api-crossref: 40/sec
+api-datacite: 15/sec
+changelog: 11/sec
+consumer_offsets: 0.5/sec
+
+elastic indexing looks like only 8/sec or so. Probably need to batch.
+
+Tried running additional fatcat-elasticsearch-release-worker processes, and
+throughput goes linearly.
+
+Are consumer group names not actually topic-dependent? Hrm, might need to
+rename them all for prod/qa split.