Note: as of 2022 this file is ancient and need review ## Kafka Pipelines - after network split, mass restarting import/harvest stuff seemed to completely reset consumergroups (!). bunch of LeaderNotFoundError => change/update consumer group config => ensure we are recording timestamps to allow timestamp-based resets - refactor python kafka clients (slack convo with kenji+dvd) => try librdkafka? => switch to python-kafka? - monitoring/alerting of consumergroup offsets => start with crude python script? - document: need to restart all consumers after brokers restart - operate on batches, using threads/async, and reduce worker (process) counts dramatically source of kafka-manager weirdness? Dec 02 01:05:40 wbgrp-svc263.us.archive.org kafka-manager[7032]: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'user_data': java.nio.BufferUnderflowException Dec 02 01:05:40 wbgrp-svc263.us.archive.org kafka-manager[7032]: [error] k.m.a.c.KafkaManagedOffsetCache - Failed to get member metadata from group summary and member summary : grobid-hbase-insert : MemberSummary(pykafka-8128e0be-4952-4e79-8644-a52987421259,pykafka,/207.241.225.228,[B@6c368f37,[B@2b007e01) ## Other - paper match heuristic: include 10.1007%2F978-3-319-49304-6_18 (URL-escaped slash) - catch EOFFail fetching from wayback - "author counts match" in scoring - refactor "scorable" to "matchable" - look at refactoring to reduce JSON serializations - QA tool for matches (PDF + Crossref JSON + landing page?) => python; talks directly to HBase - author counts should match (+/- one?) match strategies (hbase columns): - legacy_doi - url_doi - grobid_crossref (doi) - grobid_fatcat (fatcat ID) scalding: - better JSON library - less verbose sbt test output (set log level to WARN) - auto-formatting: addSbtPlugin("com.geirsson" % "sbt-scalafmt" % "1.6.0-RC3") pig: - potentially want to *not* de-dupe CDX lines by uniq sha1 in all cases; run this as a second-stage filter? for example, may want many URL links in fatcat for a single file (different links, different policies) - fix pig gitlab-ci tests (JAVA_HOME) python: - include input file name (and chunk? and CDX?) in sentry context - how to get argument (like --hbase-table) into mrjob.conf, or similar?