HTML ingest deployment notes

1 files changed, 71 insertions, 1 deletions

diff --git a/notes/html_ingest_notes.md b/notes/html_ingest_notes.md
index 6bb876a..a1a91f3 100644
--- a/notes/html_ingest_notes.md
+++ b/notes/html_ingest_notes.md
@@ -18,7 +18,7 @@ x CDX lookup "closest" to capture datetime (or by month)
 x firstmonday no extracted fulltext/XML
 x apply URL base fixup to fulltext URLs
 x XML alternative detection
-- basic ingest worker, kafka topics, persist workers, sql table, etc
+x basic ingest worker, kafka topics, persist workers, sql table, etc
 - ingest worker: landing page to actual fulltext (eg, OJS)
 - broken? https://betterexplained.com/articles/colorized-math-equations/
 
@@ -246,3 +246,73 @@ Examples/bugs:
 
     generally, author detection not great.
     not, apparently, using detection of dc.authors etc
+
+
+## Prod Deployment Notes (2020-12-14)
+
+Created `html_meta` table in `sandcrawler-db`.
+
+Updated ansible roles to deploy persist and import workers. Then ran the roles
+and enabled:
+
+- sandcrawler database (aitio)
+    - sandcrawler-persist-ingest-file-worker@1: restarted
+- blobs (wbgrp-svc169)
+    - sandcrawler-persist-html-teixml-worker@1: started and enabled
+    - sandcrawler-persist-xml-doc-worker@1: started and enabled
+- fatcat prod worker (wbgrp-svc502)
+    - fatcat-import-ingest-web-worker: started and enabled
+
+Test some d-lib and first monday ingests:
+
+    # dlib
+    ./fatcat_ingest.py --env prod --enqueue-kafka --kafka-hosts wbgrp-svc263.us.archive.org --ingest-type html --limit 50 container --container-id ugbiirfvufgcjkx33r3cmemcuu
+    => Counter({'estimate': 803, 'ingest_request': 50, 'elasticsearch_release': 50, 'kafka': 50})
+
+    # first monday
+    ./fatcat_ingest.py --env prod --enqueue-kafka --kafka-hosts wbgrp-svc263.us.archive.org --ingest-type html --limit 50 container --container-id svz5ul6qozdjhjhk7d627avuja
+
+Starting:
+
+    d-lib: 253 / 1056 preserved (https://fatcat.wiki/container/ugbiirfvufgcjkx33r3cmemcuu/coverage)
+
+Initially, `fatcat-import-ingest-web-worker` is seeing these but doesn't seem
+to be importing.
+
+    # postgresql shell
+    select sha1hex, updated, status, scope, has_teixml, has_thumbnail, word_count from html_meta;
+    => initially has_teixml is false for all
+    => fixed in an update
+
+    # weed shell
+    > fs.ls /buckets/sandcrawler/html_body
+    [...]
+    > fs.cat /buckets/sandcrawler/html_body/77/75/7775adf8c7e19151bbe887bfa08a575483291d7c.tei.xml
+    [looks like fine TEI-XML]
+
+Going to debug ingest issue by dumping results to disk and importing manually
+(best way to see counts):
+
+    kafkacat -C -b wbgrp-svc284.us.archive.org:9092 -t sandcrawler-prod.ingest-file-results -o -10 | rg html | head -n10 | jq . -c > web_ingest_results.json
+
+    export FATCAT_AUTH_WORKER_CRAWL=[...]
+    ./fatcat_import.py ingest-web-results web_ingest_results.json
+    => Counter({'total': 10, 'skip-update-disabled': 9, 'skip': 1, 'skip-hit': 1, 'insert': 0, 'update': 0, 'exists': 0})
+
+    # did some patching (f7a75a01), then re-ran twice and got:
+    => Counter({'total': 10, 'insert': 9, 'skip': 1, 'skip-hit': 1, 'update': 0, 'exists': 0})
+    => Counter({'total': 10, 'exists': 9, 'skip': 1, 'skip-hit': 1, 'insert': 0, 'update': 0})
+
+    # looks good!
+
+Re-ingesting all of d-lib:
+
+    ./fatcat_ingest.py --env prod --enqueue-kafka --kafka-hosts wbgrp-svc263.us.archive.org --ingest-type html container --container-id ugbiirfvufgcjkx33r3cmemcuu
+    => Expecting 803 release objects in search queries
+    => Counter({'ingest_request': 803, 'elasticsearch_release': 803, 'estimate': 803, 'kafka': 803})
+
+TODO:
+
+- release ES transform isn't counting these as `in_ia` or preserved (code-only change)
+- no indication in search results (ES schema change)
+- ingest tool should probably look at `in_ia_html` or `in_ia_pdf` for PDF/XML queries (or a `types_in_ia` list?)