various ingest/task notes

1 files changed, 28 insertions, 1 deletions

diff --git a/notes/ingest/2022-01-13_doi_crawl.md b/notes/ingest/2022-01-13_doi_crawl.md
index 09a3b46..a6f08dd 100644
--- a/notes/ingest/2022-01-13_doi_crawl.md
+++ b/notes/ingest/2022-01-13_doi_crawl.md
@@ -137,7 +137,7 @@ many of these are likely to crawl successfully.
         > /srv/fatcat/tasks/ingest_nonoa_doi.json.gz
     # re-running 2022-02-08 after this VM was upgraded
     # Expecting 8321448 release objects in search queries
-    # TODO: in-progress
+    # DONE
 
 This is large enough that it will probably be a bulk ingest, and then probably
 a follow-up crawl.
@@ -219,3 +219,30 @@ Added to `JOURNALS-PATCH-CRAWL-2022-01`
 Unless it is a 404, should retry.
 
 TODO: generate this list
+
+## Non-OA DOI Bulk Ingest
+
+Had previously run:
+
+    cat ingest_nonoa_doi.json.gz \
+        | rg -v "doi.org/10.2139/" \
+        | rg -v "doi.org/10.1021/" \
+        | rg -v "doi.org/10.1121/" \
+        | rg -v "doi.org/10.1515/" \
+        | rg -v "doi.org/10.1093/" \
+        | rg -v "europepmc.org" \
+        | pv -l \
+        | gzip \
+        > nonoa_doi.filtered.ingests.json.gz
+    # 7.35M 0:01:13 [99.8k/s]
+
+Starting a bulk ingest of these on 2022-03-18, which is *before* the crawl has
+entirely finished, but after almost all queues (domains) have been done for
+several days.
+
+    zcat nonoa_doi.filtered.ingests.json.gz \
+        | rg -v "\\\\" \
+        | jq . -c \
+        | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1
+
+Looks like many jstage `no-capture` status; these are still (slowly) crawling.