Ran a bulk dump using fatcat ingest tool several months ago, and had Martin run a crawl. Crawl is now done, so going to ingest, hoping to get the majority of the millions of remaining arxiv.org PDFs. zcat /grande/snapshots/fatcat_missing_arxiv_ingest_request.2020-08-21.json.gz | wc -l => 1,288,559 zcat /grande/snapshots/fatcat_missing_arxiv_ingest_request.2020-08-21.json.gz | rg -v "\\\\" | jq . -c | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1