Rough plan: - run bulk and/or regular ingest requests for just those of AIT partners (200k?) - persist ingest requests (22 million or so) - run bulk ingest over 'no status' / 'no match' requests (aka, those not in unpaywall) - crawl those which are no-capture ## Generate Requests Newer version of `mag_ingest_request.sh` script requires venv with urlcanon installed. Starting with the 2020-01-23 MAG dump, will generate a full ingest request set (including DOI `ext_id` when available), with any dominant domains removed (eg, arxiv.org): export LC_ALL=C cat PaperUrls_mag_url_doi.all.txt | rg -a -v arxiv.org | rg -a "://" | ./mag_ingest_request.py - --created-date 2020-01-23 | pv -l > ingest_requests_mag-2020-01-23.doi.json => previously 25.6M => 25.6M 2:29:43 [2.85k/s] export LC_ALL=C zcat PaperUrls_mag_url_pmid.txt.gz | rg -a -v arxiv.org | rg -a "://" | ./mag_ingest_request.py - --created-date 2020-01-23 --pmid | pv -l > ingest_requests_mag-2020-01-23.pmid.json => 4.3M 0:25:45 [2.78k/s] export LC_ALL=C cat ingest_requests_mag-2020-01-23.json | jq -r "[.base_url, .ext_ids.doi] | @tsv" | sort -u -S 4G > ingest_requests_mag-2020-01-23.full.seed_id zcat PaperUrls_PaperExtendedAttributes_pdf.txt.gz | wc -l => 6,504,907 zcat PaperUrls_mag_url_pmid.txt.gz | wc -l => 4,369,832 cat ingest_requests_mag-2020-01-23.json | jq .ext_ids.doi -r | rg -a -v '^null$' | wc -l => previously 15,707,405 => 15,702,581 cat ingest_requests_mag-2020-01-23.pmid.json | jq .base_url -r | rg ' ' | wc -l => 0 URL encoding seems to be working ## Persist Ingest Requests First pmid ingest requests, then the all/doi file. The reason to do this order is that the all/doi file will have some rows with no DOI (and thus no `ext_id`), while the PMID file will not. # small sample head /schnell/mag/20200123/ingest_requests_mag-2020-01-23.pmid.json | ./persist_tool.py ingest-request - Worker: Counter({'total': 10, 'skip-result-fields': 10}) JSON lines pushed: Counter({'total': 10, 'pushed': 10}) cat /schnell/mag/20200123/ingest_requests_mag-2020-01-23.pmid.json | ./persist_tool.py ingest-request - => 4.3M 0:16:46 [4.27k/s] Worker: Counter({'total': 4295026, 'insert-requests': 4241862, 'update-requests': 0}) JSON lines pushed: Counter({'total': 4295026, 'pushed': 4295026}) => hit a bug on first attempt, which is why total/insert results don't match cat /schnell/mag/20200123/ingest_requests_mag-2020-01-23.doi.json | ./persist_tool.py ingest-request - => 25.6M 2:21:54 [3.01k/s] Worker: Counter({'total': 25596559, 'insert-requests': 21348393, 'update-requests': 0}) JSON lines pushed: Counter({'pushed': 25596559, 'total': 25596559}) ## Crawl/Dupe Status SELECT ingest_file_result.status, COUNT(*) FROM ingest_request LEFT JOIN ingest_file_result ON ingest_file_result.ingest_type = ingest_request.ingest_type AND ingest_file_result.base_url = ingest_request.base_url WHERE ingest_request.ingest_type = 'pdf' AND ingest_request.link_source = 'mag' GROUP BY status ORDER BY COUNT DESC LIMIT 20; After just PMID links: status | count ---------------------+--------- | 3000115 success | 1126881 no-capture | 69459 terminal-bad-status | 30259 redirect-loop | 11656 no-pdf-link | 2836 wrong-mimetype | 1456 link-loop | 1259 wayback-error | 1232 cdx-error | 932 null-body | 85 petabox-error | 50 bad-redirect | 1 (13 rows) After all links: SELECT COUNT(*) FROM ingest_request WHERE ingest_request.ingest_type = 'pdf' AND ingest_request.link_source = 'mag'; => 25596563 status | count ---------------------+---------- | 21130841 success | 3915682 no-capture | 391813 terminal-bad-status | 76488 redirect-loop | 44202 wrong-mimetype | 16418 no-pdf-link | 10995 wayback-error | 3679 cdx-error | 3414 link-loop | 2098 null-body | 709 petabox-error | 221 bad-gzip-encoding | 2 bad-redirect | 1 (14 rows) Somewhat more un-ingested than expected. Dump requests: COPY ( SELECT row_to_json(ingest_request.*) FROM ingest_request LEFT JOIN ingest_file_result ON ingest_file_result.ingest_type = ingest_request.ingest_type AND ingest_file_result.base_url = ingest_request.base_url WHERE ingest_request.ingest_type = 'pdf' AND ingest_request.link_source = 'mag' AND ingest_file_result.status IS NULL ) TO '/grande/snapshots/mag_noingest_20200305.rows.json'; => COPY 21,130,841 Transform and shuf: ./scripts/ingestrequest_row2json.py /grande/snapshots/mag_noingest_20200305.rows.json | pv -l | shuf | gzip > /grande/snapshots/mag_noingest_20200305.shuf.json.gz => 21.1M 0:18:57 [18.6k/s] ## Bulk Ingest Partner Output These are subsets of the full list from potential AIT-S partners; want to run these through the pipeline before the full batch. Duplication against the full batch should be minimal. Size: bnewbold@ia601101$ cat ingest_requests_mag-2020-01-23.cornell.json | jq .ext_ids.doi | rg -v '^null$' | wc -l 29007 bnewbold@ia601101$ wc -l ingest_requests_mag-2020-01-23.cornell.json 34265 ingest_requests_mag-2020-01-23.cornell.json Test ingest: head -n200 ingest_requests_mag-2020-01-23.cornell.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1 Full ingests: cat ingest_requests_mag-2020-01-23.cornell.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1 cat ingest_requests_mag-2020-01-23.alberta.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1 cat ingest_requests_mag-2020-01-23.columbia.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1 cat ingest_requests_mag-2020-01-23.emory.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1 cat ingest_requests_mag-2020-01-23.stanford.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1 ## Bulk Ingest Shard it into batches of roughly 1 million: cd /grande/snapshots/ zcat /grande/snapshots/mag_noingest_20200305.shuf.json.gz | split -n r/20 -d - mag_noingest_20200305.ingest_request.split_ --additional-suffix=.json Add a single batch like: cat mag_noingest_20200305.ingest_request.split_00.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1 partner ingests (see above) => 2020-03-05 12:49: 118,396 1056543 mag_noingest_20200305.ingest_request.split_00.json => 2020-03-05 14:34: 1,055,224 => check on stats/ratios; filter by ingest update time? 1056542 mag_noingest_20200305.ingest_request.split_01.json 1056542 mag_noingest_20200305.ingest_request.split_02.json 1056542 mag_noingest_20200305.ingest_request.split_03.json 1056542 mag_noingest_20200305.ingest_request.split_04.json 1056542 mag_noingest_20200305.ingest_request.split_05.json 1056542 mag_noingest_20200305.ingest_request.split_06.json 1056542 mag_noingest_20200305.ingest_request.split_07.json 1056542 mag_noingest_20200305.ingest_request.split_08.json 1056542 mag_noingest_20200305.ingest_request.split_09.json => 2020-03-05 18:04: 10,009,297 => 2020-03-06 16:53: 6,553,946 1056542 mag_noingest_20200305.ingest_request.split_10.json 1056542 mag_noingest_20200305.ingest_request.split_11.json 1056542 mag_noingest_20200305.ingest_request.split_12.json 1056542 mag_noingest_20200305.ingest_request.split_13.json 1056542 mag_noingest_20200305.ingest_request.split_14.json 1056542 mag_noingest_20200305.ingest_request.split_15.json 1056542 mag_noingest_20200305.ingest_request.split_16.json 1056542 mag_noingest_20200305.ingest_request.split_17.json 1056542 mag_noingest_20200305.ingest_request.split_18.json 1056542 mag_noingest_20200305.ingest_request.split_19.json => 2020-03-06 16:59: 17,001,032 Stats from bulk ingest: SELECT ingest_file_result.status, COUNT(*) FROM ingest_request LEFT JOIN ingest_file_result ON ingest_file_result.ingest_type = ingest_request.ingest_type AND ingest_file_result.base_url = ingest_request.base_url WHERE ingest_request.ingest_type = 'pdf' AND ingest_request.link_source = 'mag' GROUP BY status ORDER BY COUNT DESC LIMIT 20; status | count ---------------------+---------- no-capture | 12237193 success | 11991293 no-pdf-link | 521691 redirect-loop | 437192 terminal-bad-status | 231181 link-loop | 92633 cdx-error | 33631 wrong-mimetype | 28638 wayback-error | 19651 null-body | 2682 petabox-error | 727 | 47 bad-redirect | 44 bad-gzip-encoding | 7 (14 rows) Failures by domain: SELECT domain, status, COUNT((domain, status)) FROM ( SELECT ingest_file_result.ingest_type, ingest_file_result.status, substring(ingest_file_result.terminal_url FROM '[^/]+://([^/]*)') AS domain FROM ingest_file_result LEFT JOIN ingest_request ON ingest_file_result.ingest_type = ingest_request.ingest_type AND ingest_file_result.base_url = ingest_request.base_url WHERE ingest_file_result.ingest_type = 'pdf' AND ingest_request.link_source = 'mag' ) t1 WHERE t1.domain != '' AND t1.status != 'success' AND t1.status != 'no-capture' GROUP BY domain, status ORDER BY COUNT DESC LIMIT 30; domain | status | count --------------------------------------+---------------------+-------- dialnet.unirioja.es | redirect-loop | 240967 onlinelibrary.wiley.com | no-pdf-link | 147696 agupubs.onlinelibrary.wiley.com | no-pdf-link | 72639 iopscience.iop.org | terminal-bad-status | 69591 febs.onlinelibrary.wiley.com | no-pdf-link | 49874 www.researchgate.net | redirect-loop | 42859 journals.sagepub.com | no-pdf-link | 27448 papers.ssrn.com | redirect-loop | 27328 dialnet.unirioja.es | terminal-bad-status | 20320 physoc.onlinelibrary.wiley.com | no-pdf-link | 20232 science.sciencemag.org | link-loop | 17811 espace.library.uq.edu.au | redirect-loop | 17185 bpspubs.onlinelibrary.wiley.com | no-pdf-link | 15785 obgyn.onlinelibrary.wiley.com | no-pdf-link | 15301 anthrosource.onlinelibrary.wiley.com | no-pdf-link | 13746 www.tandfonline.com | no-pdf-link | 13303 aasldpubs.onlinelibrary.wiley.com | no-pdf-link | 11070 link.springer.com | redirect-loop | 10594 www.redalyc.org:9081 | no-pdf-link | 10515 watermark.silverchair.com | terminal-bad-status | 9739 www.bmj.com | link-loop | 9389 www.repository.naturalis.nl | redirect-loop | 8213 bjp.rcpsych.org | link-loop | 8045 aslopubs.onlinelibrary.wiley.com | no-pdf-link | 7814 nph.onlinelibrary.wiley.com | no-pdf-link | 7801 iopscience.iop.org | redirect-loop | 7697 journals.tubitak.gov.tr | wrong-mimetype | 7159 www.biorxiv.org | wrong-mimetype | 7067 www.erudit.org | redirect-loop | 6819 besjournals.onlinelibrary.wiley.com | no-pdf-link | 6254 (30 rows) Domains to follow-up (eg, sandcrawler ingest tests/tweaks): - dialnet.unirioja.es | redirect-loop | 240967 - www.researchgate.net | redirect-loop | 42859 - www.redalyc.org:9081 | no-pdf-link | 10515 - www.repository.naturalis.nl | redirect-loop | 8213 - bjp.rcpsych.org | link-loop | 8045 - journals.tubitak.gov.tr | wrong-mimetype | 7159 - www.erudit.org | redirect-loop | 6819 The dialnet.unirioja.es ones may be worth re-crawling via heritrix? Top uncrawled domains: SELECT domain, status, COUNT((domain, status)) FROM ( SELECT ingest_file_result.ingest_type, ingest_file_result.status, substring(ingest_file_result.base_url FROM '[^/]+://([^/]*)') AS domain FROM ingest_file_result LEFT JOIN ingest_request ON ingest_file_result.ingest_type = ingest_request.ingest_type AND ingest_file_result.base_url = ingest_request.base_url WHERE ingest_file_result.ingest_type = 'pdf' AND ingest_request.link_source = 'mag' ) t1 WHERE t1.domain != '' AND t1.status = 'no-capture' GROUP BY domain, status ORDER BY COUNT DESC LIMIT 30; domain | status | count ---------------------------------+------------+-------- ieeexplore.ieee.org | no-capture | 957835 link.springer.com | no-capture | 394121 www.researchgate.net | no-capture | 376974 cyberleninka.ru | no-capture | 376012 iopscience.iop.org | no-capture | 348791 papers.ssrn.com | no-capture | 286860 dergipark.org.tr | no-capture | 217556 dialnet.unirioja.es | no-capture | 214398 academic.oup.com | no-capture | 212364 www.tandfonline.com | no-capture | 148940 journals.sagepub.com | no-capture | 144695 www.papersearch.net | no-capture | 138986 absimage.aps.org | no-capture | 111976 apps.dtic.mil | no-capture | 106984 www.cambridge.org | no-capture | 97533 www.bmj.com | no-capture | 92437 bioone.org | no-capture | 87573 science.sciencemag.org | no-capture | 75723 shodhganga.inflibnet.ac.in:8080 | no-capture | 75395 www.jstor.org | no-capture | 73230 works.bepress.com | no-capture | 68747 www.scielo.org.co | no-capture | 59650 hrcak.srce.hr | no-capture | 59332 muse.jhu.edu | no-capture | 57828 onlinelibrary.wiley.com | no-capture | 55621 www.jbc.org | no-capture | 54608 www.jstage.jst.go.jp | no-capture | 53631 www.redalyc.org | no-capture | 50406 lup.lub.lu.se | no-capture | 47469 www.dtic.mil | no-capture | 41820 (30 rows) ## Heritrix Seedlist Generation Dump ingest requests (filtered for some domains that don't expect to crawl via heritrix): COPY ( SELECT row_to_json(ingest_request.*) FROM ingest_request LEFT JOIN ingest_file_result ON ingest_file_result.ingest_type = ingest_request.ingest_type AND ingest_file_result.base_url = ingest_request.base_url WHERE ingest_request.ingest_type = 'pdf' AND ingest_request.link_source = 'mag' AND ingest_file_result.status = 'no-capture' AND ingest_request.base_url NOT LIKE '%journals.sagepub.com%' AND ingest_request.base_url NOT LIKE '%pubs.acs.org%' AND ingest_request.base_url NOT LIKE '%ahajournals.org%' AND ingest_request.base_url NOT LIKE '%www.journal.csj.jp%' AND ingest_request.base_url NOT LIKE '%aip.scitation.org%' AND ingest_request.base_url NOT LIKE '%academic.oup.com%' AND ingest_request.base_url NOT LIKE '%tandfonline.com%' ) TO '/grande/snapshots/mag_nocapture_20200313.rows.json'; => COPY 11714199 # in sandcrawler pipenv ./scripts/ingestrequest_row2json.py /grande/snapshots/mag_nocapture_20200313.rows.json > /grande/snapshots/mag_nocapture_20200313.json ## Bulk Ingest of Heritrix Content Small sample: head -n 1000 mag_nocapture_20200313.json | rg -v "\\\\" | jq . -c | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1 Full run: cat mag_nocapture_20200313.json | rg -v "\\\\" | jq . -c | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1 2020-04-07 12:19 (pacific): 11,703,871 ## Post-bulk-ingest Around 2020-04-28, seems like main wave of bulk ingest is complete. Will need to re-try things like cdx-error. Current status: status | count -------------------------------+---------- success | 18491799 redirect-loop | 1968530 no-capture | 1373657 no-pdf-link | 1311842 link-loop | 1296439 terminal-bad-status | 627577 cdx-error | 418278 wrong-mimetype | 50141 wayback-error | 37159 petabox-error | 11249 null-body | 6295 gateway-timeout | 3051 spn2-cdx-lookup-failure | 328 spn2-error:invalid-url-syntax | 93 bad-redirect | 75 | 47 invalid-host-resolution | 28 spn2-error | 10 bad-gzip-encoding | 7 redirects-exceeded | 2 (20 rows) Lots of cdx-error to retry. The no-capture links are probably a mix of domain-blocklist and things that failed in bulk mode. Will dump and re-attempt them: COPY ( SELECT row_to_json(ingest_request.*) FROM ingest_request LEFT JOIN ingest_file_result ON ingest_file_result.ingest_type = ingest_request.ingest_type AND ingest_file_result.base_url = ingest_request.base_url WHERE ingest_request.ingest_type = 'pdf' AND ingest_request.link_source = 'mag' AND ingest_file_result.status = 'no-capture' AND ingest_request.base_url NOT LIKE '%journals.sagepub.com%' AND ingest_request.base_url NOT LIKE '%pubs.acs.org%' AND ingest_request.base_url NOT LIKE '%ahajournals.org%' AND ingest_request.base_url NOT LIKE '%www.journal.csj.jp%' AND ingest_request.base_url NOT LIKE '%aip.scitation.org%' AND ingest_request.base_url NOT LIKE '%academic.oup.com%' AND ingest_request.base_url NOT LIKE '%tandfonline.com%' ) TO '/grande/snapshots/mag_nocapture_20200420.rows.json'; => 859849 What domains are these? cat mag_nocapture_20200420.rows.json | jq .base_url -r | cut -f3 -d/ | sort | uniq -c | sort -nr | head -n30 Let's filter down more: cat mag_nocapture_20200420.rows.json | rg -v 'www.researchgate.net' | rg -v 'muse.jhu.edu' | rg -v 'www.omicsonline.org' | rg -v 'link.springer.com' | rg -v 'iopscience.iop.org' | rg -v 'ieeexplore.ieee.org' | shuf > mag_nocapture_20200420.rows.filtered.json wc -l mag_nocapture_20200420.rows.filtered.json 423085 mag_nocapture_20200420.rows.filtered.json Ok, enqueue! cat mag_nocapture_20200420.rows.filtered.json | shuf | jq . -c | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests -p -1 ## Final Stats ... for this round of ingest: SELECT ingest_file_result.status, COUNT(*) FROM ingest_request LEFT JOIN ingest_file_result ON ingest_file_result.ingest_type = ingest_request.ingest_type AND ingest_file_result.base_url = ingest_request.base_url WHERE ingest_request.ingest_type = 'pdf' AND ingest_request.link_source = 'mag' GROUP BY status ORDER BY COUNT DESC LIMIT 20; status | count -------------------------------------+---------- success | 18712849 redirect-loop | 2008110 no-pdf-link | 1337012 link-loop | 1326761 no-capture | 1030693 terminal-bad-status | 637143 gateway-timeout | 193194 cdx-error | 125907 spn2-cdx-lookup-failure | 77842 wrong-mimetype | 50882 wayback-error | 40278 invalid-host-resolution | 35201 petabox-error | 11254 null-body | 6485 spn2-error | 1643 spn2-error:job-failed | 747 spn2-error:invalid-url-syntax | 325 spn2-error:soft-time-limit-exceeded | 190 bad-redirect | 77 | 47 (20 rows) Failures by domain: SELECT domain, status, COUNT((domain, status)) FROM ( SELECT ingest_file_result.ingest_type, ingest_file_result.status, substring(ingest_file_result.terminal_url FROM '[^/]+://([^/]*)') AS domain FROM ingest_file_result LEFT JOIN ingest_request ON ingest_file_result.ingest_type = ingest_request.ingest_type AND ingest_file_result.base_url = ingest_request.base_url WHERE ingest_file_result.ingest_type = 'pdf' AND ingest_request.link_source = 'mag' ) t1 WHERE t1.domain != '' AND t1.status != 'success' GROUP BY domain, status ORDER BY COUNT DESC LIMIT 30; domain | status | count ---------------------------------+---------------------+-------- ieeexplore.ieee.org | redirect-loop | 677712 cyberleninka.ru | link-loop | 308390 papers.ssrn.com | link-loop | 281804 ieeexplore.ieee.org | link-loop | 273559 dialnet.unirioja.es | redirect-loop | 240504 dialnet.unirioja.es | terminal-bad-status | 232481 onlinelibrary.wiley.com | no-pdf-link | 220932 iopscience.iop.org | terminal-bad-status | 172480 validate.perfdrive.com | no-pdf-link | 172312 link.springer.com | redirect-loop | 130398 agupubs.onlinelibrary.wiley.com | no-pdf-link | 113382 iopscience.iop.org | redirect-loop | 105234 www.bmj.com | link-loop | 100354 www.researchgate.net | redirect-loop | 84366 www.cambridge.org | link-loop | 83171 jamanetwork.com | no-pdf-link | 75053 febs.onlinelibrary.wiley.com | no-pdf-link | 74872 www.jstor.org | redirect-loop | 72059 journals.sagepub.com | no-pdf-link | 63028 science.sciencemag.org | redirect-loop | 62927 profile.thieme.de | no-pdf-link | 62406 cyberleninka.ru | redirect-loop | 56733 link.springer.com | link-loop | 47608 physoc.onlinelibrary.wiley.com | no-pdf-link | 30180 science.sciencemag.org | link-loop | 29908 papers.ssrn.com | redirect-loop | 27255 obgyn.onlinelibrary.wiley.com | no-pdf-link | 26789 www.computer.org | no-pdf-link | 26444 watermark.silverchair.com | terminal-bad-status | 25934 www.nature.com | redirect-loop | 25306 (30 rows)