Rough plan:
- run bulk and/or regular ingest requests for just those of AIT partners (200k?)
- persist ingest requests (22 million or so)
- run bulk ingest over 'no status' / 'no match' requests (aka, those not in unpaywall)
- crawl those which are no-capture
## Generate Requests
Newer version of `mag_ingest_request.sh` script requires venv with urlcanon
installed.
Starting with the 2020-01-23 MAG dump, will generate a full ingest request set
(including DOI `ext_id` when available), with any dominant domains removed (eg,
arxiv.org):
export LC_ALL=C
cat PaperUrls_mag_url_doi.all.txt | rg -a -v arxiv.org | rg -a "://" | ./mag_ingest_request.py - --created-date 2020-01-23 | pv -l > ingest_requests_mag-2020-01-23.doi.json
=> previously 25.6M
=> 25.6M 2:29:43 [2.85k/s]
export LC_ALL=C
zcat PaperUrls_mag_url_pmid.txt.gz | rg -a -v arxiv.org | rg -a "://" | ./mag_ingest_request.py - --created-date 2020-01-23 --pmid | pv -l > ingest_requests_mag-2020-01-23.pmid.json
=> 4.3M 0:25:45 [2.78k/s]
export LC_ALL=C
cat ingest_requests_mag-2020-01-23.json | jq -r "[.base_url, .ext_ids.doi] | @tsv" | sort -u -S 4G > ingest_requests_mag-2020-01-23.full.seed_id
zcat PaperUrls_PaperExtendedAttributes_pdf.txt.gz | wc -l
=> 6,504,907
zcat PaperUrls_mag_url_pmid.txt.gz | wc -l
=> 4,369,832
cat ingest_requests_mag-2020-01-23.json | jq .ext_ids.doi -r | rg -a -v '^null$' | wc -l
=> previously 15,707,405
=> 15,702,581
cat ingest_requests_mag-2020-01-23.pmid.json | jq .base_url -r | rg ' ' | wc -l
=> 0
URL encoding seems to be working
## Persist Ingest Requests
First pmid ingest requests, then the all/doi file. The reason to do this order
is that the all/doi file will have some rows with no DOI (and thus no
`ext_id`), while the PMID file will not.
# small sample
head /schnell/mag/20200123/ingest_requests_mag-2020-01-23.pmid.json | ./persist_tool.py ingest-request -
Worker: Counter({'total': 10, 'skip-result-fields': 10})
JSON lines pushed: Counter({'total': 10, 'pushed': 10})
cat /schnell/mag/20200123/ingest_requests_mag-2020-01-23.pmid.json | ./persist_tool.py ingest-request -
=> 4.3M 0:16:46 [4.27k/s]
Worker: Counter({'total': 4295026, 'insert-requests': 4241862, 'update-requests': 0})
JSON lines pushed: Counter({'total': 4295026, 'pushed': 4295026})
=> hit a bug on first attempt, which is why total/insert results don't match
cat /schnell/mag/20200123/ingest_requests_mag-2020-01-23.doi.json | ./persist_tool.py ingest-request -
=> 25.6M 2:21:54 [3.01k/s]
Worker: Counter({'total': 25596559, 'insert-requests': 21348393, 'update-requests': 0})
JSON lines pushed: Counter({'pushed': 25596559, 'total': 25596559})
## Crawl/Dupe Status
SELECT ingest_file_result.status, COUNT(*)
FROM ingest_request
LEFT JOIN ingest_file_result
ON ingest_file_result.ingest_type = ingest_request.ingest_type
AND ingest_file_result.base_url = ingest_request.base_url
WHERE
ingest_request.ingest_type = 'pdf'
AND ingest_request.link_source = 'mag'
GROUP BY status
ORDER BY COUNT DESC
LIMIT 20;
After just PMID links:
status | count
---------------------+---------
| 3000115
success | 1126881
no-capture | 69459
terminal-bad-status | 30259
redirect-loop | 11656
no-pdf-link | 2836
wrong-mimetype | 1456
link-loop | 1259
wayback-error | 1232
cdx-error | 932
null-body | 85
petabox-error | 50
bad-redirect | 1
(13 rows)
After all links:
SELECT COUNT(*)
FROM ingest_request
WHERE
ingest_request.ingest_type = 'pdf'
AND ingest_request.link_source = 'mag';
=> 25596563
status | count
---------------------+----------
| 21130841
success | 3915682
no-capture | 391813
terminal-bad-status | 76488
redirect-loop | 44202
wrong-mimetype | 16418
no-pdf-link | 10995
wayback-error | 3679
cdx-error | 3414
link-loop | 2098
null-body | 709
petabox-error | 221
bad-gzip-encoding | 2
bad-redirect | 1
(14 rows)
Somewhat more un-ingested than expected.
Dump requests:
COPY (
SELECT row_to_json(ingest_request.*) FROM ingest_request
LEFT JOIN ingest_file_result
ON ingest_file_result.ingest_type = ingest_request.ingest_type
AND ingest_file_result.base_url = ingest_request.base_url
WHERE
ingest_request.ingest_type = 'pdf'
AND ingest_request.link_source = 'mag'
AND ingest_file_result.status IS NULL
) TO '/grande/snapshots/mag_noingest_20200305.rows.json';
=> COPY 21,130,841
Transform and shuf:
./scripts/ingestrequest_row2json.py /grande/snapshots/mag_noingest_20200305.rows.json | pv -l | shuf | gzip > /grande/snapshots/mag_noingest_20200305.shuf.json.gz
=> 21.1M 0:18:57 [18.6k/s]
## Bulk Ingest Partner Output
These are subsets of the full list from potential AIT-S partners; want to run
these through the pipeline before the full batch. Duplication against the full
batch should be minimal.
Size:
bnewbold@ia601101$ cat ingest_requests_mag-2020-01-23.cornell.json | jq .ext_ids.doi | rg -v '^null$' | wc -l
29007
bnewbold@ia601101$ wc -l ingest_requests_mag-2020-01-23.cornell.json
34265 ingest_requests_mag-2020-01-23.cornell.json
Test ingest:
head -n200 ingest_requests_mag-2020-01-23.cornell.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1
Full ingests:
cat ingest_requests_mag-2020-01-23.cornell.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1
cat ingest_requests_mag-2020-01-23.alberta.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1
cat ingest_requests_mag-2020-01-23.columbia.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1
cat ingest_requests_mag-2020-01-23.emory.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1
cat ingest_requests_mag-2020-01-23.stanford.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1
## Bulk Ingest
Shard it into batches of roughly 1 million:
cd /grande/snapshots/
zcat /grande/snapshots/mag_noingest_20200305.shuf.json.gz | split -n r/20 -d - mag_noingest_20200305.ingest_request.split_ --additional-suffix=.json
Add a single batch like:
cat mag_noingest_20200305.ingest_request.split_00.json | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1
partner ingests (see above)
=> 2020-03-05 12:49: 118,396
1056543 mag_noingest_20200305.ingest_request.split_00.json
=> 2020-03-05 14:34: 1,055,224
=> check on stats/ratios; filter by ingest update time?
1056542 mag_noingest_20200305.ingest_request.split_01.json
1056542 mag_noingest_20200305.ingest_request.split_02.json
1056542 mag_noingest_20200305.ingest_request.split_03.json
1056542 mag_noingest_20200305.ingest_request.split_04.json
1056542 mag_noingest_20200305.ingest_request.split_05.json
1056542 mag_noingest_20200305.ingest_request.split_06.json
1056542 mag_noingest_20200305.ingest_request.split_07.json
1056542 mag_noingest_20200305.ingest_request.split_08.json
1056542 mag_noingest_20200305.ingest_request.split_09.json
=> 2020-03-05 18:04: 10,009,297
=> 2020-03-06 16:53: 6,553,946
1056542 mag_noingest_20200305.ingest_request.split_10.json
1056542 mag_noingest_20200305.ingest_request.split_11.json
1056542 mag_noingest_20200305.ingest_request.split_12.json
1056542 mag_noingest_20200305.ingest_request.split_13.json
1056542 mag_noingest_20200305.ingest_request.split_14.json
1056542 mag_noingest_20200305.ingest_request.split_15.json
1056542 mag_noingest_20200305.ingest_request.split_16.json
1056542 mag_noingest_20200305.ingest_request.split_17.json
1056542 mag_noingest_20200305.ingest_request.split_18.json
1056542 mag_noingest_20200305.ingest_request.split_19.json
=> 2020-03-06 16:59: 17,001,032
Stats from bulk ingest:
SELECT ingest_file_result.status, COUNT(*)
FROM ingest_request
LEFT JOIN ingest_file_result
ON ingest_file_result.ingest_type = ingest_request.ingest_type
AND ingest_file_result.base_url = ingest_request.base_url
WHERE
ingest_request.ingest_type = 'pdf'
AND ingest_request.link_source = 'mag'
GROUP BY status
ORDER BY COUNT DESC
LIMIT 20;
status | count
---------------------+----------
no-capture | 12237193
success | 11991293
no-pdf-link | 521691
redirect-loop | 437192
terminal-bad-status | 231181
link-loop | 92633
cdx-error | 33631
wrong-mimetype | 28638
wayback-error | 19651
null-body | 2682
petabox-error | 727
| 47
bad-redirect | 44
bad-gzip-encoding | 7
(14 rows)
Failures by domain:
SELECT domain, status, COUNT((domain, status))
FROM (
SELECT
ingest_file_result.ingest_type,
ingest_file_result.status,
substring(ingest_file_result.terminal_url FROM '[^/]+://([^/]*)') AS domain
FROM ingest_file_result
LEFT JOIN ingest_request
ON ingest_file_result.ingest_type = ingest_request.ingest_type
AND ingest_file_result.base_url = ingest_request.base_url
WHERE
ingest_file_result.ingest_type = 'pdf'
AND ingest_request.link_source = 'mag'
) t1
WHERE t1.domain != ''
AND t1.status != 'success'
AND t1.status != 'no-capture'
GROUP BY domain, status
ORDER BY COUNT DESC
LIMIT 30;
domain | status | count
--------------------------------------+---------------------+--------
dialnet.unirioja.es | redirect-loop | 240967
onlinelibrary.wiley.com | no-pdf-link | 147696
agupubs.onlinelibrary.wiley.com | no-pdf-link | 72639
iopscience.iop.org | terminal-bad-status | 69591
febs.onlinelibrary.wiley.com | no-pdf-link | 49874
www.researchgate.net | redirect-loop | 42859
journals.sagepub.com | no-pdf-link | 27448
papers.ssrn.com | redirect-loop | 27328
dialnet.unirioja.es | terminal-bad-status | 20320
physoc.onlinelibrary.wiley.com | no-pdf-link | 20232
science.sciencemag.org | link-loop | 17811
espace.library.uq.edu.au | redirect-loop | 17185
bpspubs.onlinelibrary.wiley.com | no-pdf-link | 15785
obgyn.onlinelibrary.wiley.com | no-pdf-link | 15301
anthrosource.onlinelibrary.wiley.com | no-pdf-link | 13746
www.tandfonline.com | no-pdf-link | 13303
aasldpubs.onlinelibrary.wiley.com | no-pdf-link | 11070
link.springer.com | redirect-loop | 10594
www.redalyc.org:9081 | no-pdf-link | 10515
watermark.silverchair.com | terminal-bad-status | 9739
www.bmj.com | link-loop | 9389
www.repository.naturalis.nl | redirect-loop | 8213
bjp.rcpsych.org | link-loop | 8045
aslopubs.onlinelibrary.wiley.com | no-pdf-link | 7814
nph.onlinelibrary.wiley.com | no-pdf-link | 7801
iopscience.iop.org | redirect-loop | 7697
journals.tubitak.gov.tr | wrong-mimetype | 7159
www.biorxiv.org | wrong-mimetype | 7067
www.erudit.org | redirect-loop | 6819
besjournals.onlinelibrary.wiley.com | no-pdf-link | 6254
(30 rows)
Domains to follow-up (eg, sandcrawler ingest tests/tweaks):
- dialnet.unirioja.es | redirect-loop | 240967
- www.researchgate.net | redirect-loop | 42859
- www.redalyc.org:9081 | no-pdf-link | 10515
- www.repository.naturalis.nl | redirect-loop | 8213
- bjp.rcpsych.org | link-loop | 8045
- journals.tubitak.gov.tr | wrong-mimetype | 7159
- www.erudit.org | redirect-loop | 6819
The dialnet.unirioja.es ones may be worth re-crawling via heritrix?
Top uncrawled domains:
SELECT domain, status, COUNT((domain, status))
FROM (
SELECT
ingest_file_result.ingest_type,
ingest_file_result.status,
substring(ingest_file_result.base_url FROM '[^/]+://([^/]*)') AS domain
FROM ingest_file_result
LEFT JOIN ingest_request
ON ingest_file_result.ingest_type = ingest_request.ingest_type
AND ingest_file_result.base_url = ingest_request.base_url
WHERE
ingest_file_result.ingest_type = 'pdf'
AND ingest_request.link_source = 'mag'
) t1
WHERE t1.domain != ''
AND t1.status = 'no-capture'
GROUP BY domain, status
ORDER BY COUNT DESC
LIMIT 30;
domain | status | count
---------------------------------+------------+--------
ieeexplore.ieee.org | no-capture | 957835
link.springer.com | no-capture | 394121
www.researchgate.net | no-capture | 376974
cyberleninka.ru | no-capture | 376012
iopscience.iop.org | no-capture | 348791
papers.ssrn.com | no-capture | 286860
dergipark.org.tr | no-capture | 217556
dialnet.unirioja.es | no-capture | 214398
academic.oup.com | no-capture | 212364
www.tandfonline.com | no-capture | 148940
journals.sagepub.com | no-capture | 144695
www.papersearch.net | no-capture | 138986
absimage.aps.org | no-capture | 111976
apps.dtic.mil | no-capture | 106984
www.cambridge.org | no-capture | 97533
www.bmj.com | no-capture | 92437
bioone.org | no-capture | 87573
science.sciencemag.org | no-capture | 75723
shodhganga.inflibnet.ac.in:8080 | no-capture | 75395
www.jstor.org | no-capture | 73230
works.bepress.com | no-capture | 68747
www.scielo.org.co | no-capture | 59650
hrcak.srce.hr | no-capture | 59332
muse.jhu.edu | no-capture | 57828
onlinelibrary.wiley.com | no-capture | 55621
www.jbc.org | no-capture | 54608
www.jstage.jst.go.jp | no-capture | 53631
www.redalyc.org | no-capture | 50406
lup.lub.lu.se | no-capture | 47469
www.dtic.mil | no-capture | 41820
(30 rows)
## Heritrix Seedlist Generation
Dump ingest requests (filtered for some domains that don't expect to crawl via
heritrix):
COPY (
SELECT row_to_json(ingest_request.*) FROM ingest_request
LEFT JOIN ingest_file_result
ON ingest_file_result.ingest_type = ingest_request.ingest_type
AND ingest_file_result.base_url = ingest_request.base_url
WHERE
ingest_request.ingest_type = 'pdf'
AND ingest_request.link_source = 'mag'
AND ingest_file_result.status = 'no-capture'
AND ingest_request.base_url NOT LIKE '%journals.sagepub.com%'
AND ingest_request.base_url NOT LIKE '%pubs.acs.org%'
AND ingest_request.base_url NOT LIKE '%ahajournals.org%'
AND ingest_request.base_url NOT LIKE '%www.journal.csj.jp%'
AND ingest_request.base_url NOT LIKE '%aip.scitation.org%'
AND ingest_request.base_url NOT LIKE '%academic.oup.com%'
AND ingest_request.base_url NOT LIKE '%tandfonline.com%'
) TO '/grande/snapshots/mag_nocapture_20200313.rows.json';
=> COPY 11714199
# in sandcrawler pipenv
./scripts/ingestrequest_row2json.py /grande/snapshots/mag_nocapture_20200313.rows.json > /grande/snapshots/mag_nocapture_20200313.json
## Bulk Ingest of Heritrix Content
Small sample:
head -n 1000 mag_nocapture_20200313.json | rg -v "\\\\" | jq . -c | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1
Full run:
cat mag_nocapture_20200313.json | rg -v "\\\\" | jq . -c | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1
2020-04-07 12:19 (pacific): 11,703,871
## Post-bulk-ingest
Around 2020-04-28, seems like main wave of bulk ingest is complete. Will need
to re-try things like cdx-error.
Current status:
status | count
-------------------------------+----------
success | 18491799
redirect-loop | 1968530
no-capture | 1373657
no-pdf-link | 1311842
link-loop | 1296439
terminal-bad-status | 627577
cdx-error | 418278
wrong-mimetype | 50141
wayback-error | 37159
petabox-error | 11249
null-body | 6295
gateway-timeout | 3051
spn2-cdx-lookup-failure | 328
spn2-error:invalid-url-syntax | 93
bad-redirect | 75
| 47
invalid-host-resolution | 28
spn2-error | 10
bad-gzip-encoding | 7
redirects-exceeded | 2
(20 rows)
Lots of cdx-error to retry.
The no-capture links are probably a mix of domain-blocklist and things that
failed in bulk mode. Will dump and re-attempt them:
COPY (
SELECT row_to_json(ingest_request.*) FROM ingest_request
LEFT JOIN ingest_file_result
ON ingest_file_result.ingest_type = ingest_request.ingest_type
AND ingest_file_result.base_url = ingest_request.base_url
WHERE
ingest_request.ingest_type = 'pdf'
AND ingest_request.link_source = 'mag'
AND ingest_file_result.status = 'no-capture'
AND ingest_request.base_url NOT LIKE '%journals.sagepub.com%'
AND ingest_request.base_url NOT LIKE '%pubs.acs.org%'
AND ingest_request.base_url NOT LIKE '%ahajournals.org%'
AND ingest_request.base_url NOT LIKE '%www.journal.csj.jp%'
AND ingest_request.base_url NOT LIKE '%aip.scitation.org%'
AND ingest_request.base_url NOT LIKE '%academic.oup.com%'
AND ingest_request.base_url NOT LIKE '%tandfonline.com%'
) TO '/grande/snapshots/mag_nocapture_20200420.rows.json';
=> 859849
What domains are these?
cat mag_nocapture_20200420.rows.json | jq .base_url -r | cut -f3 -d/ | sort | uniq -c | sort -nr | head -n30
Let's filter down more:
cat mag_nocapture_20200420.rows.json | rg -v 'www.researchgate.net' | rg -v 'muse.jhu.edu' | rg -v 'www.omicsonline.org' | rg -v 'link.springer.com' | rg -v 'iopscience.iop.org' | rg -v 'ieeexplore.ieee.org' | shuf > mag_nocapture_20200420.rows.filtered.json
wc -l mag_nocapture_20200420.rows.filtered.json
423085 mag_nocapture_20200420.rows.filtered.json
Ok, enqueue!
cat mag_nocapture_20200420.rows.filtered.json | shuf | jq . -c | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests -p -1
## Final Stats
... for this round of ingest:
SELECT ingest_file_result.status, COUNT(*)
FROM ingest_request
LEFT JOIN ingest_file_result
ON ingest_file_result.ingest_type = ingest_request.ingest_type
AND ingest_file_result.base_url = ingest_request.base_url
WHERE
ingest_request.ingest_type = 'pdf'
AND ingest_request.link_source = 'mag'
GROUP BY status
ORDER BY COUNT DESC
LIMIT 20;
status | count
-------------------------------------+----------
success | 18712849
redirect-loop | 2008110
no-pdf-link | 1337012
link-loop | 1326761
no-capture | 1030693
terminal-bad-status | 637143
gateway-timeout | 193194
cdx-error | 125907
spn2-cdx-lookup-failure | 77842
wrong-mimetype | 50882
wayback-error | 40278
invalid-host-resolution | 35201
petabox-error | 11254
null-body | 6485
spn2-error | 1643
spn2-error:job-failed | 747
spn2-error:invalid-url-syntax | 325
spn2-error:soft-time-limit-exceeded | 190
bad-redirect | 77
| 47
(20 rows)
Failures by domain:
SELECT domain, status, COUNT((domain, status))
FROM (
SELECT
ingest_file_result.ingest_type,
ingest_file_result.status,
substring(ingest_file_result.terminal_url FROM '[^/]+://([^/]*)') AS domain
FROM ingest_file_result
LEFT JOIN ingest_request
ON ingest_file_result.ingest_type = ingest_request.ingest_type
AND ingest_file_result.base_url = ingest_request.base_url
WHERE
ingest_file_result.ingest_type = 'pdf'
AND ingest_request.link_source = 'mag'
) t1
WHERE t1.domain != ''
AND t1.status != 'success'
GROUP BY domain, status
ORDER BY COUNT DESC
LIMIT 30;
domain | status | count
---------------------------------+---------------------+--------
ieeexplore.ieee.org | redirect-loop | 677712
cyberleninka.ru | link-loop | 308390
papers.ssrn.com | link-loop | 281804
ieeexplore.ieee.org | link-loop | 273559
dialnet.unirioja.es | redirect-loop | 240504
dialnet.unirioja.es | terminal-bad-status | 232481
onlinelibrary.wiley.com | no-pdf-link | 220932
iopscience.iop.org | terminal-bad-status | 172480
validate.perfdrive.com | no-pdf-link | 172312
link.springer.com | redirect-loop | 130398
agupubs.onlinelibrary.wiley.com | no-pdf-link | 113382
iopscience.iop.org | redirect-loop | 105234
www.bmj.com | link-loop | 100354
www.researchgate.net | redirect-loop | 84366
www.cambridge.org | link-loop | 83171
jamanetwork.com | no-pdf-link | 75053
febs.onlinelibrary.wiley.com | no-pdf-link | 74872
www.jstor.org | redirect-loop | 72059
journals.sagepub.com | no-pdf-link | 63028
science.sciencemag.org | redirect-loop | 62927
profile.thieme.de | no-pdf-link | 62406
cyberleninka.ru | redirect-loop | 56733
link.springer.com | link-loop | 47608
physoc.onlinelibrary.wiley.com | no-pdf-link | 30180
science.sciencemag.org | link-loop | 29908
papers.ssrn.com | redirect-loop | 27255
obgyn.onlinelibrary.wiley.com | no-pdf-link | 26789
www.computer.org | no-pdf-link | 26444
watermark.silverchair.com | terminal-bad-status | 25934
www.nature.com | redirect-loop | 25306
(30 rows)