Notes here about re-ingesting or re-crawling large batches. Goal around end of
2020 is to generate a broad patch crawl of terminal no-capture attempts for all
major sources crawled thus far. Have already tried run this process for unpaywall.

For each, want filtered ingest request JSON objects (filtering out platforms
that don't crawl well, and possibly things like figshare+zenodo), and a broader
seedlist (including terminal URLs). Will de-dupe all the seedlist URLs and do a
heritrix crawl with new config, then re-ingest all the requests individually.

Summary of what to do here:

    OA DOI: expecting some 2.4 million seeds
    OAI-PMH: expecting some 5 million no-capture URLs, plus more from missing PDF URL not found
    Unpaywall: another ~900k no-capture URLs (maybe filtered?)

For all, re-attempt for these status codes:

     no-capture
     cdx-error
     wayback-error
     petabox-error

And at least do bulk re-ingest for these, if updated before 2020-11-20 or so:

     no-pdf-link

## OAI-PMH

Need to re-ingest all of the (many!) no-capture and no-pdf-link

TODO: repec-specific URL extraction?

Skip these OAI prefixes:

     kb.dk
     bnf.fr
     hispana.mcu.es
     bdr.oai.bsb-muenchen.de
     ukm.si
     hsp.org

Skip these domains:

    www.kb.dk (kb.dk)
    kb-images.kb.dk (kb.dk)
    mdz-nbn-resolving.de (TODO: what prefix?)
    aggr.ukm.um.si (ukm.si)

Check PDF link extraction for these prefixes, or skip them (TODO):

    repec (mixed success)
    biodiversitylibrary.org
    juser.fz-juelich.de
    americanae.aecid.es
    www.irgrid.ac.cn
    hal
    espace.library.uq.edu.au
    igi.indrastra.com
    invenio.nusl.cz
    hypotheses.org
    t2r2.star.titech.ac.jp
    quod.lib.umich.edu

    domain: hemerotecadigital.bne.es
    domain: bib-pubdb1.desy.de
    domain: publikationen.bibliothek.kit.edu
    domain: edoc.mpg.de
    domain: bibliotecadigital.jcyl.es
    domain: lup.lub.lu.se
    domain: orbi.uliege.be

TODO:
- consider deleting ingest requests from skipped prefixes (large database use)


## Unpaywall

About 900k `no-pdf-link`, and up to 2.5 million more `no-pdf-link`.

Re-bulk-ingest filtered requests which hit `no-pdf-link` before 2020-11-20:

    COPY (
        SELECT row_to_json(ingest_request.*)
        FROM ingest_request
        LEFT JOIN ingest_file_result
            ON ingest_file_result.ingest_type = ingest_request.ingest_type
            AND ingest_file_result.base_url = ingest_request.base_url
        WHERE
            ingest_request.ingest_type = 'pdf'
            AND ingest_request.link_source = 'unpaywall'
            AND date(ingest_request.created) < '2020-11-20'
            AND ingest_file_result.status = 'no-pdf-link'
            AND ingest_request.base_url NOT LIKE '%journals.sagepub.com%'
            AND ingest_request.base_url NOT LIKE '%pubs.acs.org%'
            AND ingest_request.base_url NOT LIKE '%ahajournals.org%'
            AND ingest_request.base_url NOT LIKE '%www.journal.csj.jp%'
            AND ingest_request.base_url NOT LIKE '%aip.scitation.org%'
            AND ingest_request.base_url NOT LIKE '%academic.oup.com%'
            AND ingest_request.base_url NOT LIKE '%tandfonline.com%'
            AND ingest_request.base_url NOT LIKE '%://archive.org/%'
            AND ingest_request.base_url NOT LIKE '%://web.archive.org/%'
            AND ingest_request.base_url NOT LIKE '%://www.archive.org/%'
    ) TO '/grande/snapshots/unpaywall_nopdflink_2020-12-08.rows.json';
    => COPY 1309990

    ./scripts/ingestrequest_row2json.py /grande/snapshots/unpaywall_nopdflink_2020-12-08.rows.json | pv -l | shuf > /grande/snapshots/unpaywall_nopdflink_2020-12-08.ingest_request.json
    => 1.31M 0:00:51 [25.6k/s]

    cat /grande/snapshots/unpaywall_nopdflink_2020-12-08.rows.json | rg -v "\\\\" | jq . -c | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests-bulk -p -1