18 files changed, 413 insertions, 0 deletions
diff --git a/notes/misc/2020-08_metadata.md b/notes/misc/2020-08_metadata.md
new file mode 100644
index 00000000..12cd6fb0
--- /dev/null
+++ b/notes/misc/2020-08_metadata.md
@@ -0,0 +1,79 @@
+
+## Artificial Containers
+
+    biorxiv
+    medrxiv
+        doi_prefix:10.1101
+        publisher:"Cold Spring Harbor Laboratory"
+    -> article-journal? article? should match "paper" filter
+    -> status: draft? submitted?
+    -> there is some flag in crossref metadata...
+
+    arxiv
+    -> article-journal?
+    -> set container_name?
+
+    protocols.io
+        doi_prefix:10.17504/protocols.io.
+        container_name:protocols.io
+
+    10.25384/sage. -> sage.figshare.com
+    -> at least set container_name
+
+    figshare
+        doi_prefix:10.6084
+    -> at least set container_name
+
+    zenodo
+    -> at least set container_name
+
+Maybe? Later?
+
+    PsycEXTRA
+        container_name:"PsycEXTRA Dataset"
+        doi_prefix:10.1037
+        crossref
+    => 300k+ releases
+    => subtitle is 'number' (like "(577982012-038)")
+    => dataset
+    => publication status unknown
+
+    f1000 reviews
+        container_name:"F1000 - Post-publication peer review of the biomedical literature"
+        title:"Faculty of 1000 evaluation for "[...]
+        doi_prefix:10.3410/
+        crossref
+    => 222k releases
+    => type -> peer-review (?)
+
+    IUPAC Standards Online
+
+    GBIF
+        doi_prefix: 10.15468/dl.
+    => 838k releases
+
+==================
+
+
+later fatcat:
+- pmid+crossref pre-prints
+    https://fatcat.wiki/release/d4lrxugtqbapxgi4jrrlmzjily
+- zenodo: handle "repost from another ISSN" case (drop issn/container_id)
+- doi_prefix:10.18720 no container metadata; should be thesis type?
+- research square (10.21203) metadata (journal articles, pre-print or published?)
+- journals.ub.uni-heidelberg.de metadata is poor? no journal link
+- try_work_lookup() -> part of try update?
+    => zenodo "isidentical"
+    => zenodo "isversionof"
+    => figshare "isversionof"
+    => later, try_work_fuzzy()
+- biorxiv, medrxiv container name (and/or `container_id`?)
+    => and "article" not "post"
+- datacite container:"microPublication Biology" -> micropub type?
+- ES container index: `publisher_type` (?)
+- arxiv: remove release_type="report" logic
+- arxiv: don't include DOI, just merge under work
+- datacite release_type: resourceType=SaComponent -> 'component'
+    https://api.datacite.org/dois/10.1371/journal.pbio.0020429.g004
+- datacite title `{:unav}` (PLOS)
+    https://fatcat.wiki/release/search?q=doi_prefix%3A10.1371+unav
diff --git a/notes/misc/2020_ingest_ideas.md b/notes/misc/2020_ingest_ideas.md
new file mode 100644
index 00000000..fc5ea807
--- /dev/null
+++ b/notes/misc/2020_ingest_ideas.md
@@ -0,0 +1,14 @@
+
+https://philpapers.org/ 
+=> 2.4m entries
+
+https://philarchive.org/
+=> 50k OA papers
+=> OAI-PMH
+
+https://isidore.science/
+=> humanities in french, english, spanish
+=> APIs
+
+http://ascl.net/
+=> Astrophysics Source Code Library
diff --git a/notes/misc/2022-04_missing_oa.md b/notes/misc/2022-04_missing_oa.md
new file mode 100644
index 00000000..9a5541b9
--- /dev/null
+++ b/notes/misc/2022-04_missing_oa.md
@@ -0,0 +1,202 @@
+
+Short data exploration of what OA content is missing, and how it might be crawled.
+
+Starting with "front page" query:
+
+    is_oa:true year:>1995 year:<=2021 (type:article-journal OR type:article OR type:paper-conference) !doi_prefix:10.5281 !doi_prefix:10.6084
+
+    doi_prefix:10.6084 is figshare
+    doi_prefix:10.5281 is zenodo
+
+    14,658,673	66.56%	preserved and publicly accessible (bright)
+    3,453,052	15.68%	preserved but not publicly accessible (dark)
+    3,911,614	17.77%	no known independent preservation
+    22,023,339	100%	total
+
+Virtually all of the "dark" is also `in_shadows:true`. So the
+`preservation:none` is the high-impact target for crawling.
+
+Limiting to `publisher_type:big5`, almost zero `preservation:none`, and 1.34
+million (41%) dark.
+
+## Publisher Type
+
+Created a kibana graph of the above filters, graphing `publisher_type` ("Publisher Type breakdown of missing OA)":
+
+    <missing>   1769k   54%
+    longtail     852k   26%
+    society      195k    6%
+    unipress     130k    4%
+    scielo       114k    3.5%
+    then: repository, oa, commercial, big5
+
+## Containers
+
+    !container_id:* preservation:none is_oa:true year:>1995 year:<=2021 (type:article-journal OR type:article OR type:paper-conference) !doi_prefix:10.5281 !doi_prefix:10.6084
+
+    1,993,639 missing preservation
+
+These are virtually all Datacite DOIs (not including figshare/zenodo), and
+start in 2008, ramping up. They are almost all missing `publisher_type` (which
+makes sense because they have no container).
+
+With the filters from above, here are some top containers missing content:
+
+    Missing	                    1,993,639
+    e27twid5qnbqbboxlkrja2xz2a	12,537
+        "Proceedings of Indian National Science Academy"
+        almost zero preservation. DOAJ website is 404 for article (!), no longer in DOAJ (!)
+        some kind of bad metadata situation? almost all from 2015
+    fmoqnzpewvfrnm2ni4mbvvlney	9,350
+        "Chinese Medical Journal"
+        PMIDs only
+        missing/unpreserved is pre-2015 (significant!)
+    7l5xye7sc5emxfprwmqw2a7yxq	8,999
+        "Tidsskrift for Den norske legeforening" (norwegian medical)
+        bunch of PMIDs only; sporadic preservation coverage
+    ujftxdg3knebxhrqg4qjznz2he	5,903
+        "International Research Journal" (russian)
+        these are by-issue, with DOIs redirecting to pages inside issue (!)
+    kfzef6kfwbhpnfw3cifit7zw7q	5,678
+        "lectures"
+        hosted on openeditions
+        HTML ingest would work (!)
+    gr4g5qzzcnembf4om6yjb6qf34	5,020
+        "计算机科学"
+        mostly via dblp. some DOIs, presumably chinese?
+    bl77onlbbbhu5d6ohpjw2ypojy	4,994
+        "EOS" (from American Geophysical Union / AGU)
+        large publication, mostly preserved (dark)
+        mix of wiley.com OA (but hard to crawl?) and web/HTML stuff
+    3afvqhtpnjd5nmiphwxlxzirde	4,877
+        "Medical Science Monitor"
+        large publication, mixed preservation
+        annoying PDF link situation (hard to crawl?)
+    tulajqojzjabfc4iybyv6poi2e	4,786
+        "Dermatology Online Journal"
+        large publication, mixed preservation
+        some just pmid
+        some HTML or ePub-only
+        escholarship.org
+
+A take-away here for me is that containers are pretty heterogenous and have
+diverse issues.
+
+TODO: ingest things like: https://escholarship.org/uc/item/02v86610
+    from container_tulajqojzjabfc4iybyv6poi2e
+
+### revues.org / openedition
+
+Many of these seem like they would ingest fine via HTML.
+
+    doi_prefix:10.4000
+
+    151,565	34.3%	preserved and publicly accessible (bright)
+      7,211	1.64%	preserved but not publicly accessible (dark)
+    283,139	64.08%	no known independent preservation
+    441,915	100%	total
+
+    article-journal	230,146	    63% preserved
+    chapter	        200,724	     2% preserved
+    book	        10,971	    12% preserved
+    paper-conference	74
+
+Chapters and books don't seem as amenable to ingest... and indeed are mostly
+not marked `is_oa:true`.
+
+DONE: bulk html-mode ingest, expecting about 80k requests:
+
+    doi_prefix:10.4000 in_ia:false type:article-journal is_oa:true
+
+    ./fatcat_ingest.py --env prod --enqueue-kafka --kafka-hosts wbgrp-svc280.us.archive.org,wbgrp-svc284.us.archive.org,wbgrp-svc350.us.archive.org --kafka-request-topic sandcrawler-prod.ingest-file-requests-bulk \
+        --ingest-type html \
+        query "doi_prefix:10.4000 in_ia:false type:article-journal is_oa:true"
+    => Expecting 80032 release objects in search queries
+    => Counter({'ingest_request': 80032, 'elasticsearch_release': 80032, 'estimate': 80032, 'kafka': 80032})
+
+NOTE: have this be the default ingest type for this DOI prefix? not sure, some
+do come through as PDF just fine
+
+## Source of Records
+
+Starting with the 3,844,142 or so `preservation:none`.
+
+    doi                 3.204m
+        datacite            1.995m
+        crossref            1.087m
+        <unknown>           109k
+        jalc                12k
+    doaj_id             553k
+    pmid                192k
+    dblp_id             29k
+    arxiv_id, pmcid     0
+
+I'm surprised how good dblp coverage is? Oh, but those are almost entirely
+missing OA status, that explains it.
+
+    # NOTE: not specifically OA
+    dblp_id:* year:>1995 year:<=2021 (type:article-journal OR type:article OR type:paper-conference)
+
+    406,235	    22.54%	preserved and publicly accessible (bright)
+    59,009	    3.28%	preserved but not publicly accessible (dark)
+    1,337,554	74.2%	no known independent preservation
+    1,802,798	100%	total
+
+Looks like doi and DOAJ are big sources.
+
+    # NOTE: DOAJ implies OA, I checked and numbers are ~same
+    doaj_id:* is_oa:true year:>1995 year:<=2021 (type:article-journal OR type:article OR type:paper-conference)
+
+    588,364	    47.27%	preserved and publicly accessible (bright)
+    103,206	    8.3%	preserved but not publicly accessible (dark)
+    553,353	    44.45%	no known independent preservation
+    1,244,923	100%	total
+
+DOAJ ingest seems important to optimize!
+
+    !publisher_type:big5 container_id:* doaj_id:* is_oa:true year:>1995 year:<=2021 (type:article-journal OR type:article OR type:paper-conference)
+    => 548,709 missing preservation
+
+    doaj_id:*
+    => 589,915 missing preservation
+
+Datacite the biggest category though, even with zenodo/figshare removed.
+
+TODO: largest datacite DOI prefixes
+TODO: check sandcrawler DB to see DOAJ ingest status; maybe these are entirely missing URLs? or just not crawling well?
+TODO: dig in to "longtail" more... some random ones?
+
+## Largest DOI Prefixes
+
+    <missing>	640,104
+    10.48550 	1,543,167
+        the new arxiv.org prefix
+    10.4000	68,267
+        revues / openedition (handled above)
+    10.25384	60,063
+        figshare / SAGE
+    10.3917	52,195
+        cairn.info
+    10.25673	41,565
+        some random IR? opendata.uni-halle.de
+        TODO: ingest this type of item, possibly using dataset->file crawler
+    10.3406	33,778
+        persee.fr
+        blocks bots (don't attempt ingest)
+    10.3205	33,540
+        "german medical science"
+        HTML articles, PDF links
+        TODO: fix ingest
+        https://www.egms.de/static/en/journals/gms/2020-18/000284.shtml
+    10.17605	30,365
+        osf.io
+        TODO: fix ingest (?)
+    10.25446	26,614
+        figshare / oxford
+        "File(s) not publicly available"
+        but "CC BY 4.0"? ugh
+
+TODO: HTML crawl cairn.info (10.3917)
+TODO: ignore 10.25384, 10.25446 (figshare)
+TODO: ignore arixv.org prefix (10.48550) in default dashboard
+TODO: handle arxiv.org DOIs better (merge, count as preserved, etc)
diff --git a/notes/UNSORTED.txt b/notes/misc/UNSORTED.txt
index 850b54d0..850b54d0 100644
--- a/notes/UNSORTED.txt
+++ b/notes/misc/UNSORTED.txt
diff --git a/notes/example_entities.txt b/notes/misc/example_entities.txt
index e4016d8a..e4016d8a 100644
--- a/notes/example_entities.txt
+++ b/notes/misc/example_entities.txt
diff --git a/notes/misc/examples/content_scope.txt b/notes/misc/examples/content_scope.txt
new file mode 100644
index 00000000..321dd056
--- /dev/null
+++ b/notes/misc/examples/content_scope.txt
@@ -0,0 +1,45 @@
+
+sha1:fe27d2d036d478fb692be95045b72773e0dc27ac
+https://fatcat.wiki/file/4tcvwhzunrgvri4x3uruug62jq
+
+    cover page... an ILL request? via ILL request.
+
+    "metadata": {
+        "author": "Emmanuel Lemoine",
+        "creator": "Okina",
+        "producer": "mPDF 6.0",
+        "title": "Chloro complexes of cobalt(II) in aprotic solvents: stability and structural modifications due to solvent effect"
+    },
+    "pdf_created": "2017-01-26T10:43:21+00:00",
+    "pdf_version": "1.4",
+    "permanent_id": "2d231660c0e26f92aad7cb2f62b5e03a",
+
+    SELECT *
+    FROM pdf_meta
+    WHERE
+        status = 'success'
+        AND page_count < 3
+        AND (metadata->>'creator')::text = 'Okina'
+    LIMIT 5;
+
+    SELECT COUNT(*)
+    FROM pdf_meta
+    WHERE
+        status = 'success'
+        AND page_count < 3
+        AND (metadata->>'creator')::text = 'Okina'
+    ;
+    # 4235
+
+    TODO: 'COPY TO'...
+
+    SELECT pdf_meta.sha1hex
+    FROM pdf_meta
+    LEFT JOIN fatcat_file ON pdf_meta.sha1hex = fatcat_file.sha1hex
+    WHERE
+        status = 'success'
+        AND page_count < 3
+        AND (metadata->>'creator')::text = 'Okina'
+        AND (metadata->>'publisher')::text LIKE 'mPDF%'
+        AND fatcat_file.ident IS NOT NULL
+    ;
diff --git a/notes/misc/examples/grobid_500.txt b/notes/misc/examples/grobid_500.txt
new file mode 100644
index 00000000..5e64c781
--- /dev/null
+++ b/notes/misc/examples/grobid_500.txt
@@ -0,0 +1,4 @@
+
+seems like a legit/fine PDF file:
+https://fatcat.wiki/file/nrydu6nutvedximcb4lpdsrp6u
+
diff --git a/notes/misc/examples/personal_favorites.md b/notes/misc/examples/personal_favorites.md
new file mode 100644
index 00000000..2ecee2d8
--- /dev/null
+++ b/notes/misc/examples/personal_favorites.md
@@ -0,0 +1,2 @@
+
+International Journal of Crashworthiness
diff --git a/notes/misc/examples/random_journals.txt b/notes/misc/examples/random_journals.txt
new file mode 100644
index 00000000..f5cb0e69
--- /dev/null
+++ b/notes/misc/examples/random_journals.txt
@@ -0,0 +1,5 @@
+
+"Rejecta Mathematica"
+only published articles which failed peer review.
+no longer online, but may be in wayback
+https://en.wikipedia.org/wiki/Rejecta_Mathematica
diff --git a/notes/misc/examples/random_works.txt b/notes/misc/examples/random_works.txt
new file mode 100644
index 00000000..3f5bb7e3
--- /dev/null
+++ b/notes/misc/examples/random_works.txt
@@ -0,0 +1,9 @@
+
+"The limitations of using languages for description", Marvin Minsky
+http://web.mit.edu/dxh/www/1970_Marvin_Lecture_Transcript_Italy_Limitations_Language.pdf
+
+"A Supercut of Supercuts: Aesthetics, Histories, Databases"
+https://vimeo.com/440746435
+https://www.openscreensjournal.com/article/id/6946/
+
+Dummy article in springer (paywalled!): https://doi.org/10.1007/s10096-005-0027-5
diff --git a/notes/misc/examples/video_works.txt b/notes/misc/examples/video_works.txt
new file mode 100644
index 00000000..6c0a450f
--- /dev/null
+++ b/notes/misc/examples/video_works.txt
@@ -0,0 +1,4 @@
+
+https://doi.org/10.24350/cirm.v.19933803
+    "Imaging with nonlinear and fractionally damped waves"
+    https://library.cirm-math.fr/Record.htm?record=19280247124910084299&confirm=on
diff --git a/notes/misc/horror_stories.md b/notes/misc/horror_stories.md
new file mode 100644
index 00000000..eaac48e7
--- /dev/null
+++ b/notes/misc/horror_stories.md
@@ -0,0 +1,10 @@
+
+Two different DOIs for the same work, from different publishers:
+
+    Intravenous Administration of Human γ-Globulin
+    S. Barandun, P. Kistler, F. Jeunet, H. Isliker
+    1962, Vox Sanguinis
+
+    https://fatcat.wiki/release/search?q=%22Intravenous+administration+of+human+%CE%B3-globulin%22&generic=1
+    10.1111/j.1423-0410.1962.tb03240.x
+    10.1159/000464763
diff --git a/notes/rust_libraries.txt b/notes/misc/rust_libraries.txt
index d5c8c18a..d5c8c18a 100644
--- a/notes/rust_libraries.txt
+++ b/notes/misc/rust_libraries.txt
diff --git a/notes/test_works.txt b/notes/misc/test_works.txt
index 59b01701..59b01701 100644
--- a/notes/test_works.txt
+++ b/notes/misc/test_works.txt
diff --git a/notes/misc/thesis_uk.md b/notes/misc/thesis_uk.md
new file mode 100644
index 00000000..cbcca6d5
--- /dev/null
+++ b/notes/misc/thesis_uk.md
@@ -0,0 +1,6 @@
+
+large number of doctoral thesis metadata, from EThOS
+https://bl.iro.bl.uk/concern/datasets/c815b271-09be-4123-8156-405094429198
+
+will get via OAI-PMH, presumably. but, requires login for actual download?
+sigh.
diff --git a/notes/misc/unsorted.txt b/notes/misc/unsorted.txt
new file mode 100644
index 00000000..17ff839c
--- /dev/null
+++ b/notes/misc/unsorted.txt
@@ -0,0 +1,19 @@
+
+fatcat misc:
+- opencitations: https://arxiv.org/abs/1906.11964
+- https://pub.uni-bielefeld.de/record/2934907
+- re-read: scratch/issn/web_archiving.md
+- should expansion of 'wip' entities be allowed?
+- could now just not show 'wip' entities (unless part of editgroup)
+-  release_ref | 19904400 | Missing Index? |  4141039616 | 81833687 |  61929287
+- privacy/security issue with libmacaroon logging failed caveat verification
+- blank box on editgroup pages when not logged in
+- don't have "Editable catalog of bibliographic and fulltext file metadata" be the thing in snippets?
+- web: '|dictsort' in a bunch of places (for stability)
+- example HTML paper: https://andrewgyork.github.io/rescan_line_sted/
+- pubmed importer should include section in ALLCAPS: for multi-part abstracts
+- https://github.com/rholder/retrying
+- feature: push-button "update metadata from crossref"
+- demo ORCID: 0000-0002-1825-0097
+- link: https://www.jstor.org/dfr/about/technical-specifications
+- after indexing, optimise the Elasticsearch index by merging into a single segment: curl -XPOST 'http://localhost:9200/scholar/_forcemerge?max_num_segments=1'
diff --git a/notes/misc/webface_iteration.md b/notes/misc/webface_iteration.md
new file mode 100644
index 00000000..a7f11d15
--- /dev/null
+++ b/notes/misc/webface_iteration.md
@@ -0,0 +1,14 @@
+
+## Design Examples
+
+metamath
+
+- example: <https://wapm.io/package/liftm/metamath>
+- somewhat similar to existing fatcat release layout
+- tabs are better? tabs scroll left/right on mobile
+- CSS/etc is heavy, though design is simple
+
+lib.rs
+
+sourcehut
+
diff --git a/notes/webface_notes.txt b/notes/misc/webface_notes.txt
index 37a56c5c..37a56c5c 100644
--- a/notes/webface_notes.txt
+++ b/notes/misc/webface_notes.txt