datacite: resolve formatting issues in tests

4 files changed, 88 insertions, 0 deletions

diff --git a/notes/bulk_edits/2020-03-23_jalc.md b/notes/bulk_edits/2020-03-23_jalc.md
new file mode 100644
index 00000000..d63c3759
--- /dev/null
+++ b/notes/bulk_edits/2020-03-23_jalc.md
@@ -0,0 +1,23 @@
+
+2019-10-01 JaLC metadata snapshot: <https://archive.org/download/jalc-bulk-metadata-2019>
+
+Extracted .rdf file instead of piping it through zcat.
+
+Use correct bot:
+
+    export FATCAT_AUTH_WORKER_JALC=blah
+
+Start small; do a random bunch (10k) single-threaded to pre-create containers:
+
+    head -n100 /srv/fatcat/datasets/JALC-LOD-20191001.rdf | ./fatcat_import.py --batch-size 100 jalc - /srv/fatcat/datasets/ISSN-to-ISSN-L.txt
+    shuf -n100 /srv/fatcat/datasets/JALC-LOD-20191001.rdf | ./fatcat_import.py --batch-size 100 jalc - /srv/fatcat/datasets/ISSN-to-ISSN-L.txt
+    shuf -n10000 /srv/fatcat/datasets/JALC-LOD-20191001.rdf | ./fatcat_import.py --batch-size 100 jalc - /srv/fatcat/datasets/ISSN-to-ISSN-L.txt
+
+Seemed like lots of individual containers getting added after repeating, so
+just going to import single-threaded to avoid duplicate container creation:
+
+    cat /srv/fatcat/datasets/JALC-LOD-20191001.rdf | ./fatcat_import.py --batch-size 100 jalc - /srv/fatcat/datasets/ISSN-to-ISSN-L.txt
+    => Counter({'total': 8419745, 'exists': 6480683, 'insert': 1934082, 'skip': 4980, 'inserted.container': 134, 'update': 0})
+
+Had a bit fewer than 4,568,120 "doi_registrar:jalc" releases before this
+import, 6,502,202 after (based on `doi_registrar:jalc` query).
diff --git a/notes/cleanup_tasks.txt b/notes/cleanup_tasks.txt
new file mode 100644
index 00000000..bf418e59
--- /dev/null
+++ b/notes/cleanup_tasks.txt
@@ -0,0 +1,18 @@
+
+Cambridge Chemical Database (NCI)
+
+    doi_prefix:10.3406 release_type:article
+
+    193,346+ entities
+
+    should be 'dataset' not 'article'
+
+    datacite importer
+
+Frontiers
+
+    Frontiers non-PDF abstracts, which have DOIs like `10.3389/conf.*`. Should
+    crawl these, but `release_type` should be... `abstract`? There are at least
+    18,743 of these. Should be fixed in both crossref-bot, then a retro-active
+    cleanup.
+
diff --git a/notes/example_entities.txt b/notes/example_entities.txt
new file mode 100644
index 00000000..416da610
--- /dev/null
+++ b/notes/example_entities.txt
@@ -0,0 +1,26 @@
+
+errata/update:
+    Fourth Test of General Relativity: Preliminary Results
+    10.1103/physrevlett.20.1265
+    10.1103/physrevlett.21.266.3 
+
+    same title; later is errata to the first.
+    very minor: The term "baud length" was consistently misprinted as "band length."
+
+DOIs for individual images
+    https://commons.wikimedia.org/wiki/Category:Media_from_Williams_et_al._2010_-_10.1371/journal.pone.0010676
+
+long-tail journal not in fatcat; web-native, tricky to crawl
+    https://angryoldmanmagazine.com/
+
+dataset
+    "ISSN-Matching of Gold OA Journals (ISSN-GOLD-OA) 2.0"
+    https://pub.uni-bielefeld.de/data/2913654
+    2 files
+    has DOI: 10.4119/unibi/2913654
+
+release group; single PDF is valid copy of two DOIs:
+    https://fatcat.wiki/file/wr64e37yvfcidgbowtslx7omne
+    10.5167/uzh-146424
+    10.1016/j.physletb.2017.12.006
+    ALSO: has CC-BY license_slug
diff --git a/notes/merge_releases_examples.txt b/notes/merge_releases_examples.txt
new file mode 100644
index 00000000..ca65705e
--- /dev/null
+++ b/notes/merge_releases_examples.txt
@@ -0,0 +1,21 @@
+
+https://fatcat.wiki/release/search?q=Validation+of+middle-atmospheric+campaign-based+water+vapour+measured+by+the+ground-based+microwave+radiometer
+
+    4 releases, all dois. 3x have same author list, 1 same authors different order
+
+https://fatcat.wiki/release/search?q=Perspectives+and+pregnancy+outcomes+of+maternal+Ramadan+fasting+in+the+second+trimester+of+pregnancy
+
+    6 releases:
+        2 figshare article
+        2 figshare files
+        1 primary
+        1 correction
+
+https://figshare.com/articles/Plasmodium_falciparum_evades_innate_immunity_by_hybrid_ABO_blood_group_phenotype_formation/8208689/119
+
+    119 versions (!)
+
+https://fatcat.wiki/release/search?q=NeuroTrends+Visualization
+
+    45 versions across two figshare works 
+