aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
-rw-r--r--README.md56
1 files changed, 35 insertions, 21 deletions
diff --git a/README.md b/README.md
index dec3262..a4b48ea 100644
--- a/README.md
+++ b/README.md
@@ -1,12 +1,42 @@
-This folder contains scripts to merge journal metadat from multiple sources and
-provide a snapshot for bulk importing into fatcat.
+Chocula is a python script to parse and merge journal-level metadata from
+various sources into a consistent sqlite3 database file for analysis.
-Specific bots will probably be needed to do continous updates; that's out of
-scope for this first import.
+See `chocula_schema.sql` for the output schema.
+## History / Name
-## Sources
+This is the 3rd or 4th iteration of open access journal metadata munging as
+part of the fatcat project; earlier attempts were crude ISSN spreadsheet
+munging, then the `oa-journals-analysis` repo (Jupyter notebook and a web
+interface), then the `fatcat:extra/journal_metadata/` script for bootstrapping
+fatcat container metadata. This repo started as the fatcat `journal_metadata`
+directory and retains the git history of that folder.
+
+The name "chocula" comes from a half-baked pun on Count Chocula... something
+something counting, serials, cereal.
+[Read more about Count Chocula](https://teamyacht.com/ernstchoukula.com/Ernst-Choukula.html).
+
+## ISSN-L Munging
+
+Unfortunately, there seem to be plenty of legitimate ISSNs that don't end up in
+the ISSN-L table. On the portal.issn.org public site, these are listed as:
+
+ "This provisional record has been produced before publication of the
+ resource. The published resource has not yet been checked by the ISSN
+ Network.It is only available to subscribing users."
+
+For example:
+
+- 2199-3246/2199-3254: Digital Experiences in Mathematics Education
+
+Previously these were allowed through into fatcat, so some 2000+ entries exist.
+This allowed through at least 110 totally bogus ISSNs. Currently, chocula
+filters out "unknown" ISSN-Ls unless they are coming from existing fatcat
+entities.
+
+
+## Sources (out of date)
The `./data/fetch.sh` script will fetch mirrored snapshots of all these
datasets.
@@ -60,22 +90,6 @@ ISSN-L, then write out to disk as JSON. Then the journal-metadata importer
takes a subset of fields and inserts to fatcat. Lastly, the elasticsearch
transformer takes a subset/combination of
-## ISSN-L Munging
-
-Unfortunately, there seem to be plenty of legitimate ISSNs that don't end up in
-the ISSN-L table. On the portal.issn.org public site, these are listed as:
-
- "This provisional record has been produced before publication of the
- resource. The published resource has not yet been checked by the ISSN
- Network.It is only available to subscribing users."
-
-For example:
-
-- 2199-3246/2199-3254: Digital Experiences in Mathematics Education
-
-Instead of just dropping these entirely, we're currently munging these by
-putting the electronic or print ISSN in the ISSN-L position.
-
## Python Helpers/Libraries
- ftfy