From 4ab53ddfeef8fa99f5cf507f582c224a32e4c8b9 Mon Sep 17 00:00:00 2001 From: Martin Czygan Date: Thu, 27 Aug 2020 16:52:13 +0200 Subject: update project README --- projects/oai_harvest_md/.gitignore | 1 + projects/oai_harvest_md/Makefile | 5 +++++ projects/oai_harvest_md/README.md | 10 ++++++++++ 3 files changed, 16 insertions(+) create mode 100644 projects/oai_harvest_md/.gitignore create mode 100644 projects/oai_harvest_md/Makefile (limited to 'projects/oai_harvest_md') diff --git a/projects/oai_harvest_md/.gitignore b/projects/oai_harvest_md/.gitignore new file mode 100644 index 0000000..96077ed --- /dev/null +++ b/projects/oai_harvest_md/.gitignore @@ -0,0 +1 @@ +oai.ndjson.zst diff --git a/projects/oai_harvest_md/Makefile b/projects/oai_harvest_md/Makefile new file mode 100644 index 0000000..f9deb7f --- /dev/null +++ b/projects/oai_harvest_md/Makefile @@ -0,0 +1,5 @@ +SHELL := /bin/bash + +oai_harvest_20200215.ndjson.zst: + wget -c https://archive.org/download/oai_harvest_20200215/oai.ndjson.zst + diff --git a/projects/oai_harvest_md/README.md b/projects/oai_harvest_md/README.md index bbaa915..5f2b655 100644 --- a/projects/oai_harvest_md/README.md +++ b/projects/oai_harvest_md/README.md @@ -1,5 +1,7 @@ # OAI metadata matching +Goal: end-to-end data workflow (acquisition, harvest, matching, new release entities). + ## Plan * [ ] get JSON version, via [oai_harvest_20200215](https://archive.org/details/oai_harvest_20200215) @@ -8,3 +10,11 @@ * [ ] (b) for items w/o doi, get a list of (id, title) * [ ] run fuzzy matching over title list to find out which one we have +## Get data + +``` +$ make +``` + +* compressed 12G, around 100G uncompressed + -- cgit v1.2.3