From 403b1a2d4591d878145a021a7c1e15e2d60c47d8 Mon Sep 17 00:00:00 2001 From: Martin Czygan Date: Wed, 18 Dec 2019 20:21:49 +0100 Subject: improve datacite field mapping and import Current version succeeded to import a random sample of 100000 records (0.5%) from datacite. The --debug (write JSON to stdout) and --insert-log-file (log batch before committing to db) flags are temporary added to help debugging. Add few unit tests. Some edge cases: a) Existing keys without value requires a slightly awkward: ``` titles = attributes.get('titles', []) or [] ``` b) There can be 0, 1, or more (first one wins) titles. c) Date handling is probably not ideal. Datacite has a potentiall fine grained list of dates. The test case (tests/files/datacite_sample.jsonl) refers to https://ssl.fao.org/glis/doi/10.18730/8DYM9, which has date (main descriptor) 1986. The datacite record contains: 2017 (publicationYear, probably the year of record creation with reference system), 1978-06-03 (collected, e.g. experimental sample), 1986 ("Accepted"). The online version of the resource knows even one more date (2019-06-05 10:14:43 by WIEWS update). --- python/tests/files/datacite_1k_records.jsonl.gz | Bin 0 -> 684605 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 python/tests/files/datacite_1k_records.jsonl.gz (limited to 'python/tests/files/datacite_1k_records.jsonl.gz') diff --git a/python/tests/files/datacite_1k_records.jsonl.gz b/python/tests/files/datacite_1k_records.jsonl.gz new file mode 100644 index 00000000..28ea6e37 Binary files /dev/null and b/python/tests/files/datacite_1k_records.jsonl.gz differ -- cgit v1.2.3