diff options
author | Martin Czygan <martin.czygan@gmail.com> | 2019-12-18 20:21:49 +0100 |
---|---|---|
committer | Martin Czygan <martin.czygan@gmail.com> | 2019-12-28 23:07:31 +0100 |
commit | 403b1a2d4591d878145a021a7c1e15e2d60c47d8 (patch) | |
tree | 082ddd601a58b25be4ee176fdda97f935e23ea4b /python/tests/transform_ingest.py | |
parent | 76d6d4d2de6580ae147e40c43c18f04cc48b62ec (diff) | |
download | fatcat-403b1a2d4591d878145a021a7c1e15e2d60c47d8.tar.gz fatcat-403b1a2d4591d878145a021a7c1e15e2d60c47d8.zip |
improve datacite field mapping and import
Current version succeeded to import a random sample of 100000 records
(0.5%) from datacite.
The --debug (write JSON to stdout) and --insert-log-file (log batch
before committing to db) flags are temporary added to help debugging.
Add few unit tests.
Some edge cases:
a) Existing keys without value requires a slightly awkward:
```
titles = attributes.get('titles', []) or []
```
b) There can be 0, 1, or more (first one wins) titles.
c) Date handling is probably not ideal. Datacite has a potentiall fine
grained list of dates.
The test case (tests/files/datacite_sample.jsonl) refers to
https://ssl.fao.org/glis/doi/10.18730/8DYM9, which has date (main
descriptor) 1986. The datacite record contains: 2017 (publicationYear,
probably the year of record creation with reference system), 1978-06-03
(collected, e.g. experimental sample), 1986 ("Accepted"). The online
version of the resource knows even one more date (2019-06-05 10:14:43 by
WIEWS update).
Diffstat (limited to 'python/tests/transform_ingest.py')
0 files changed, 0 insertions, 0 deletions