summaryrefslogtreecommitdiffstats
path: root/python/fatcat_import.py
diff options
context:
space:
mode:
authorMartin Czygan <martin.czygan@gmail.com>2019-12-18 20:21:49 +0100
committerMartin Czygan <martin.czygan@gmail.com>2019-12-28 23:07:31 +0100
commit403b1a2d4591d878145a021a7c1e15e2d60c47d8 (patch)
tree082ddd601a58b25be4ee176fdda97f935e23ea4b /python/fatcat_import.py
parent76d6d4d2de6580ae147e40c43c18f04cc48b62ec (diff)
downloadfatcat-403b1a2d4591d878145a021a7c1e15e2d60c47d8.tar.gz
fatcat-403b1a2d4591d878145a021a7c1e15e2d60c47d8.zip
improve datacite field mapping and import
Current version succeeded to import a random sample of 100000 records (0.5%) from datacite. The --debug (write JSON to stdout) and --insert-log-file (log batch before committing to db) flags are temporary added to help debugging. Add few unit tests. Some edge cases: a) Existing keys without value requires a slightly awkward: ``` titles = attributes.get('titles', []) or [] ``` b) There can be 0, 1, or more (first one wins) titles. c) Date handling is probably not ideal. Datacite has a potentiall fine grained list of dates. The test case (tests/files/datacite_sample.jsonl) refers to https://ssl.fao.org/glis/doi/10.18730/8DYM9, which has date (main descriptor) 1986. The datacite record contains: 2017 (publicationYear, probably the year of record creation with reference system), 1978-06-03 (collected, e.g. experimental sample), 1986 ("Accepted"). The online version of the resource knows even one more date (2019-06-05 10:14:43 by WIEWS update).
Diffstat (limited to 'python/fatcat_import.py')
-rwxr-xr-xpython/fatcat_import.py15
1 files changed, 14 insertions, 1 deletions
diff --git a/python/fatcat_import.py b/python/fatcat_import.py
index d7651792..90bb01a1 100755
--- a/python/fatcat_import.py
+++ b/python/fatcat_import.py
@@ -170,7 +170,10 @@ def run_datacite(args):
dci = DataciteImporter(args.api,
args.issn_map_file,
edit_batch_size=args.batch_size,
- bezerk_mode=args.bezerk_mode)
+ bezerk_mode=args.bezerk_mode,
+ debug=args.debug,
+ lang_detect=args.lang_detect,
+ insert_log_file=args.insert_log_file)
if args.kafka_mode:
KafkaJsonPusher(fci, args.kafka_hosts, args.kafka_env, "api-datacite",
"fatcat-import", consume_batch_size=args.batch_size).run()
@@ -464,6 +467,16 @@ def main():
sub_datacite.add_argument('--bezerk-mode',
action='store_true',
help="don't lookup existing DOIs, just insert (clobbers; only for fast bootstrap)")
+ sub_datacite.add_argument('--debug',
+ action='store_true',
+ help="write converted JSON to stdout")
+ sub_datacite.add_argument('--lang-detect',
+ action='store_true',
+ help="try to detect language (slow)")
+ sub_datacite.add_argument('--insert-log-file',
+ default='',
+ type=str,
+ help="write inserted documents into file (for debugging)")
sub_datacite.set_defaults(
func=run_datacite,
auth_var="FATCAT_API_AUTH_TOKEN",