summaryrefslogtreecommitdiffstats
path: root/python/fatcat_import.py
Commit message (Collapse)AuthorAgeFilesLines
* improve datacite field mapping and importMartin Czygan2019-12-281-1/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current version succeeded to import a random sample of 100000 records (0.5%) from datacite. The --debug (write JSON to stdout) and --insert-log-file (log batch before committing to db) flags are temporary added to help debugging. Add few unit tests. Some edge cases: a) Existing keys without value requires a slightly awkward: ``` titles = attributes.get('titles', []) or [] ``` b) There can be 0, 1, or more (first one wins) titles. c) Date handling is probably not ideal. Datacite has a potentiall fine grained list of dates. The test case (tests/files/datacite_sample.jsonl) refers to https://ssl.fao.org/glis/doi/10.18730/8DYM9, which has date (main descriptor) 1986. The datacite record contains: 2017 (publicationYear, probably the year of record creation with reference system), 1978-06-03 (collected, e.g. experimental sample), 1986 ("Accepted"). The online version of the resource knows even one more date (2019-06-05 10:14:43 by WIEWS update).
* datacite: importer skeletonMartin Czygan2019-12-281-0/+30
| | | | | | * contributors, title, date, publisher, container, license Field and value analysis via https://github.com/miku/indigo.
* savepapernow result importerBryan Newbold2019-12-121-0/+24
| | | | Based on ingest-file-results importer
* improve argparse usageBryan Newbold2019-12-111-18/+30
| | | | | | | | | | | | | | Use --fatcat-api-url instead of (ambiguous) --host-url for commands that aren't deployed/running via systemd. TODO: update the other --host-url usage, and either roll-out change consistently or support the old arg as an alias during cut-over Use argparse.ArgumentDefaultsHelpFormatter (thanks Martin!) Add help messages for all sub-commands, both as documentation and as a way to get argparse to print available commands in a more readable format.
* tweaks to file ingest importerBryan Newbold2019-12-031-0/+6
| | | | | - allow overriding source filter whitelist (common case for CLI use) - fix editgroup description env variable pass-through
* have ingest-file-results importer operate as crawl-botBryan Newbold2019-11-151-1/+1
| | | | As opposed to sandcrawler-bot
* better ingest-file-results import nameBryan Newbold2019-11-151-1/+1
|
* ingest file result importerBryan Newbold2019-11-151-0/+34
|
* small fixes to confluent-kafka importers/workersBryan Newbold2019-09-201-1/+1
| | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again)
* convert importers to confluent-kafka libraryBryan Newbold2019-09-201-2/+3
|
* start chocula importerBryan Newbold2019-09-031-0/+14
|
* support extids in matched importerBryan Newbold2019-06-201-0/+4
|
* faster LargeFile XML importer for PubMedBryan Newbold2019-05-291-1/+1
|
* make pubmed ref lookups configurableBryan Newbold2019-05-221-1/+8
|
* creative importer for bulk JSTOR importsBryan Newbold2019-05-221-0/+18
|
* pubmed importer command and tweaksBryan Newbold2019-05-221-0/+25
|
* arxiv importer robustification and CLI implBryan Newbold2019-05-211-0/+21
|
* JALC bulk file importerBryan Newbold2019-05-211-0/+21
|
* fix default mimetype (impacted pre-1923 files)Bryan Newbold2019-05-151-1/+5
|
* editgroup description overrideBryan Newbold2019-04-221-1/+11
|
* minor arabesque tweaksBryan Newbold2019-04-181-12/+22
|
* arabesque importer using crawl-bot credsBryan Newbold2019-04-181-1/+1
|
* arabesque import tweaksBryan Newbold2019-04-181-0/+4
|
* early version of arabesque importerBryan Newbold2019-04-121-0/+28
|
* importer for CDL/DASH dat pilot dweb datasetsBryan Newbold2019-03-191-1/+29
|
* new importer: wayback_staticBryan Newbold2019-03-191-0/+48
|
* reduce default import batch size to 50Bryan Newbold2019-01-291-1/+1
|
* batch size as a general import paramBryan Newbold2019-01-281-13/+4
|
* add missing bezerk-mode flag to GROBID importBryan Newbold2019-01-281-3/+8
|
* fix typo in crossref importerBryan Newbold2019-01-281-1/+1
|
* update journal meta import/transformBryan Newbold2019-01-251-3/+3
|
* more import script fixesBryan Newbold2019-01-231-1/+4
|
* update importer scriptBryan Newbold2019-01-231-33/+24
|
* pubmed+datacite tokens; no journal,grobid,matched tokensBryan Newbold2019-01-221-2/+2
|
* issn => journal_metadata in several placesBryan Newbold2019-01-171-9/+9
|
* start refactoring API object passingBryan Newbold2019-01-081-13/+36
|
* crossref importer checks for existing DOIsBryan Newbold2018-11-211-3/+7
|
* correct kafka topic namesBryan Newbold2018-11-201-1/+1
|
* start supporting kafka importersBryan Newbold2018-11-191-3/+17
| | | | A nice feature would be some/any log output as to progress.
* bunch of pylint cleanupBryan Newbold2018-11-151-1/+1
|
* large refactor of python names/pathsBryan Newbold2018-11-151-39/+37
| | | | | | | - Add __init__.py files for fatcat_tools submodules, and use them in imports - Add a bunch of comments to files. - rename a number of classes and functions to be less verbose
* shuffle around fatcat_tools layoutBryan Newbold2018-11-131-5/+5
|
* more python module refactoringBryan Newbold2018-11-121-5/+5
|
* remove more old python cruftBryan Newbold2018-11-121-8/+0
|
* fixes for grobid metadata importerBryan Newbold2018-09-281-0/+15
|
* fix issues with extid mapping in crossref-importerBryan Newbold2018-09-201-1/+1
|
* switch manifest importer to be json-basedBryan Newbold2018-09-141-16/+2
|
* add insert counting to importersBryan Newbold2018-09-141-0/+22
|
* extid support for crossref importerBryan Newbold2018-09-121-2/+5
|
* rename python scriptsBryan Newbold2018-07-261-0/+94
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-30 16:41:47 +0000