summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/importers
Commit message (Collapse)AuthorAgeFilesLines
* datacite: address raw_name index form commentMartin Czygan2020-01-021-0/+43
| | | | | | | | | > The convention for display_name and raw_name is to be how the name would normally be printed, not in index form (surname comma given_name). So we might need to un-encode names like "Tricart, Pierre". Use an additional `index_form_to_display_name` function to convert index from to display form, heuristically.
* datacite: add two more skipable tokensMartin Czygan2020-01-021-1/+1
|
* datacite: names can be 'Unav', tooMartin Czygan2020-01-021-1/+4
|
* datacite: avoid more None valuesMartin Czygan2020-01-011-4/+4
|
* datacite: address 'Unpublished' publisherMartin Czygan2019-12-311-9/+10
|
* datacite: ensure name schema is definedMartin Czygan2019-12-311-1/+2
|
* datacite: fix typoMartin Czygan2019-12-311-1/+1
|
* datacite: isascii was added in 3.7, onlyMartin Czygan2019-12-311-1/+7
|
* datacite: skip non-ascii doi for nowMartin Czygan2019-12-311-0/+4
| | | | | | Example of a non-ascii doi: * https://doi.org/10.13125/américacrítica/3017
* datacite: clean doiMartin Czygan2019-12-311-1/+13
| | | | | | | address issue with EN DASH DOI. > "external identifier doesn't match required pattern for a DOI (expected, eg, '10.1234/aksjdfh'): 10.25513/1812-3996.2017.1.34–42"
* datacite: update docsMartin Czygan2019-12-311-9/+9
|
* datacite: perform additional checks on contribMartin Czygan2019-12-301-3/+9
|
* datacite: check for empty title after cleanMartin Czygan2019-12-291-2/+5
|
* datacite: update docs with observed valuesMartin Czygan2019-12-291-1/+3
|
* datacite: page number misses are too commonMartin Czygan2019-12-281-1/+2
| | | | | | Should be a level debug, not info. Examples: E675, n/a, 15D.2.1, 15D.2.1, A.1E.1, A.1E.1, ...
* datacite: suppress debug-like language lookup miss messageMartin Czygan2019-12-281-1/+3
|
* datacite: treat untyped names as peopleMartin Czygan2019-12-281-1/+1
|
* datacite: include container_name top level key in extraMartin Czygan2019-12-281-7/+21
|
* datacite: use clean on field valuesMartin Czygan2019-12-281-2/+28
|
* datacite: include doi in error messagesMartin Czygan2019-12-281-8/+8
|
* datacite: limit abstract lengthMartin Czygan2019-12-281-0/+6
|
* datacite: use iso 639-1 codesMartin Czygan2019-12-281-7/+4
|
* address first round of MR14 commentsMartin Czygan2019-12-281-148/+319
| | | | | | | | | | | | | * add missing langdetect * use entity_to_dict for json debug output * factor out code for fields in function and add table driven tests * update citeproc types * add author as default role * add raw_affiliation * include relations from datacite * remove url (covered by doi already) Using yapf for python formatting.
* datacite: move common date patterns out of the loopMartin Czygan2019-12-281-3/+4
| | | | Additionally, try the unspecific (%Y) pattern last.
* improve datacite field mapping and importMartin Czygan2019-12-281-41/+139
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current version succeeded to import a random sample of 100000 records (0.5%) from datacite. The --debug (write JSON to stdout) and --insert-log-file (log batch before committing to db) flags are temporary added to help debugging. Add few unit tests. Some edge cases: a) Existing keys without value requires a slightly awkward: ``` titles = attributes.get('titles', []) or [] ``` b) There can be 0, 1, or more (first one wins) titles. c) Date handling is probably not ideal. Datacite has a potentiall fine grained list of dates. The test case (tests/files/datacite_sample.jsonl) refers to https://ssl.fao.org/glis/doi/10.18730/8DYM9, which has date (main descriptor) 1986. The datacite record contains: 2017 (publicationYear, probably the year of record creation with reference system), 1978-06-03 (collected, e.g. experimental sample), 1986 ("Accepted"). The online version of the resource knows even one more date (2019-06-05 10:14:43 by WIEWS update).
* datacite: add missing mappings and notesMartin Czygan2019-12-281-266/+175
|
* datacite: basic field mappingsMartin Czygan2019-12-281-41/+181
| | | | | | | | | | Currently using two external libraries: * dateparser * langcodes Note: This commit includes lots of wip docs and field stat in comment, which should be removed.
* datacite: importer skeletonMartin Czygan2019-12-282-0/+459
| | | | | | * contributors, title, date, publisher, container, license Field and value analysis via https://github.com/miku/indigo.
* orcid: skip non-person ORCID recordsBryan Newbold2019-12-261-0/+4
|
* allow arabesque backfill ingests for some source typesBryan Newbold2019-12-241-0/+5
|
* make chocula URL updates more conservativeBryan Newbold2019-12-241-5/+5
|
* pubmed: if doing update, also do subtitle schema updateBryan Newbold2019-12-231-1/+9
|
* pubmed: improve warning and stderr formattingBryan Newbold2019-12-231-5/+6
|
* pubmed: use standard identifier cleanersBryan Newbold2019-12-231-17/+14
|
* pubmed: remove unused extid mapping codeBryan Newbold2019-12-231-29/+0
|
* pubmed: do reference lookups by defaultBryan Newbold2019-12-231-1/+1
|
* pubmed: null doi parsing checkBryan Newbold2019-12-231-1/+1
|
* add basic MedlineDate year parsingBryan Newbold2019-12-231-0/+11
|
* fix spn/ingest importer duplication checkBryan Newbold2019-12-221-6/+8
| | | | | | Check was happing after the `return True` by mistake, allowing duplicates in SPN editgroups, and potentially in ingest request editgroups as well.
* write diagnostic messages to stderrMartin Czygan2019-12-161-2/+2
| | | | | During debugging, it can be helpful to keep stdout (e.g. processing results) and dignostic messages separate.
* Merge branch 'martin-importers-common-doc-fix' into 'master'Martin Czygan2019-12-141-13/+10
|\ | | | | | | | | Update EntityImporter docstring. See merge request webgroup/fatcat!9
| * complete parse_record docstringMartin Czygan2019-12-141-0/+6
| |
| * Update EntityImporter docstring.Martin Czygan2019-12-131-13/+4
| | | | | | | | I believe the required method is `parse_record`, not `parse`.
* | add ingest import file collision protectionBryan Newbold2019-12-131-0/+6
| | | | | | | | | | | | | | | | The common case is the same URL being submitted repeatedly during testing. This is only within-editgroup, and per importer (eg, won't work across spn importer "submitted" editgroups), but is better than nothing.
* | update ingest request schemaBryan Newbold2019-12-131-2/+7
| | | | | | | | | | This is mostly changing ingest_type from 'file' to 'pdf', and adding 'link_source'/'link_source_id', plus some small cleanups.
* | remove default mimetype from ingest-file importerBryan Newbold2019-12-131-2/+1
| | | | | | | | We really should just use file_meta result or nothing.
* | revert accidentally commited test timingBryan Newbold2019-12-131-2/+2
| | | | | | | | Also fix a spurious typo.
* | ensure importer description arg isn't clobberedBryan Newbold2019-12-123-5/+5
| |
* | savepapernow result importerBryan Newbold2019-12-122-4/+65
| | | | | | | | Based on ingest-file-results importer
* | flush importer editgroups every few minutesBryan Newbold2019-12-121-5/+20
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-28 07:03:59 +0000