summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* pytest: explicitly indicate all in-scope test filesBryan Newbold2020-01-041-3/+1
| | | | | | | | | | | The purpose of this change is to test errors when pytest tries to recursively update assertion statements in all dependent packages. The reason pytest does this is to add pretty printing, which is nice, but probably shouldn't be done in all dependency libraries. This fixes test problems with both CSL (citeproc_styles) and dateparser (when actually imported in code, which currently on master does not happen).
* datacite: always include "datacite" key in extraMartin Czygan2020-01-0415-28/+28
| | | | | | > always include extra values for the respective DOI registrars (datacite, crossref, jalc), even if they are empty ({}), to be used as a flag so we know which DOI registrar supplied the metadata.
* datacite: use normal.clean_doiMartin Czygan2020-01-032-15/+1
|
* datacite: parse_datacite_dates returns monthMartin Czygan2020-01-032-17/+51
| | | | As [...] we will soon add support for release_month field in the release schema.
* datacite: prepare release_month (stub)Martin Czygan2020-01-032-24/+24
|
* datacite: lowercase only onceMartin Czygan2020-01-031-3/+4
|
* add pycountry dependencyMartin Czygan2020-01-032-1/+9
|
* add missing pathlib2 dependencyMartin Czygan2020-01-032-1/+18
| | | | | first seen in CI (jobs/230137), slightly related: https://github.com/pytest-dev/pytest/issues/3953
* update potentially outdated Pipfile.lockMartin Czygan2020-01-031-96/+86
| | | | | | | | via: $ pipenv lock CI complained with a slightly cryptic: > TypeError: __init__() missing 1 required positional argument: 'self'
* datacite: remove --lang-detect flagMartin Czygan2020-01-037-25/+21
| | | | Estimated time for a single call is in the order of 50ms.
* datacite: add another test caseMartin Czygan2020-01-023-1/+71
|
* datacite: open case for editing after creationMartin Czygan2020-01-021-0/+2
|
* datacite: add helper script to create new test caseMartin Czygan2020-01-021-0/+14
|
* datacite: address raw_name index form commentMartin Czygan2020-01-0221-112/+171
| | | | | | | | | > The convention for display_name and raw_name is to be how the name would normally be printed, not in index form (surname comma given_name). So we might need to un-encode names like "Tricart, Pierre". Use an additional `index_form_to_display_name` function to convert index from to display form, heuristically.
* datacite: add two more skipable tokensMartin Czygan2020-01-021-1/+1
|
* datacite: add conversion fixturesMartin Czygan2020-01-0250-1/+3949
| | | | | | | | | | | | | The `test_datacite_conversions` function will compare an input (datacite) document to an expected output (release entity as JSON). This way, it should not be too hard to add more cases by adding: input, output - and by increasing the counter in the range loop within the test. To view input and result side by side with vim, change into the test directory and run: tests/files/datacite $ ./caseview.sh 18
* datacite: names can be 'Unav', tooMartin Czygan2020-01-021-1/+4
|
* datacite: avoid more None valuesMartin Czygan2020-01-011-4/+4
|
* datacite: address 'Unpublished' publisherMartin Czygan2019-12-311-9/+10
|
* datacite: ensure name schema is definedMartin Czygan2019-12-311-1/+2
|
* datacite: fix typoMartin Czygan2019-12-311-1/+1
|
* datacite: isascii was added in 3.7, onlyMartin Czygan2019-12-311-1/+7
|
* datacite: skip non-ascii doi for nowMartin Czygan2019-12-311-0/+4
| | | | | | Example of a non-ascii doi: * https://doi.org/10.13125/américacrítica/3017
* datacite: clean doiMartin Czygan2019-12-311-1/+13
| | | | | | | address issue with EN DASH DOI. > "external identifier doesn't match required pattern for a DOI (expected, eg, '10.1234/aksjdfh'): 10.25513/1812-3996.2017.1.34–42"
* datacite: update docsMartin Czygan2019-12-311-9/+9
|
* datacite: perform additional checks on contribMartin Czygan2019-12-301-3/+9
|
* datacite: check for empty title after cleanMartin Czygan2019-12-291-2/+5
|
* datacite: update docs with observed valuesMartin Czygan2019-12-291-1/+3
|
* datacite: page number misses are too commonMartin Czygan2019-12-281-1/+2
| | | | | | Should be a level debug, not info. Examples: E675, n/a, 15D.2.1, 15D.2.1, A.1E.1, A.1E.1, ...
* datacite: suppress debug-like language lookup miss messageMartin Czygan2019-12-281-1/+3
|
* datacite: adjust testsMartin Czygan2019-12-281-2/+1
|
* datacite: treat untyped names as peopleMartin Czygan2019-12-281-1/+1
|
* datacite: include container_name top level key in extraMartin Czygan2019-12-281-7/+21
|
* datacite: use clean on field valuesMartin Czygan2019-12-281-2/+28
|
* datacite: include doi in error messagesMartin Czygan2019-12-281-8/+8
|
* remove langcodes dependencyMartin Czygan2019-12-282-15/+0
|
* datacite: limit abstract lengthMartin Czygan2019-12-281-0/+6
|
* datacite: use iso 639-1 codesMartin Czygan2019-12-281-7/+4
|
* datacite: use specific auth varMartin Czygan2019-12-281-1/+1
|
* datacite: add missing --extid-map-file flagMartin Czygan2019-12-281-0/+4
|
* address first round of MR14 commentsMartin Czygan2019-12-284-150/+503
| | | | | | | | | | | | | * add missing langdetect * use entity_to_dict for json debug output * factor out code for fields in function and add table driven tests * update citeproc types * add author as default role * add raw_affiliation * include relations from datacite * remove url (covered by doi already) Using yapf for python formatting.
* datacite: move common date patterns out of the loopMartin Czygan2019-12-281-3/+4
| | | | Additionally, try the unspecific (%Y) pattern last.
* improve datacite field mapping and importMartin Czygan2019-12-285-59/+245
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current version succeeded to import a random sample of 100000 records (0.5%) from datacite. The --debug (write JSON to stdout) and --insert-log-file (log batch before committing to db) flags are temporary added to help debugging. Add few unit tests. Some edge cases: a) Existing keys without value requires a slightly awkward: ``` titles = attributes.get('titles', []) or [] ``` b) There can be 0, 1, or more (first one wins) titles. c) Date handling is probably not ideal. Datacite has a potentiall fine grained list of dates. The test case (tests/files/datacite_sample.jsonl) refers to https://ssl.fao.org/glis/doi/10.18730/8DYM9, which has date (main descriptor) 1986. The datacite record contains: 2017 (publicationYear, probably the year of record creation with reference system), 1978-06-03 (collected, e.g. experimental sample), 1986 ("Accepted"). The online version of the resource knows even one more date (2019-06-05 10:14:43 by WIEWS update).
* datacite: add missing mappings and notesMartin Czygan2019-12-281-266/+175
|
* datacite: basic field mappingsMartin Czygan2019-12-281-41/+181
| | | | | | | | | | Currently using two external libraries: * dateparser * langcodes Note: This commit includes lots of wip docs and field stat in comment, which should be removed.
* datacite: importer skeletonMartin Czygan2019-12-284-0/+514
| | | | | | * contributors, title, date, publisher, container, license Field and value analysis via https://github.com/miku/indigo.
* bulk edit updatesBryan Newbold2019-12-261-3/+4
|
* orcid: skip non-person ORCID recordsBryan Newbold2019-12-261-0/+4
|
* Merge branch 'martin-datacite-daily-harvest' into 'master'Martin Czygan2019-12-263-5/+73
|\ | | | | | | | | Datacite daily harvest See merge request webgroup/fatcat!6
| * datacite: fix harvest testMartin Czygan2019-12-271-1/+1
| | | | | | | | | | | | Produced messages should match: jq '.data|length' tests/files/datacite_api.json
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-11 02:06:14 +0000