aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
...
| * datacite: parse_datacite_dates returns monthMartin Czygan2020-01-032-17/+51
| | | | | | | | As [...] we will soon add support for release_month field in the release schema.
| * datacite: prepare release_month (stub)Martin Czygan2020-01-032-24/+24
| |
| * datacite: lowercase only onceMartin Czygan2020-01-031-3/+4
| |
| * add pycountry dependencyMartin Czygan2020-01-032-1/+9
| |
| * add missing pathlib2 dependencyMartin Czygan2020-01-032-1/+18
| | | | | | | | | | first seen in CI (jobs/230137), slightly related: https://github.com/pytest-dev/pytest/issues/3953
| * update potentially outdated Pipfile.lockMartin Czygan2020-01-031-96/+86
| | | | | | | | | | | | | | | | via: $ pipenv lock CI complained with a slightly cryptic: > TypeError: __init__() missing 1 required positional argument: 'self'
| * datacite: remove --lang-detect flagMartin Czygan2020-01-037-25/+21
| | | | | | | | Estimated time for a single call is in the order of 50ms.
| * datacite: add another test caseMartin Czygan2020-01-023-1/+71
| |
| * datacite: open case for editing after creationMartin Czygan2020-01-021-0/+2
| |
| * datacite: add helper script to create new test caseMartin Czygan2020-01-021-0/+14
| |
| * datacite: address raw_name index form commentMartin Czygan2020-01-0221-112/+171
| | | | | | | | | | | | | | | | | | > The convention for display_name and raw_name is to be how the name would normally be printed, not in index form (surname comma given_name). So we might need to un-encode names like "Tricart, Pierre". Use an additional `index_form_to_display_name` function to convert index from to display form, heuristically.
| * datacite: add two more skipable tokensMartin Czygan2020-01-021-1/+1
| |
| * datacite: add conversion fixturesMartin Czygan2020-01-0250-1/+3949
| | | | | | | | | | | | | | | | | | | | | | | | | | The `test_datacite_conversions` function will compare an input (datacite) document to an expected output (release entity as JSON). This way, it should not be too hard to add more cases by adding: input, output - and by increasing the counter in the range loop within the test. To view input and result side by side with vim, change into the test directory and run: tests/files/datacite $ ./caseview.sh 18
| * datacite: names can be 'Unav', tooMartin Czygan2020-01-021-1/+4
| |
| * datacite: avoid more None valuesMartin Czygan2020-01-011-4/+4
| |
| * datacite: address 'Unpublished' publisherMartin Czygan2019-12-311-9/+10
| |
| * datacite: ensure name schema is definedMartin Czygan2019-12-311-1/+2
| |
| * datacite: fix typoMartin Czygan2019-12-311-1/+1
| |
| * datacite: isascii was added in 3.7, onlyMartin Czygan2019-12-311-1/+7
| |
| * datacite: skip non-ascii doi for nowMartin Czygan2019-12-311-0/+4
| | | | | | | | | | | | Example of a non-ascii doi: * https://doi.org/10.13125/américacrítica/3017
| * datacite: clean doiMartin Czygan2019-12-311-1/+13
| | | | | | | | | | | | | | address issue with EN DASH DOI. > "external identifier doesn't match required pattern for a DOI (expected, eg, '10.1234/aksjdfh'): 10.25513/1812-3996.2017.1.34–42"
| * datacite: update docsMartin Czygan2019-12-311-9/+9
| |
| * datacite: perform additional checks on contribMartin Czygan2019-12-301-3/+9
| |
| * datacite: check for empty title after cleanMartin Czygan2019-12-291-2/+5
| |
| * datacite: update docs with observed valuesMartin Czygan2019-12-291-1/+3
| |
| * datacite: page number misses are too commonMartin Czygan2019-12-281-1/+2
| | | | | | | | | | | | Should be a level debug, not info. Examples: E675, n/a, 15D.2.1, 15D.2.1, A.1E.1, A.1E.1, ...
| * datacite: suppress debug-like language lookup miss messageMartin Czygan2019-12-281-1/+3
| |
| * datacite: adjust testsMartin Czygan2019-12-281-2/+1
| |
| * datacite: treat untyped names as peopleMartin Czygan2019-12-281-1/+1
| |
| * datacite: include container_name top level key in extraMartin Czygan2019-12-281-7/+21
| |
| * datacite: use clean on field valuesMartin Czygan2019-12-281-2/+28
| |
| * datacite: include doi in error messagesMartin Czygan2019-12-281-8/+8
| |
| * remove langcodes dependencyMartin Czygan2019-12-282-15/+0
| |
| * datacite: limit abstract lengthMartin Czygan2019-12-281-0/+6
| |
| * datacite: use iso 639-1 codesMartin Czygan2019-12-281-7/+4
| |
| * datacite: use specific auth varMartin Czygan2019-12-281-1/+1
| |
| * datacite: add missing --extid-map-file flagMartin Czygan2019-12-281-0/+4
| |
| * address first round of MR14 commentsMartin Czygan2019-12-284-150/+503
| | | | | | | | | | | | | | | | | | | | | | | | | | * add missing langdetect * use entity_to_dict for json debug output * factor out code for fields in function and add table driven tests * update citeproc types * add author as default role * add raw_affiliation * include relations from datacite * remove url (covered by doi already) Using yapf for python formatting.
| * datacite: move common date patterns out of the loopMartin Czygan2019-12-281-3/+4
| | | | | | | | Additionally, try the unspecific (%Y) pattern last.
| * improve datacite field mapping and importMartin Czygan2019-12-285-59/+245
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current version succeeded to import a random sample of 100000 records (0.5%) from datacite. The --debug (write JSON to stdout) and --insert-log-file (log batch before committing to db) flags are temporary added to help debugging. Add few unit tests. Some edge cases: a) Existing keys without value requires a slightly awkward: ``` titles = attributes.get('titles', []) or [] ``` b) There can be 0, 1, or more (first one wins) titles. c) Date handling is probably not ideal. Datacite has a potentiall fine grained list of dates. The test case (tests/files/datacite_sample.jsonl) refers to https://ssl.fao.org/glis/doi/10.18730/8DYM9, which has date (main descriptor) 1986. The datacite record contains: 2017 (publicationYear, probably the year of record creation with reference system), 1978-06-03 (collected, e.g. experimental sample), 1986 ("Accepted"). The online version of the resource knows even one more date (2019-06-05 10:14:43 by WIEWS update).
| * datacite: add missing mappings and notesMartin Czygan2019-12-281-266/+175
| |
| * datacite: basic field mappingsMartin Czygan2019-12-281-41/+181
| | | | | | | | | | | | | | | | | | | | Currently using two external libraries: * dateparser * langcodes Note: This commit includes lots of wip docs and field stat in comment, which should be removed.
| * datacite: importer skeletonMartin Czygan2019-12-284-0/+514
| | | | | | | | | | | | * contributors, title, date, publisher, container, license Field and value analysis via https://github.com/miku/indigo.
* | 2019-01-07 status updateBryan Newbold2020-01-072-0/+36
| |
* | chocula bulk edit noteBryan Newbold2020-01-072-0/+15
| |
* | importers: control update behavior with more-standard flagBryan Newbold2020-01-066-3/+15
| |
* | proposals: standardize a bitBryan Newbold2020-01-039-3/+34
| |
* | notes on search query parsing (WIP)Bryan Newbold2020-01-031-0/+22
| |
* | fatcat identifiers proposal (WIP)Bryan Newbold2020-01-031-0/+25
| |
* | proposal: python3.7 upgradeBryan Newbold2020-01-031-0/+101
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-25 21:13:42 +0000