aboutsummaryrefslogtreecommitdiffstats
path: root/python
Commit message (Collapse)AuthorAgeFilesLines
...
* update ingest worker for schema tweaksBryan Newbold2020-01-151-8/+15
| | | | | | Should be backwards compatible with old ingest results. Fixed a bug with glutton ident detection.
* ingest: allow more sources to auto-importBryan Newbold2020-01-151-1/+2
|
* datacite: skip records without a doiMartin Czygan2020-01-131-0/+4
|
* add missing sentry/raven tagsBryan Newbold2020-01-103-2/+19
| | | | | | Good to have exceptions tracked and stored even for commands run from the command line. But in particular the importer runs as a kafka worker and should be tracking excpetions.
* datacite: add entry to license slug mapMartin Czygan2020-01-092-0/+2
|
* datacite: ignore known unknown values in resourceType*Martin Czygan2020-01-094-3/+97
|
* datacite: abstracts may be strings or list of stringsMartin Czygan2020-01-096-3/+202
|
* datacite: improve license_slug handlingMartin Czygan2020-01-094-62/+134
|
* datacite: add 'Unknown' to blacklistMartin Czygan2020-01-092-8/+6
|
* datacite: get rid of schemaVersionMartin Czygan2020-01-0918-35/+14
|
* update depsMartin Czygan2020-01-081-86/+69
|
* Merge branch 'martin-datacite-import'Martin Czygan2020-01-0864-5/+5736
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pipfile.lock is broken. * martin-datacite-import: (68 commits) datacite: pass in doi into factored out method datacite: reformat test cases and use jq . --sort-keys datacite: factor out contributor handling datacite: catch type mismatch in language detection datacite: adjust tests for release_month datacite: name extra.month, extra.release_month datacite: mark additional files as stub datacite: CCDC are entries, mostly datacite: use more specific release_type, if possible datacite: ignore certain names datacite: over 3% records have the same title: stub datacite: fill a few more release_type gaps datacite: adding datacite-specific extra metadata datacite: apply pylint suggestions datacite: fix typos datacite: set release_stage to published by default datacite: month field should be top-level datacite: include month in extra datacite: indicate mismatched file in test datacite: clean abstracts, use unknown value tokens ...
| * datacite: pass in doi into factored out methodMartin Czygan2020-01-081-2/+3
| |
| * datacite: reformat test cases and use jq . --sort-keysMartin Czygan2020-01-0854-2299/+2301
| |
| * datacite: factor out contributor handlingMartin Czygan2020-01-086-82/+210
| | | | | | | | | | | | | | Use values from: * attributes.creators[] * attributes.contributors[]
| * datacite: catch type mismatch in language detectionMartin Czygan2020-01-081-3/+2
| |
| * datacite: adjust tests for release_monthMartin Czygan2020-01-0812-12/+12
| |
| * datacite: name extra.month, extra.release_monthMartin Czygan2020-01-081-1/+3
| |
| * datacite: mark additional files as stubMartin Czygan2020-01-084-1/+77
| |
| * datacite: CCDC are entries, mostlyMartin Czygan2020-01-082-1/+5
| |
| * datacite: use more specific release_type, if possibleMartin Czygan2020-01-081-0/+6
| |
| * datacite: ignore certain namesMartin Czygan2020-01-081-0/+6
| |
| * datacite: over 3% records have the same title: stubMartin Czygan2020-01-081-0/+7
| | | | | | | | | | | | | | | | | | | | The GBIF (https://www.gbif.org/) deposits most records under the titles: * 599243 GBIF Occurrence Download * 41176 Occurrence Download Mark them as "stub" for the moment (https://guide.fatcat.wiki/entity_release.html#release_type-vocabulary).
| * datacite: fill a few more release_type gapsMartin Czygan2020-01-081-17/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * citeproc: http://docs.citationstyles.org/en/stable/specification.html#appendix-iii-types * resourceTypeGeneral: https://schema.datacite.org/meta/kernel-4.0/doc/DataCite-MetadataKernel_v4.0.pdf#page=32 * resourceType: uncontrolled, over 170000 distinct values, frequent: null, Dataset, JournalArticle, PGRFA Material, Journal Article, Dataset/UNITE Species Hypothesis, ... General frequency: * "attributes.types": 18210075, * "attributes.types.ris": 18058890, * "attributes.types.bibtex": 18058888, * "attributes.types.citeproc": 18058890, * "attributes.types.schemaOrg": 18058929, * "attributes.types.resourceType": 12737988, * "attributes.types.resourceTypeGeneral": 16576139,
| * datacite: adding datacite-specific extra metadataMartin Czygan2020-01-0731-1468/+1598
| | | | | | | | | | | | | | | | | | | | | | | | | | * attributes.metadataVersion * attributes.schemaVersion * attributes.version (source dependent values, follows suggestions in https://schema.datacite.org/meta/kernel-4.3/doc/DataCite-MetadataKernel_v4.3.pdf#page=26, but values vary) Furthermore: * attributes.types.resourceTypeGeneral * attributes.types.resourceType
| * datacite: apply pylint suggestionsMartin Czygan2020-01-071-8/+10
| |
| * datacite: fix typosMartin Czygan2020-01-072-2/+2
| |
| * datacite: set release_stage to published by defaultMartin Czygan2020-01-061-4/+5
| | | | | | | | | | | | Set to `None` only if there is no publisher yet. Docs: https://support.datacite.org/docs/doi-states
| * datacite: month field should be top-levelMartin Czygan2020-01-0612-16/+16
| |
| * datacite: include month in extraMartin Czygan2020-01-0612-11/+15
| | | | | | | | | | > include release_month as a top-level extra field [...] to auto-populate the schema field from that
| * datacite: indicate mismatched file in testMartin Czygan2020-01-061-1/+1
| |
| * datacite: clean abstracts, use unknown value tokensMartin Czygan2020-01-064-7/+29
| | | | | | | | | | | | | | | | Datacite defines placeholders for unknown values: * https://support.datacite.org/docs/schema-values-unknown-information-v43 Clean abstracts.
| * datacite: clean abstract as wellMartin Czygan2020-01-061-1/+1
| |
| * datacite: filter out 'Cites' relation as wellMartin Czygan2020-01-061-1/+1
| |
| * pytest: explicitly indicate all in-scope test filesBryan Newbold2020-01-041-3/+1
| | | | | | | | | | | | | | | | | | | | | | The purpose of this change is to test errors when pytest tries to recursively update assertion statements in all dependent packages. The reason pytest does this is to add pretty printing, which is nice, but probably shouldn't be done in all dependency libraries. This fixes test problems with both CSL (citeproc_styles) and dateparser (when actually imported in code, which currently on master does not happen).
| * datacite: always include "datacite" key in extraMartin Czygan2020-01-0415-28/+28
| | | | | | | | | | | | > always include extra values for the respective DOI registrars (datacite, crossref, jalc), even if they are empty ({}), to be used as a flag so we know which DOI registrar supplied the metadata.
| * datacite: use normal.clean_doiMartin Czygan2020-01-032-15/+1
| |
| * datacite: parse_datacite_dates returns monthMartin Czygan2020-01-032-17/+51
| | | | | | | | As [...] we will soon add support for release_month field in the release schema.
| * datacite: prepare release_month (stub)Martin Czygan2020-01-032-24/+24
| |
| * datacite: lowercase only onceMartin Czygan2020-01-031-3/+4
| |
| * add pycountry dependencyMartin Czygan2020-01-032-1/+9
| |
| * add missing pathlib2 dependencyMartin Czygan2020-01-032-1/+18
| | | | | | | | | | first seen in CI (jobs/230137), slightly related: https://github.com/pytest-dev/pytest/issues/3953
| * update potentially outdated Pipfile.lockMartin Czygan2020-01-031-96/+86
| | | | | | | | | | | | | | | | via: $ pipenv lock CI complained with a slightly cryptic: > TypeError: __init__() missing 1 required positional argument: 'self'
| * datacite: remove --lang-detect flagMartin Czygan2020-01-037-25/+21
| | | | | | | | Estimated time for a single call is in the order of 50ms.
| * datacite: add another test caseMartin Czygan2020-01-023-1/+71
| |
| * datacite: open case for editing after creationMartin Czygan2020-01-021-0/+2
| |
| * datacite: add helper script to create new test caseMartin Czygan2020-01-021-0/+14
| |
| * datacite: address raw_name index form commentMartin Czygan2020-01-0221-112/+171
| | | | | | | | | | | | | | | | | | > The convention for display_name and raw_name is to be how the name would normally be printed, not in index form (surname comma given_name). So we might need to un-encode names like "Tricart, Pierre". Use an additional `index_form_to_display_name` function to convert index from to display form, heuristically.
| * datacite: add two more skipable tokensMartin Czygan2020-01-021-1/+1
| |
| * datacite: add conversion fixturesMartin Czygan2020-01-0250-1/+3949
| | | | | | | | | | | | | | | | | | | | | | | | | | The `test_datacite_conversions` function will compare an input (datacite) document to an expected output (release entity as JSON). This way, it should not be too hard to add more cases by adding: input, output - and by increasing the counter in the range loop within the test. To view input and result side by side with vim, change into the test directory and run: tests/files/datacite $ ./caseview.sh 18
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-16 05:39:30 +0000