| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
| | |
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The GBIF (https://www.gbif.org/) deposits most records under the titles:
* 599243 GBIF Occurrence Download
* 41176 Occurrence Download
Mark them as "stub" for the moment
(https://guide.fatcat.wiki/entity_release.html#release_type-vocabulary).
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* citeproc: http://docs.citationstyles.org/en/stable/specification.html#appendix-iii-types
* resourceTypeGeneral: https://schema.datacite.org/meta/kernel-4.0/doc/DataCite-MetadataKernel_v4.0.pdf#page=32
* resourceType: uncontrolled, over 170000 distinct values, frequent:
null, Dataset, JournalArticle, PGRFA Material, Journal Article,
Dataset/UNITE Species Hypothesis, ...
General frequency:
* "attributes.types": 18210075,
* "attributes.types.ris": 18058890,
* "attributes.types.bibtex": 18058888,
* "attributes.types.citeproc": 18058890,
* "attributes.types.schemaOrg": 18058929,
* "attributes.types.resourceType": 12737988,
* "attributes.types.resourceTypeGeneral": 16576139,
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* attributes.metadataVersion
* attributes.schemaVersion
* attributes.version (source dependent values, follows suggestions in
https://schema.datacite.org/meta/kernel-4.3/doc/DataCite-MetadataKernel_v4.3.pdf#page=26,
but values vary)
Furthermore:
* attributes.types.resourceTypeGeneral
* attributes.types.resourceType
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| | |
Set to `None` only if there is no publisher yet.
Docs: https://support.datacite.org/docs/doi-states
|
| | |
|
| |
| |
| |
| |
| | |
> include release_month as a top-level extra field [...] to
auto-populate the schema field from that
|
| |
| |
| |
| |
| |
| |
| |
| | |
Datacite defines placeholders for unknown values:
* https://support.datacite.org/docs/schema-values-unknown-information-v43
Clean abstracts.
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| | |
> always include extra values for the respective DOI registrars
(datacite, crossref, jalc), even if they are empty ({}), to be used as a
flag so we know which DOI registrar supplied the metadata.
|
| | |
|
| |
| |
| |
| | |
As [...] we will soon add support for release_month field in the release schema.
|
| | |
|
| | |
|
| |
| |
| |
| | |
Estimated time for a single call is in the order of 50ms.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
> The convention for display_name and raw_name is to be how the name
would normally be printed, not in index form (surname comma given_name).
So we might need to un-encode names like "Tricart, Pierre".
Use an additional `index_form_to_display_name` function to convert index
from to display form, heuristically.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| | |
Example of a non-ascii doi:
* https://doi.org/10.13125/américacrítica/3017
|
| |
| |
| |
| |
| |
| |
| | |
address issue with EN DASH DOI.
> "external identifier doesn't match required pattern for a DOI (expected,
eg, '10.1234/aksjdfh'): 10.25513/1812-3996.2017.1.34–42"
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| | |
Should be a level debug, not info.
Examples: E675, n/a, 15D.2.1, 15D.2.1, A.1E.1, A.1E.1, ...
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* add missing langdetect
* use entity_to_dict for json debug output
* factor out code for fields in function and add table driven tests
* update citeproc types
* add author as default role
* add raw_affiliation
* include relations from datacite
* remove url (covered by doi already)
Using yapf for python formatting.
|
| |
| |
| |
| | |
Additionally, try the unspecific (%Y) pattern last.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Current version succeeded to import a random sample of 100000 records
(0.5%) from datacite.
The --debug (write JSON to stdout) and --insert-log-file (log batch
before committing to db) flags are temporary added to help debugging.
Add few unit tests.
Some edge cases:
a) Existing keys without value requires a slightly awkward:
```
titles = attributes.get('titles', []) or []
```
b) There can be 0, 1, or more (first one wins) titles.
c) Date handling is probably not ideal. Datacite has a potentiall fine
grained list of dates.
The test case (tests/files/datacite_sample.jsonl) refers to
https://ssl.fao.org/glis/doi/10.18730/8DYM9, which has date (main
descriptor) 1986. The datacite record contains: 2017 (publicationYear,
probably the year of record creation with reference system), 1978-06-03
(collected, e.g. experimental sample), 1986 ("Accepted"). The online
version of the resource knows even one more date (2019-06-05 10:14:43 by
WIEWS update).
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Currently using two external libraries:
* dateparser
* langcodes
Note: This commit includes lots of wip docs and field stat in comment,
which should be removed.
|
| |
| |
| |
| |
| |
| | |
* contributors, title, date, publisher, container, license
Field and value analysis via https://github.com/miku/indigo.
|
|/ |
|
| |
|
|
|
|
|
| |
The bracket syntax is inclusive. See also:
https://www.elastic.co/guide/en/elasticsearch/reference/7.5/query-dsl-query-string-query.html#_ranges
|