| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
| |
|
|
|
|
|
| |
Seeing a bunch of these due to re-ingests not including this field
because of an earlier persist bug.
|
|
|
|
| |
As reminded in code review, thanks Martin.
|
| |
|
|
|
|
|
|
|
| |
For cases where there might be both PMC and DOI urls, do the europmc.org
PMC ones over DOI option.
May want to turn this into a config or command-line option in the future.
|
|
|
|
|
|
| |
This is a large refactor of the ingest script. It adds a number of
filtering options (for all modes), and new modes for free-form queries
or limiting to specific external identifiers.
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Records from https://www.micropublication.org/ did not have a date in
FC, although raw data contained date strings - they were not using the
finer-grained "attributes.date" but "attributes.published" and/or
"attributes.publicationYear".
Support for those fields has been added, including a test case.
During this test (#30) a processing gap for names became clear (author
may have "given_name" and "surname", but no "name"). This bug has been
fixed, too.
|
| |
|
|
|
|
| |
From martin, thanks.
|
| |
|
|
|
|
|
|
|
| |
`ingest_oa_only` behavior, and other filters, now handled in the entity
update worker, instead of in the transform function.
Also add a DOI prefix blocklist feature.
|
|
|
|
| |
Refactoring to move this filter elsewhere
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
My current understanding is that consumer group names should be
one-to-one with topic names. I previously though offsets were stored on
a {topic, group} key, but they seem to be mixed and having too many
workers in the same group is bad. In particular, we don't want
cross-talk or load between QA and prod.
All these topics are caught up in prod, so deploying this change and
restarting workers should be safe.
This commit does not update the elasticsearch or entity updates workers.
|
| |
|
| |
|
|\
| |
| |
| |
| | |
pagination: use correct endpoint in navigation
See merge request webgroup/fatcat!19
|
| | |
|
| |
| |
| |
| | |
Should have run tests before pushing!
|
| | |
|
|/
|
|
|
|
|
|
|
| |
Technically, [...] DOI names may incorporate any printable characters
from the Universal Character Set (UCS-2), of ISO/IEC 10646, which is the
character set defined by Unicode (https://www.doi.org/doi_handbook/2_Numbering.html#2.5.1).
For mostly QA reasons, we currently treat a DOI with an "en dash" as
invalid.
|
| |
|
|
|
|
|
|
| |
Should be backwards compatible with old ingest results.
Fixed a bug with glutton ident detection.
|
| |
|
| |
|
|
|
|
|
|
| |
Good to have exceptions tracked and stored even for commands run from
the command line. But in particular the importer runs as a kafka worker
and should be tracking excpetions.
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Pipfile.lock is broken.
* martin-datacite-import: (68 commits)
datacite: pass in doi into factored out method
datacite: reformat test cases and use jq . --sort-keys
datacite: factor out contributor handling
datacite: catch type mismatch in language detection
datacite: adjust tests for release_month
datacite: name extra.month, extra.release_month
datacite: mark additional files as stub
datacite: CCDC are entries, mostly
datacite: use more specific release_type, if possible
datacite: ignore certain names
datacite: over 3% records have the same title: stub
datacite: fill a few more release_type gaps
datacite: adding datacite-specific extra metadata
datacite: apply pylint suggestions
datacite: fix typos
datacite: set release_stage to published by default
datacite: month field should be top-level
datacite: include month in extra
datacite: indicate mismatched file in test
datacite: clean abstracts, use unknown value tokens
...
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| |
| | |
Use values from:
* attributes.creators[]
* attributes.contributors[]
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The GBIF (https://www.gbif.org/) deposits most records under the titles:
* 599243 GBIF Occurrence Download
* 41176 Occurrence Download
Mark them as "stub" for the moment
(https://guide.fatcat.wiki/entity_release.html#release_type-vocabulary).
|