| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
| | |
|
| | |
|
|/
|
|
|
|
|
|
|
|
|
|
|
| |
Records from https://www.micropublication.org/ did not have a date in
FC, although raw data contained date strings - they were not using the
finer-grained "attributes.date" but "attributes.published" and/or
"attributes.publicationYear".
Support for those fields has been added, including a test case.
During this test (#30) a processing gap for names became clear (author
may have "given_name" and "surname", but no "name"). This bug has been
fixed, too.
|
| |
|
|
|
|
| |
From martin, thanks.
|
| |
|
|
|
|
|
|
|
| |
`ingest_oa_only` behavior, and other filters, now handled in the entity
update worker, instead of in the transform function.
Also add a DOI prefix blocklist feature.
|
|
|
|
| |
Refactoring to move this filter elsewhere
|
| |
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
My current understanding is that consumer group names should be
one-to-one with topic names. I previously though offsets were stored on
a {topic, group} key, but they seem to be mixed and having too many
workers in the same group is bad. In particular, we don't want
cross-talk or load between QA and prod.
All these topics are caught up in prod, so deploying this change and
restarting workers should be safe.
This commit does not update the elasticsearch or entity updates workers.
|
| |
|
|
|
|
| |
For ease of reading and comparison
|
| |
|
|\
| |
| |
| |
| | |
fix a typo in openapi definition
See merge request webgroup/fatcat!20
|
| | |
|
|\ \
| | |
| | |
| | |
| | | |
fix two typos in editing guide
See merge request webgroup/fatcat!21
|
| |/ |
|
| | |
|
| | |
|
| | |
|
| | |
|
|\ \
| |/
|/|
| |
| | |
fix typos in openapi spec descriptions
See merge request webgroup/fatcat!22
|
|/ |
|
|\
| |
| |
| |
| | |
pagination: use correct endpoint in navigation
See merge request webgroup/fatcat!19
|
| | |
|
| |
| |
| |
| | |
Should have run tests before pushing!
|
| | |
|
| | |
|
|\ \
| | |
| | |
| | |
| | | |
fatcatd: fix corner-case in abstracts insertion
See merge request webgroup/fatcat!17
|
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Both the release_abstract relation table and the abstracts table inserts
were being gated by a check on new abstracts table rows.
I *think* the chance of this having caused problems is low. The most
likely would have been updates to exiting entities that somehow removed
the abstracts content, but not the sha1 keys. This is not the default
behavior of the API: either the entire abstract (content and hash) is
returned, or the abstracts are hidden entirely (via `hide` flag).
Still, best to be careful!
|
|\ \
| |/
|/|
| |
| | |
do not normalize "en dash" in DOI
See merge request webgroup/fatcat!18
|
|/
|
|
|
|
|
|
|
| |
Technically, [...] DOI names may incorporate any printable characters
from the Universal Character Set (UCS-2), of ISO/IEC 10646, which is the
character set defined by Unicode (https://www.doi.org/doi_handbook/2_Numbering.html#2.5.1).
For mostly QA reasons, we currently treat a DOI with an "en dash" as
invalid.
|
| |
|
|
|
|
|
|
| |
Should be backwards compatible with old ingest results.
Fixed a bug with glutton ident detection.
|
| |
|
|
|
|
| |
This skips postgres-internal tables in size output
|
|\
| |
| |
| |
| | |
datacite: skip records without a doi
See merge request webgroup/fatcat!16
|
|/ |
|
|
|
|
|
|
| |
Good to have exceptions tracked and stored even for commands run from
the command line. But in particular the importer runs as a kafka worker
and should be tracking excpetions.
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Pipfile.lock is broken.
* martin-datacite-import: (68 commits)
datacite: pass in doi into factored out method
datacite: reformat test cases and use jq . --sort-keys
datacite: factor out contributor handling
datacite: catch type mismatch in language detection
datacite: adjust tests for release_month
datacite: name extra.month, extra.release_month
datacite: mark additional files as stub
datacite: CCDC are entries, mostly
datacite: use more specific release_type, if possible
datacite: ignore certain names
datacite: over 3% records have the same title: stub
datacite: fill a few more release_type gaps
datacite: adding datacite-specific extra metadata
datacite: apply pylint suggestions
datacite: fix typos
datacite: set release_stage to published by default
datacite: month field should be top-level
datacite: include month in extra
datacite: indicate mismatched file in test
datacite: clean abstracts, use unknown value tokens
...
|