| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
According to sentry, running `c.get('nameIdentifiers', []) or []` on a c with value:
```
{'affiliation': [],
'familyName': 'Guidon',
'givenName': 'Manuel',
'nameIdentifiers': {'nameIdentifier': 'https://orcid.org/0000-0003-3543-6683',
'nameIdentifierScheme': 'ORCID',
'schemeUri': 'https://orcid.org'},
'nameType': 'Personal'}
```
results in a string, which I cannot reproduce. The document in question at:
https://api.datacite.org/dois/10.26275/kuw1-fdls seems fine, too.
|
| |
|
|
| |
refs: #44035
|
| | |
|
| | |
|
| |
|
|
| |
via "missed potential license", refs #58
|
| | |
|
| | |
|
| |\ |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
One of these (in ingest importer pipeline) is an actual bug, the others
are just changing the syntax to be more explicit/conservative.
The ingest importer bug seems to have resulted in some bad file match
imports; scale of impact is unknown.
|
| | |
| |
| |
| |
| |
| |
| | |
Until reviewing I didn't realize we were even doing this currently.
Hopefluly has not impacted too many imports, as almost all ingests use
an external identifer, so only those with identifers not in fatcat for
whatever reason.
|
| | |
| |
| |
| |
| |
| |
| | |
Up to now, we expected the description to be a string or list. Add
handling for int as well.
First appeared: Apr 22 19:58:39.
|
| |/
|
|
|
|
|
| |
It was possible that contribs got added which had no raw name. One
example would be a name consisting of whitespace only.
This fix adds a final check for this case.
|
| | |
|
| |
|
|
|
|
|
| |
The primary motivation for this change is that fatcat *requires* a
non-empty title for each release entity. Pubmed/Medline occasionally
indexes just a VenacularTitle with no ArticleTitle for foreign
publications, and currently those records don't end up in fatcat at all.
|
| | |
|
| |
|
|
| |
See previous pubmed commit for details.
|
| |
|
|
|
|
|
|
|
|
|
| |
Yikes! Apparently when a tag has child tags, .string will return None
instead of all the strings. .get_text() returns all of it:
https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text
https://www.crummy.com/software/BeautifulSoup/bs4/doc/#string
I've things like identifiers as .string, when we expect only a single
string inside.
|
| |\
| |
| | |
Correct spelling mistakes
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
Works around a bug in production:
AttributeError: 'NoneType' object has no attribute 'replace'
(datacite.py:724)
NOTE: there are no tests for this code path
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
Example of entities with bogus years:
https://fatcat.wiki/release/search?q=doi_registrar%3Adatacite+year%3A%3E2100
We can do a clean-up task, but first need to prevent creation of new bad
metadata.
|
| | |
| |
| |
| |
| |
| |
| | |
This resolves a situation noticed in prod where we were only
importing/updating a single reference per article.
Includes a regression test.
|
| | |
| |
| |
| |
| |
| |
| | |
In particular, with daily updates in most cases the DOI will be
registered first, then the entity updated with PMID when that is
available. Often the pubmed metadata will be more complete, with
abstracts etc, and we'll want those improvements.
|
| | |
| |
| |
| |
| |
| | |
It seems like OUP pre-registers DOIs with this place-holder title, then
updates the Crossref metdata when the paper is actually published. We
should wait until the real title is available before creating an entity.
|
| |\ \
| |/
|/|
| |
| | |
pubmed and arxiv harvest preparations
See merge request webgroup/fatcat!28
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Address kafka tradeoff between long and short time-outs. Shorter
time-outs would facilitate
> consumer group re-balances and other consumer group state changes
[...] in a reasonable human time-frame.
|
| | |
| |
| |
| |
| |
| |
| | |
* add PubmedFTPWorker
* utils are currently stored alongside pubmed (e.g. ftpretr, xmlstream)
but may live elsewhere, as they are more generic
* add KafkaBs4XmlPusher
|
| | | |
|
| |\ \ |
|
| | | | |
|
| | | |
| | |
| | |
| | | |
Should probably be refactored out in to shared cleanup code.
|
| | | | |
|
| | | | |
|
| | |/ |
|
| | | |
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
Seeing a bunch of these due to re-ingests not including this field
because of an earlier persist bug.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Records from https://www.micropublication.org/ did not have a date in
FC, although raw data contained date strings - they were not using the
finer-grained "attributes.date" but "attributes.published" and/or
"attributes.publicationYear".
Support for those fields has been added, including a test case.
During this test (#30) a processing gap for names became clear (author
may have "given_name" and "surname", but no "name"). This bug has been
fixed, too.
|
