| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
| |
Oh no!
This bug may actually have had significant negative impact on metadata
in fatcat, in terms of missing container_id associations with pubmed
entities. There are about 500k release entities with a PMID but no
container_id. Of those, 89k have at least a container_name. Unclear how
many would have matched to ISSN-L and thus to a container.
|
| |
|
|
| |
We are python3.7 now, so this isn't needed.
|
| |
|
|
|
|
| |
These should not have any behavior changes, though a number of exception
catches are now more general, and there may be long-tail exceptions
getting thrown in these statements.
|
| |\
| |
| |
| |
| | |
datacite: address duplicated contributor issue
See merge request webgroup/fatcat!65
|
| | |\ |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
Use string comparison.
* https://fatcat.wiki/release/spjysmrnsrgyzgq6ise5o44rlu/contribs
* https://api.datacite.org/dois/10.25940/roper-31098406
|
| | |/
|/|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
According to sentry, running `c.get('nameIdentifiers', []) or []` on a c with value:
```
{'affiliation': [],
'familyName': 'Guidon',
'givenName': 'Manuel',
'nameIdentifiers': {'nameIdentifier': 'https://orcid.org/0000-0003-3543-6683',
'nameIdentifierScheme': 'ORCID',
'schemeUri': 'https://orcid.org'},
'nameType': 'Personal'}
```
results in a string, which I cannot reproduce. The document in question at:
https://api.datacite.org/dois/10.26275/kuw1-fdls seems fine, too.
|
| | |
| |
| |
| | |
refs: #44035
|
| |/ |
|
| | |
|
| |
|
|
| |
via "missed potential license", refs #58
|
| | |
|
| | |
|
| |\ |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
One of these (in ingest importer pipeline) is an actual bug, the others
are just changing the syntax to be more explicit/conservative.
The ingest importer bug seems to have resulted in some bad file match
imports; scale of impact is unknown.
|
| | |
| |
| |
| |
| |
| |
| | |
Until reviewing I didn't realize we were even doing this currently.
Hopefluly has not impacted too many imports, as almost all ingests use
an external identifer, so only those with identifers not in fatcat for
whatever reason.
|
| | |
| |
| |
| |
| |
| |
| | |
Up to now, we expected the description to be a string or list. Add
handling for int as well.
First appeared: Apr 22 19:58:39.
|
| |/
|
|
|
|
|
| |
It was possible that contribs got added which had no raw name. One
example would be a name consisting of whitespace only.
This fix adds a final check for this case.
|
| | |
|
| |
|
|
|
|
|
| |
The primary motivation for this change is that fatcat *requires* a
non-empty title for each release entity. Pubmed/Medline occasionally
indexes just a VenacularTitle with no ArticleTitle for foreign
publications, and currently those records don't end up in fatcat at all.
|
| | |
|
| |
|
|
| |
See previous pubmed commit for details.
|
| |
|
|
|
|
|
|
|
|
|
| |
Yikes! Apparently when a tag has child tags, .string will return None
instead of all the strings. .get_text() returns all of it:
https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text
https://www.crummy.com/software/BeautifulSoup/bs4/doc/#string
I've things like identifiers as .string, when we expect only a single
string inside.
|
| |\
| |
| | |
Correct spelling mistakes
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
Works around a bug in production:
AttributeError: 'NoneType' object has no attribute 'replace'
(datacite.py:724)
NOTE: there are no tests for this code path
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
Example of entities with bogus years:
https://fatcat.wiki/release/search?q=doi_registrar%3Adatacite+year%3A%3E2100
We can do a clean-up task, but first need to prevent creation of new bad
metadata.
|
| | |
| |
| |
| |
| |
| |
| | |
This resolves a situation noticed in prod where we were only
importing/updating a single reference per article.
Includes a regression test.
|
| | |
| |
| |
| |
| |
| |
| | |
In particular, with daily updates in most cases the DOI will be
registered first, then the entity updated with PMID when that is
available. Often the pubmed metadata will be more complete, with
abstracts etc, and we'll want those improvements.
|
| | |
| |
| |
| |
| |
| | |
It seems like OUP pre-registers DOIs with this place-holder title, then
updates the Crossref metdata when the paper is actually published. We
should wait until the real title is available before creating an entity.
|
| |\ \
| |/
|/|
| |
| | |
pubmed and arxiv harvest preparations
See merge request webgroup/fatcat!28
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Address kafka tradeoff between long and short time-outs. Shorter
time-outs would facilitate
> consumer group re-balances and other consumer group state changes
[...] in a reasonable human time-frame.
|
| | |
| |
| |
| |
| |
| |
| | |
* add PubmedFTPWorker
* utils are currently stored alongside pubmed (e.g. ftpretr, xmlstream)
but may live elsewhere, as they are more generic
* add KafkaBs4XmlPusher
|
| | | |
|
| |\ \ |
|
| | | | |
|
| | | |
| | |
| | |
| | | |
Should probably be refactored out in to shared cleanup code.
|
| | | | |
|
| | | | |
|
| | |/ |
|
| | | |
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
