| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
import refactors and deprecations
Some of these are from old stale branches (the datacite subject metadata patch), but most are from yesterday and today. Sort of a hodge-podge, but the general theme is getting around to deferred cleanups and refactors specific to importer code before making some behavioral changes.
The Datacite-specific stuff could use review here.
Remove unused/deprecated/dead code:
- cdl_dash_dat and wayback_static importers, which were for specific early example entities and have been superseded by other importers
- "extid map" sqlite3 feature from several importers, was only used for initial bulk imports (and maybe should not have been used)
Refactors:
- moved a number of large datastructures out of importer code and into a dedicated static file (`biblio_lookup_tables.py`). Didn't move all, just the ones that were either generic or very large (making it hard to read code)
- shuffled around relative imports and some function names ("clean_str" vs. "clean")
Some actual behavioral changes:
- remove some Datacite-specific license slugs
- stop trying to fix double-slashes in DOIs, that was causing more harm than help (some DOIs do actually have double-slashes!)
- remove some excess metadata from datacite 'extra' fields
|
| | |
| |
| |
| |
| |
| |
| | |
- MAX_ABSTRACT_LENGTH set in a single place (importer common)
- merge datacite license slug table in to common table, removing some
TDM-specific licenses (which do not apply in the context of preserving
the full work)
|
| | | |
|
| |/
|
|
|
|
|
|
|
|
|
| |
The intent of this change is to start updating Pubmed metadata records
when a PMCID has been assigned, but that ext_id hasn't been recorded in
fatcat yet.
It is likely that this change will result in some additional duplicate
PMCIDs in the catalog. But the principle is that the PMID is the primary
pubmed identifier, and all records with a PMID should have the PMCID
that pubmed indicates, even if there exists another incorrect record.
|
| |
|
|
|
|
|
| |
These mostly add new variable names so that existing variables aren't
overwritten with a new type; delay coercing '{}' or '[]' to 'None' until
the last minute; adding is-not-None checks to conditional clauses; and
similar small changes.
|
| |
|
|
|
| |
This commit just adds the type annotations, doesn't do fixes to code to
make type checking pass.
|
| |
|
|
|
|
| |
These perhaps were used in initial develoment or testing?
fatcat_import.py is the correct way to do these imports, even for
testing/development.
|
| | |
|
| |
|
|
| |
'==' vs 'is'; 'not a in b' vs 'a not in b'; etc
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Oh no!
This bug may actually have had significant negative impact on metadata
in fatcat, in terms of missing container_id associations with pubmed
entities. There are about 500k release entities with a PMID but no
container_id. Of those, 89k have at least a container_name. Unclear how
many would have matched to ISSN-L and thus to a container.
|
| | |
|
| |
|
|
|
|
|
|
| |
One of these (in ingest importer pipeline) is an actual bug, the others
are just changing the syntax to be more explicit/conservative.
The ingest importer bug seems to have resulted in some bad file match
imports; scale of impact is unknown.
|
| |
|
|
|
|
|
| |
The primary motivation for this change is that fatcat *requires* a
non-empty title for each release entity. Pubmed/Medline occasionally
indexes just a VenacularTitle with no ArticleTitle for foreign
publications, and currently those records don't end up in fatcat at all.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
Yikes! Apparently when a tag has child tags, .string will return None
instead of all the strings. .get_text() returns all of it:
https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text
https://www.crummy.com/software/BeautifulSoup/bs4/doc/#string
I've things like identifiers as .string, when we expect only a single
string inside.
|
| |
|
|
|
|
|
| |
This resolves a situation noticed in prod where we were only
importing/updating a single reference per article.
Includes a regression test.
|
| |
|
|
|
|
|
| |
In particular, with daily updates in most cases the DOI will be
registered first, then the entity updated with PMID when that is
available. Often the pubmed metadata will be more complete, with
abstracts etc, and we'll want those improvements.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
