| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
This kind of cleanup should be done in 'chocula' instead.
|
| | |
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
import refactors and deprecations
Some of these are from old stale branches (the datacite subject metadata patch), but most are from yesterday and today. Sort of a hodge-podge, but the general theme is getting around to deferred cleanups and refactors specific to importer code before making some behavioral changes.
The Datacite-specific stuff could use review here.
Remove unused/deprecated/dead code:
- cdl_dash_dat and wayback_static importers, which were for specific early example entities and have been superseded by other importers
- "extid map" sqlite3 feature from several importers, was only used for initial bulk imports (and maybe should not have been used)
Refactors:
- moved a number of large datastructures out of importer code and into a dedicated static file (`biblio_lookup_tables.py`). Didn't move all, just the ones that were either generic or very large (making it hard to read code)
- shuffled around relative imports and some function names ("clean_str" vs. "clean")
Some actual behavioral changes:
- remove some Datacite-specific license slugs
- stop trying to fix double-slashes in DOIs, that was causing more harm than help (some DOIs do actually have double-slashes!)
- remove some excess metadata from datacite 'extra' fields
|
| | |
| |
| |
| |
| |
| |
| | |
- MAX_ABSTRACT_LENGTH set in a single place (importer common)
- merge datacite license slug table in to common table, removing some
TDM-specific licenses (which do not apply in the context of preserving
the full work)
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Cleaning out dead code.
These importers were used to create demonstration fileset and webcapture
entities early in development. They have been replaced by the fileset
and webcapture ingest importers.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Many of these 'subject' objects have the equivalent of several lines of
text, with complex URLs that don't compress well. I think it is fine we
have included these thus far instead of parsing more deeply, but going
forward I don't think this nested 'extra' metadata is worth the database
space.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This was used during initial bulk imports, but is no longer used and
could create serious metadata problems if used accidentially.
In retrospect, it also made metadata provenance less transparent, and
may have done more harm than good overall.
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Fatcat metadata cleanups/fixups, November 2021
Three cleanups implemented in this branch:
- update non-lowercase DOIs on releases (couple hundred thousand entities)
- fix incorrectly imported file/release pairs, on the file entity side (~250k entities)
- expand truncated wayback URL timestamps in file entities (up to 10 million entities)
Instead of proposals, there are documents for each cleanup in `notes/cleanups/`.
Have done spot testing of tens of thousands of entities each in QA, and confident about running in production.
Plan is to run updates in the order above. DOI and bugfix updates will go fairly fast; the wayback timestamp updates will go slower, and result in large re-indexing load both in fatcat and scholar, because both release and work entities will get triggered for update when file entities are updated.
|
| | |/ |
|
| |/
|
|
|
|
|
|
|
|
|
| |
The intent of this change is to start updating Pubmed metadata records
when a PMCID has been assigned, but that ext_id hasn't been recorded in
fatcat yet.
It is likely that this change will result in some additional duplicate
PMCIDs in the catalog. But the principle is that the PMID is the primary
pubmed identifier, and all records with a PMID should have the PMCID
that pubmed indicates, even if there exists another incorrect record.
|
| | |
|
| | |
|
| |
|
|
| |
Perhaps this was a change when upgrading 'dateparser'?
|
| | |
|
| |
|
|
|
|
|
| |
These mostly add new variable names so that existing variables aren't
overwritten with a new type; delay coercing '{}' or '[]' to 'None' until
the last minute; adding is-not-None checks to conditional clauses; and
similar small changes.
|
| |
|
|
|
| |
This commit just adds the type annotations, doesn't do fixes to code to
make type checking pass.
|
| |
|
|
|
|
| |
These perhaps were used in initial develoment or testing?
fatcat_import.py is the correct way to do these imports, even for
testing/development.
|
| |
|
|
|
|
|
|
|
| |
Adds annotations and re-workes dataflow to satisfy existing mypy issues,
without adding any additional type annotations to, eg, function
signatures.
There will probably be many more type errors when annotations are all
added.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
'==' vs 'is'; 'not a in b' vs 'a not in b'; etc
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\
| |
| |
| |
| | |
ingest importer behavior tweaks
See merge request webgroup/fatcat!120
|
| | |
| |
| |
| | |
Behavior and motivation described in the kafka json import comment.
|
| | | |
|
| | |
| |
| |
| |
| | |
- change order of 'want()' checks, so that result counts are clearer
- don't require GROBID success for file imports with SPN
|
| | | |
|
| |/
|
|
|
| |
Do not add abstracts where `clean` results in the empty string - this
violates a constraint: `either abstract_sha1 or content is required`
|
| |
|
|
|
|
|
| |
After noticing more upper/lower ambiguity in production. In particular,
we have some old ingest requests in sandcrawler DB, which get
re-submitted/re-tried, which have capitalized DOIs in the link source id
field.
|
| |
|
|
|
| |
Caused by a partial "title entry without title" coming *first* (e.g. just
holding, e.g. a language, like: {'lang': 'da'}
|
