aboutsummaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/importers
Commit message (Collapse)AuthorAgeFilesLines
* doaj: fix tests now that container_id is requiredBryan Newbold2022-07-191-1/+1
|
* doaj: require container linkage for release importBryan Newbold2022-07-191-0/+4
|
* arxiv: work-around hack for strange titleBryan Newbold2022-07-071-0/+8
|
* fileset ingest: handle missing/partial file-level metadataBryan Newbold2022-04-051-3/+3
|
* ingest importer: improved extra/edit_extra code flowBryan Newbold2022-04-051-20/+13
|
* fileset ingest: remove a TODOBryan Newbold2022-04-041-1/+0
|
* filesets: typo bugfix, and test 'mimetype' on entity, not extraBryan Newbold2022-04-041-1/+1
|
* fileset ingest: fix mimetype handlingBryan Newbold2022-03-311-4/+5
|
* bugfix: logic flow in fileset release checkingBryan Newbold2022-03-231-3/+6
|
* single-file variant of fileset importer for dataset attemptsBryan Newbold2022-03-232-0/+202
|
* fix typo in fileset comparison helperBryan Newbold2022-03-231-1/+1
|
* ingest fileset fixes, and some test coverageBryan Newbold2022-03-232-13/+30
|
* dataset ingest: JSON object fixesBryan Newbold2022-03-221-5/+5
|
* datacite importer: skip container_id for some repository sourcesBryan Newbold2022-02-091-0/+34
|
* doaj importer: TODO note to skip some larger publishersBryan Newbold2022-02-091-0/+4
|
* crossref importer: skip affiliations lacking 'name'Bryan Newbold2021-12-151-0/+3
| | | | Relatedly, we should start handling ROR affiliations in contribs soon.
* chocula importer: handle not-upper-case ISSNsBryan Newbold2021-11-301-2/+6
|
* chocula importer: handle broken ISSNs in extra metadataBryan Newbold2021-11-301-2/+7
|
* chocula importer: tweak counting, conditions for doing updatesBryan Newbold2021-11-301-15/+7
|
* chocula importer: move issne/issnp 'extra' to top-level fields if doing updatesBryan Newbold2021-11-301-0/+6
|
* chocula: don't do name cleanups in importerBryan Newbold2021-11-301-8/+2
| | | | This kind of cleanup should be done in 'chocula' instead.
* codespell fixes in python code (comments)Bryan Newbold2021-11-241-2/+2
|
* Merge branch 'bnewbold-import-refactors' into 'master'bnewbold2021-11-1116-1380/+146
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | import refactors and deprecations Some of these are from old stale branches (the datacite subject metadata patch), but most are from yesterday and today. Sort of a hodge-podge, but the general theme is getting around to deferred cleanups and refactors specific to importer code before making some behavioral changes. The Datacite-specific stuff could use review here. Remove unused/deprecated/dead code: - cdl_dash_dat and wayback_static importers, which were for specific early example entities and have been superseded by other importers - "extid map" sqlite3 feature from several importers, was only used for initial bulk imports (and maybe should not have been used) Refactors: - moved a number of large datastructures out of importer code and into a dedicated static file (`biblio_lookup_tables.py`). Didn't move all, just the ones that were either generic or very large (making it hard to read code) - shuffled around relative imports and some function names ("clean_str" vs. "clean") Some actual behavioral changes: - remove some Datacite-specific license slugs - stop trying to fix double-slashes in DOIs, that was causing more harm than help (some DOIs do actually have double-slashes!) - remove some excess metadata from datacite 'extra' fields
| * refactor importer metadata tables into separate file; move some helpers aroundBryan Newbold2021-11-108-621/+25
| | | | | | | | | | | | | | - MAX_ABSTRACT_LENGTH set in a single place (importer common) - merge datacite license slug table in to common table, removing some TDM-specific licenses (which do not apply in the context of preserving the full work)
| * importers: refactor imports of clean() and other normalization helpersBryan Newbold2021-11-1012-95/+104
| |
| * remove cdl_dash_dat and wayback_static importersBryan Newbold2021-11-103-510/+0
| | | | | | | | | | | | | | | | Cleaning out dead code. These importers were used to create demonstration fileset and webcapture entities early in development. They have been replaced by the fileset and webcapture ingest importers.
| * datacite import: store less subject metadataBryan Newbold2021-11-101-1/+7
| | | | | | | | | | | | | | | | Many of these 'subject' objects have the equivalent of several lines of text, with complex URLs that don't compress well. I think it is fine we have included these thus far instead of parsing more deeply, but going forward I don't think this nested 'extra' metadata is worth the database space.
| * importers: use clean_doi() in many more (all?) importersBryan Newbold2021-11-096-12/+29
| |
| * remove deprecated extid sqlite3 lookup table feature from importersBryan Newbold2021-11-093-160/+0
| | | | | | | | | | | | | | | | This was used during initial bulk imports, but is no longer used and could create serious metadata problems if used accidentially. In retrospect, it also made metadata provenance less transparent, and may have done more harm than good overall.
* | Merge branch 'bnewbold-cleanups-nov2021' into 'master'bnewbold2021-11-111-0/+9
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fatcat metadata cleanups/fixups, November 2021 Three cleanups implemented in this branch: - update non-lowercase DOIs on releases (couple hundred thousand entities) - fix incorrectly imported file/release pairs, on the file entity side (~250k entities) - expand truncated wayback URL timestamps in file entities (up to 10 million entities) Instead of proposals, there are documents for each cleanup in `notes/cleanups/`. Have done spot testing of tens of thousands of entities each in QA, and confident about running in production. Plan is to run updates in the order above. DOI and bugfix updates will go fairly fast; the wayback timestamp updates will go slower, and result in large re-indexing load both in fatcat and scholar, because both release and work entities will get triggered for update when file entities are updated.
| * | imports: generic file cleanup removes exact duplicate URLsBryan Newbold2021-11-091-0/+9
| |/
* / pubmed: allow updates if PMCID does not exist yetBryan Newbold2021-11-101-1/+6
|/ | | | | | | | | | | The intent of this change is to start updating Pubmed metadata records when a PMCID has been assigned, but that ext_id hasn't been recorded in fatcat yet. It is likely that this change will result in some additional duplicate PMCIDs in the catalog. But the principle is that the PMID is the primary pubmed identifier, and all records with a PMID should have the PMCID that pubmed indicates, even if there exists another incorrect record.
* datacite importer: remove unused 'year_only' variableBryan Newbold2021-11-031-2/+3
|
* datacite: add comment about potential date parsing bugBryan Newbold2021-11-031-0/+1
|
* datacite importer: dateparser.date.DateDataParser()Bryan Newbold2021-11-031-1/+1
| | | | Perhaps this was a change when upgrading 'dateparser'?
* more involved type wrangling and fixes for importersBryan Newbold2021-11-033-12/+14
|
* typing: relatively simple type check fixesBryan Newbold2021-11-0314-87/+82
| | | | | | | These mostly add new variable names so that existing variables aren't overwritten with a new type; delay coercing '{}' or '[]' to 'None' until the last minute; adding is-not-None checks to conditional clauses; and similar small changes.
* typing: initial annotations on importersBryan Newbold2021-11-0322-274/+443
| | | | | This commit just adds the type annotations, doesn't do fixes to code to make type checking pass.
* importers: remove unused __main__ routineBryan Newbold2021-11-034-19/+0
| | | | | | These perhaps were used in initial develoment or testing? fatcat_import.py is the correct way to do these imports, even for testing/development.
* lint: resolve existing mypy type errorsBryan Newbold2021-11-023-22/+27
| | | | | | | | | Adds annotations and re-workes dataflow to satisfy existing mypy issues, without adding any additional type annotations to, eg, function signatures. There will probably be many more type errors when annotations are all added.
* re-fix some lint issues after big 'fmt'Bryan Newbold2021-11-021-2/+2
|
* fmt (black): fatcat_tools/Bryan Newbold2021-11-0222-2115/+2578
|
* python: isort everythingBryan Newbold2021-11-0217-41/+70
|
* arabesque import 'hit' field is 1/0, not true/falseBryan Newbold2021-11-021-2/+2
|
* lint: simple, safe inline lint fixesBryan Newbold2021-11-0212-22/+21
| | | | '==' vs 'is'; 'not a in b' vs 'a not in b'; etc
* lint/fmt: remove all 'import *'Bryan Newbold2021-11-025-21/+41
|
* re-fmt all the fatcat_tools __init__ files for readabilityBryan Newbold2021-11-021-17/+39
|
* small python tweaks for annotations, importsBryan Newbold2021-11-022-2/+6
|
* try some type annotationsBryan Newbold2021-11-022-55/+63
|
* fix missing variable in fileset ingestBryan Newbold2021-11-021-2/+1
|