[no description] https://git.bnewbold.net/fatcat/atom?h=v0.5.0 2021-11-18T00:36:16+00:00 2021-11-18T00:36:16+00:00 Bryan Newbold bnewbold@robocracy.org 2021-11-18T00:36:16+00:00 urn:sha1:09a1829a10caf9759762b41c36169c0b88cd5ed7 2021-11-11T01:12:18+00:00 bnewbold bnewbold@archive.org 2021-11-11T01:12:18+00:00 urn:sha1:6ad9d24e4d7d901d6fc394e6e91575f6acba7ff4 import refactors and deprecations Some of these are from old stale branches (the datacite subject metadata patch), but most are from yesterday and today. Sort of a hodge-podge, but the general theme is getting around to deferred cleanups and refactors specific to importer code before making some behavioral changes. The Datacite-specific stuff could use review here. Remove unused/deprecated/dead code: - cdl_dash_dat and wayback_static importers, which were for specific early example entities and have been superseded by other importers - "extid map" sqlite3 feature from several importers, was only used for initial bulk imports (and maybe should not have been used) Refactors: - moved a number of large datastructures out of importer code and into a dedicated static file (`biblio_lookup_tables.py`). Didn't move all, just the ones that were either generic or very large (making it hard to read code) - shuffled around relative imports and some function names ("clean_str" vs. "clean") Some actual behavioral changes: - remove some Datacite-specific license slugs - stop trying to fix double-slashes in DOIs, that was causing more harm than help (some DOIs do actually have double-slashes!) - remove some excess metadata from datacite 'extra' fields 2021-11-11T01:11:49+00:00 bnewbold bnewbold@archive.org 2021-11-11T01:11:49+00:00 urn:sha1:7e3f91f1a49ea85707cae31125021ba761f5373d Fatcat metadata cleanups/fixups, November 2021 Three cleanups implemented in this branch: - update non-lowercase DOIs on releases (couple hundred thousand entities) - fix incorrectly imported file/release pairs, on the file entity side (~250k entities) - expand truncated wayback URL timestamps in file entities (up to 10 million entities) Instead of proposals, there are documents for each cleanup in `notes/cleanups/`. Have done spot testing of tens of thousands of entities each in QA, and confident about running in production. Plan is to run updates in the order above. DOI and bugfix updates will go fairly fast; the wayback timestamp updates will go slower, and result in large re-indexing load both in fatcat and scholar, because both release and work entities will get triggered for update when file entities are updated. 2021-11-10T22:14:48+00:00 Bryan Newbold bnewbold@robocracy.org 2021-11-10T22:12:59+00:00 urn:sha1:1d1700678747ae711afbf105b962c5a1db3e7196 2021-11-10T21:52:43+00:00 Bryan Newbold bnewbold@robocracy.org 2021-11-10T21:52:39+00:00 urn:sha1:ddc757bc1d5c610f42e9f5f10a4f060f517b66ca - MAX_ABSTRACT_LENGTH set in a single place (importer common) - merge datacite license slug table in to common table, removing some TDM-specific licenses (which do not apply in the context of preserving the full work) 2021-11-10T21:28:28+00:00 Bryan Newbold bnewbold@robocracy.org 2021-11-10T21:23:12+00:00 urn:sha1:16e9979a6f347b49764c1141209e84083ea81057 2021-11-10T21:08:25+00:00 Bryan Newbold bnewbold@robocracy.org 2021-11-10T21:08:23+00:00 urn:sha1:ab4e1355bf93e3755985f1b5cd2589a78601d253 Cleaning out dead code. These importers were used to create demonstration fileset and webcapture entities early in development. They have been replaced by the fileset and webcapture ingest importers. 2021-11-10T21:04:44+00:00 Bryan Newbold bnewbold@robocracy.org 2020-08-11T00:27:26+00:00 urn:sha1:c133f3077aa975aa4706a8e5ca894fc1b71fbc67 Many of these 'subject' objects have the equivalent of several lines of text, with complex URLs that don't compress well. I think it is fine we have included these thus far instead of parsing more deeply, but going forward I don't think this nested 'extra' metadata is worth the database space. 2021-11-10T20:55:49+00:00 Bryan Newbold bnewbold@robocracy.org 2020-03-29T03:17:29+00:00 urn:sha1:13b107d8397c47a3159800a34307063f8e2b509e The intent of this change is to start updating Pubmed metadata records when a PMCID has been assigned, but that ext_id hasn't been recorded in fatcat yet. It is likely that this change will result in some additional duplicate PMCIDs in the catalog. But the principle is that the PMID is the primary pubmed identifier, and all records with a PMID should have the PMCID that pubmed indicates, even if there exists another incorrect record. 2021-11-10T04:34:55+00:00 Bryan Newbold bnewbold@robocracy.org 2021-11-10T04:31:21+00:00 urn:sha1:86056629e7778543dc31c962e7e3f273e1ec48e6