[no description] https://git.bnewbold.net/fatcat/atom?h=v0.5.1 2021-12-15T20:48:15Z 2021-12-15T20:48:15Z Bryan Newbold bnewbold@robocracy.org 2021-12-15T20:48:13Z urn:sha1:b9d5c3853fe70663b0eff101c9ac4a1cbfd354e7 Relatedly, we should start handling ROR affiliations in contribs soon. 2021-12-01T01:25:38Z Bryan Newbold bnewbold@robocracy.org 2021-12-01T01:25:38Z urn:sha1:3ba6a6b44080dd2ce2636dde853faf800f0b7628 2021-12-01T00:56:18Z Bryan Newbold bnewbold@robocracy.org 2021-12-01T00:56:18Z urn:sha1:b1efd59c2cad275d126a1bde67c11430d71878db 2021-12-01T00:25:09Z Bryan Newbold bnewbold@robocracy.org 2021-11-24T23:15:44Z urn:sha1:1177dafb9b185c7b749ff95ded1a0720792fbb5e 2021-12-01T00:25:09Z Bryan Newbold bnewbold@robocracy.org 2021-11-24T23:15:22Z urn:sha1:0d820674f17a03feee73ce38debf494c79003483 2021-12-01T00:25:09Z Bryan Newbold bnewbold@robocracy.org 2021-11-24T23:11:56Z urn:sha1:2260eefcb5b8f2d3d29be8e98b292b5e7d326844 This kind of cleanup should be done in 'chocula' instead. 2021-11-24T23:45:33Z Bryan Newbold bnewbold@robocracy.org 2021-11-24T23:45:33Z urn:sha1:5d3e62f757e968d28e90bdc4f0abe2ee07a09f3f 2021-11-11T01:12:18Z bnewbold bnewbold@archive.org 2021-11-11T01:12:18Z urn:sha1:6ad9d24e4d7d901d6fc394e6e91575f6acba7ff4 import refactors and deprecations Some of these are from old stale branches (the datacite subject metadata patch), but most are from yesterday and today. Sort of a hodge-podge, but the general theme is getting around to deferred cleanups and refactors specific to importer code before making some behavioral changes. The Datacite-specific stuff could use review here. Remove unused/deprecated/dead code: - cdl_dash_dat and wayback_static importers, which were for specific early example entities and have been superseded by other importers - "extid map" sqlite3 feature from several importers, was only used for initial bulk imports (and maybe should not have been used) Refactors: - moved a number of large datastructures out of importer code and into a dedicated static file (`biblio_lookup_tables.py`). Didn't move all, just the ones that were either generic or very large (making it hard to read code) - shuffled around relative imports and some function names ("clean_str" vs. "clean") Some actual behavioral changes: - remove some Datacite-specific license slugs - stop trying to fix double-slashes in DOIs, that was causing more harm than help (some DOIs do actually have double-slashes!) - remove some excess metadata from datacite 'extra' fields 2021-11-11T01:11:49Z bnewbold bnewbold@archive.org 2021-11-11T01:11:49Z urn:sha1:7e3f91f1a49ea85707cae31125021ba761f5373d Fatcat metadata cleanups/fixups, November 2021 Three cleanups implemented in this branch: - update non-lowercase DOIs on releases (couple hundred thousand entities) - fix incorrectly imported file/release pairs, on the file entity side (~250k entities) - expand truncated wayback URL timestamps in file entities (up to 10 million entities) Instead of proposals, there are documents for each cleanup in `notes/cleanups/`. Have done spot testing of tens of thousands of entities each in QA, and confident about running in production. Plan is to run updates in the order above. DOI and bugfix updates will go fairly fast; the wayback timestamp updates will go slower, and result in large re-indexing load both in fatcat and scholar, because both release and work entities will get triggered for update when file entities are updated. 2021-11-10T21:52:43Z Bryan Newbold bnewbold@robocracy.org 2021-11-10T21:52:39Z urn:sha1:ddc757bc1d5c610f42e9f5f10a4f060f517b66ca - MAX_ABSTRACT_LENGTH set in a single place (importer common) - merge datacite license slug table in to common table, removing some TDM-specific licenses (which do not apply in the context of preserving the full work)