fatcat/python/fatcat_tools/importers, branch v0.4.0 [no description] https://git.bnewbold.net/fatcat/atom?h=v0.4.0 2021-10-13T23:21:31Z dblp import: basic support for handles as identifiers 2021-10-13T23:21:31Z Bryan Newbold bnewbold@robocracy.org 2021-10-13T22:53:32Z urn:sha1:2d755c83895271ad214dcefc234bf7da36e572e3 dblp import: fix typos in identifier parsing 2021-10-13T23:21:31Z Bryan Newbold bnewbold@robocracy.org 2021-10-13T22:39:05Z urn:sha1:e3f892877222309db1c98009d766c658bcb913bb python: partial importer utilization of new schema changes 2021-10-13T23:21:31Z Bryan Newbold bnewbold@robocracy.org 2021-10-13T03:05:57Z urn:sha1:3052b094f2b3c1183abc17c9ca158eb6a8808a42 Merge branch 'bnewbold-ingest-tweaks' into 'master' 2021-10-02T01:22:40Z bnewbold bnewbold@archive.org 2021-10-02T01:22:40Z urn:sha1:571502e21ccfb9f3cae5b0a8f8706f9ce99a08fe ingest importer behavior tweaks See merge request webgroup/fatcat!120 kafka import: optional 'force-flush' mode for some importers 2021-10-02T00:39:43Z Bryan Newbold bnewbold@robocracy.org 2021-10-02T00:39:40Z urn:sha1:b72c18e3518e827bd09044deaadcbf0b0ca50335 Behavior and motivation described in the kafka json import comment. new SPN web (html) importer 2021-10-02T00:33:42Z Bryan Newbold bnewbold@robocracy.org 2021-10-02T00:33:42Z urn:sha1:9618d5146eea046342b69895e68b937a056d2816 ingest importer behavior tweaks 2021-10-01T22:11:40Z Bryan Newbold bnewbold@robocracy.org 2021-10-01T22:11:38Z urn:sha1:6e0736cebcb2b1e5ddbae03127572ad9d1ffca49 - change order of 'want()' checks, so that result counts are clearer - don't require GROBID success for file imports with SPN importer common: more verbose logging (with counts) 2021-10-01T22:07:20Z Bryan Newbold bnewbold@robocracy.org 2021-10-01T22:07:20Z urn:sha1:dd219464cfc90b9b469fd851b48b08668ff17ba8 datacite: skip empty abstracts 2021-10-01T14:56:59Z Martin Czygan martin.czygan@gmail.com 2021-10-01T14:56:59Z urn:sha1:bdc4347acbbdb9f58b7c3abc2578a488de3d0a85 Do not add abstracts where `clean` results in the empty string - this violates a constraint: `either abstract_sha1 or content is required` more consistent and defensive lower-casing of DOIs 2021-06-24T00:51:15Z Bryan Newbold bnewbold@robocracy.org 2021-06-17T23:26:50Z urn:sha1:fa11747574f086e99459914f93d24bad7a8eacce After noticing more upper/lower ambiguity in production. In particular, we have some old ingest requests in sandcrawler DB, which get re-submitted/re-tried, which have capitalized DOIs in the link source id field.