fatcat/python/fatcat_tools/workers, branch v0.3.3 [no description] https://git.bnewbold.net/fatcat/atom?h=v0.3.3 2020-12-16T23:00:18+00:00 entity update worker: treat fileset and webcapture updates like file updates 2020-12-16T23:00:18+00:00 Bryan Newbold bnewbold@robocracy.org 2020-12-16T22:58:07+00:00 urn:sha1:f60ba0ea04081ac0095c12d8ecbaa48b3da74aee When webcapture or fileset entities are updated, then the release entities associated with them also need to be updated (and work entities, recursively). A TODO is to handle the case where a release_id is *removed* as well as *added*, and reprocess the releases in that case as well. entity updates: don't ingest JSTOR DOI prefixes 2020-10-23T20:17:34+00:00 Bryan Newbold bnewbold@robocracy.org 2020-10-23T20:17:34+00:00 urn:sha1:bf77adc854022213951daec14bd904f483f21202 entity updater: new work update feed (ident and changelog metadata only) 2020-10-16T23:41:16+00:00 Bryan Newbold bnewbold@robocracy.org 2020-10-16T03:53:11+00:00 urn:sha1:99c54764f0cbbb05409001b4af83182842f4e52d ingest: default to crawl protocols.io DOIs 2020-09-11T01:49:35+00:00 Bryan Newbold bnewbold@robocracy.org 2020-09-11T01:49:35+00:00 urn:sha1:3bac4de14219b5d8a3ef8f5919c4f004663424c0 entity updater: handle doi=None case better 2020-08-14T23:09:32+00:00 Bryan Newbold bnewbold@robocracy.org 2020-08-14T23:09:32+00:00 urn:sha1:e024a38bd29686eaa687e54cde1dc255ba6af52b entity updater: es['publisher_type'] not always set 2020-08-14T23:05:57+00:00 Bryan Newbold bnewbold@robocracy.org 2020-08-14T23:05:55+00:00 urn:sha1:67c2dd909de3c5dada6efe8db2f59ed09e76d439 This is a small bugfix for a production issue. entity update: change big5 ingest behavior 2020-08-11T22:45:39+00:00 Bryan Newbold bnewbold@robocracy.org 2020-08-11T22:45:36+00:00 urn:sha1:03d2004717d36962aef1bd373d59ce799d7db9ab In addition to changing the OA default, this was the main intended behavior change in this group of commits: want to ingest fewer attempts that we *expect* to fail, but default to ingest/crawl attempt if we are uncertain. This is because there is a long tail of journals that register DOIs and are defacto OA (fulltext is available), but we don't have metadata indicating them as such. entity update: default to ingest non-OA works 2020-08-11T22:32:28+00:00 Bryan Newbold bnewbold@robocracy.org 2020-08-11T22:23:25+00:00 urn:sha1:5eddc9b9aefbd7ae197d441b8a7af1fded940e2d entity update: skip ingest of figshare+zenodo 'group' DOIs 2020-08-11T22:32:28+00:00 Bryan Newbold bnewbold@robocracy.org 2020-08-11T21:52:47+00:00 urn:sha1:2a492914082444690f853a55ab1394fc0cf50108 update crawl blocklist for SPNv2 requests which mostly fail 2020-08-10T22:07:19+00:00 Bryan Newbold bnewbold@robocracy.org 2020-08-10T22:07:19+00:00 urn:sha1:e9dd3c73f036d3fba2680eeaff8e62ecf2dbf9a1