sandcrawler/scalding/src, branch master [no description] https://git.bnewbold.net/sandcrawler/atom?h=master 2021-10-04T20:05:21+00:00 Merge branch 'bnewbold-backfill' into 'master' 2021-10-04T20:05:21+00:00 bnewbold bnewbold@archive.org 2021-10-04T20:05:21+00:00 urn:sha1:57f879c00b00c6cd4051f54662fea3f96f80ad35 CDX Backfill (scalding version) See merge request webgroup/sandcrawler!12 GroupFatcatWorksSubsetJob 2019-08-26T21:25:34+00:00 Bryan Newbold bnewbold@archive.org 2019-08-26T21:25:28+00:00 urn:sha1:d2d545675b3c85c3e1b41fd4bda23230d995bf47 This is a hack-y variant of GroupFatcatWorksSubsetJob which allows setting different left and right sides of the join. The initial application is to re-run work merging with only longtail-oa works on the "left", with the goal of hard-merging these releases into existing releases with actual identifiers (instead of just grouping into works). As a refactor, the normal GroupFatcatWorksJob could just be this with the same file passed as both left and right, though that requires twice as much JSON parsing/filtering. please command for groupworksfatcat 2019-08-11T03:20:34+00:00 Bryan Newbold bnewbold@archive.org 2019-08-11T02:50:18+00:00 urn:sha1:952457f9fae1cc25cdeeefc00e19ae20cf86c659 FatcatScorable and ScoreSelfFatcat job 2019-08-11T02:50:21+00:00 Bryan Newbold bnewbold@archive.org 2019-08-03T00:11:57+00:00 urn:sha1:ea9e8990139973d6f5fdf52a470bf6516c7d8c2f add fatcat ident fields in prep for self-scoring job 2019-08-11T02:50:21+00:00 Bryan Newbold bnewbold@archive.org 2019-08-03T00:11:31+00:00 urn:sha1:ca725ffd9efe847905afb918ff324b421a4d8859 scalding dump-grobid-status-code job 2019-04-12T21:19:29+00:00 Bryan Newbold bnewbold@archive.org 2019-04-12T20:48:27+00:00 urn:sha1:d93ebaa691f8b200a5761850b4533a153cb457ee fix typos in DumpGrobidXmlJob 2018-10-31T01:03:24+00:00 Bryan Newbold bnewbold@archive.org 2018-10-31T01:03:24+00:00 urn:sha1:4cb7c1bdc6710a11c869f3d398ed39762644395c quick and dirty GROBID XML dumper 2018-10-30T21:43:57+00:00 Bryan Newbold bnewbold@archive.org 2018-10-30T21:40:53+00:00 urn:sha1:3105e9fc5799063027f3273048eea27f906d4c66 new DumpGrobidMetaInsertableJob 2018-09-23T03:31:39+00:00 Bryan Newbold bnewbold@archive.org 2018-09-23T03:31:39+00:00 urn:sha1:51ca75189b7c36577f8e80b9db1f66259f0f6178 new simple file metadata dump script 2018-09-14T04:11:57+00:00 Bryan Newbold bnewbold@archive.org 2018-09-14T04:11:57+00:00 urn:sha1:5521b1b520a550373369da8b9cbd36148e071115