sandcrawler/sql, branch bnewbold-persist-grobid-errors [no description] https://git.bnewbold.net/sandcrawler/atom?h=bnewbold-persist-grobid-errors 2020-01-29T03:10:40Z grobid persist: if status_code is not set, default to 0 2020-01-29T03:10:40Z Bryan Newbold bnewbold@archive.org 2020-01-29T03:06:25Z urn:sha1:480dc3fa20102ba0a15013954e76d3b1f826026c We have to set something currently because of a NOT NULL constraint on the table. Originally I thought we would just not record rows if there was an error, and that is still sort of a valid stance. However, when doing bulk GROBID-ing from cdx table, there exist some "bad" CDX rows which cause wayback or petabox errors. We should fix bugs or delete these rows as a cleanup, but until that happens we should record the error state so we don't loop forever. One danger of this commit is that we can clobber existing good rows with new errors rapidly if there is wayback downtime or something like that. sql stats: typo fix 2020-01-29T03:10:40Z Bryan Newbold bnewbold@archive.org 2020-01-29T03:10:21Z urn:sha1:9f53880c746b9fd84261e3ab7dbbee81501df394 sql howto: database dumps 2020-01-29T03:10:40Z Bryan Newbold bnewbold@archive.org 2020-01-29T03:09:54Z urn:sha1:46171ed3a86e90d8c519a8ce94f379309936fbeb clarify ingest result schema and semantics 2020-01-15T21:54:02Z Bryan Newbold bnewbold@archive.org 2020-01-15T21:54:02Z urn:sha1:d06fd45e3c86cb080ad7724f3fc7575750a9cd69 database stats 2020-01-15T07:52:33Z Bryan Newbold bnewbold@archive.org 2020-01-15T07:52:33Z urn:sha1:9c97db0ffcb2350a7231ab388c643d953d77274f sql: more cool random queries 2020-01-03T02:13:03Z Bryan Newbold bnewbold@archive.org 2020-01-03T02:11:24Z urn:sha1:fdff17d2cdf4ac92a0458403bb4ca0b073a7752b SQL docs update for diesel change 2020-01-03T02:12:58Z Bryan Newbold bnewbold@archive.org 2019-12-25T00:35:45Z urn:sha1:b603a43e45eb3afa01efd0902c5af56f29d979a2 move SQL schema to diesel migration pattern 2020-01-03T02:12:58Z Bryan Newbold bnewbold@archive.org 2019-12-19T06:08:38Z urn:sha1:9beb3caee51c6bb0403c658a71c965dde4c8e55b add some GROBID metadata schema docs to SQL schema 2019-12-12T02:20:13Z Bryan Newbold bnewbold@archive.org 2019-12-12T02:20:13Z urn:sha1:91f5f53c90742c80890e3bd44fdc9044555b8209 add note to CDX backfill script that we should be filtering (oops) 2019-11-12T21:22:43Z Bryan Newbold bnewbold@archive.org 2019-11-12T21:22:43Z urn:sha1:9529cbb2660897ce3ffe3986f60eafbf3596495d