sandcrawler, branch trawler [no description] https://git.bnewbold.net/sandcrawler/atom?h=trawler 2021-12-09T22:12:18Z notes on re-GROBID-ing (and re-extracting) some files 2021-12-09T22:12:18Z Bryan Newbold bnewbold@archive.org 2021-12-09T22:12:18Z urn:sha1:e5c021bfeb03c50924160616dc64d44617d45933 grobid: set a maximum file size (256 MByte) 2021-12-08T03:44:53Z Bryan Newbold bnewbold@archive.org 2021-12-08T03:44:53Z urn:sha1:89b5f51e57d3a0cc043640262e396e28297e7c00 worker: add kafka_group_suffix option 2021-12-08T03:10:23Z Bryan Newbold bnewbold@archive.org 2021-12-08T03:09:54Z urn:sha1:833f9bb5181419ca9f5af0f9ba0e2e047ee164d4 ingest tool: allow configuration of GROBID endpoint 2021-12-08T03:10:23Z Bryan Newbold bnewbold@archive.org 2021-12-04T00:38:28Z urn:sha1:5c82ee1b965e1f3901294c752d8b2d24c6bdc974 2021-12-02 database table size stats 2021-12-08T03:10:23Z Bryan Newbold bnewbold@archive.org 2021-12-04T00:37:45Z urn:sha1:57441fda8be33594898c1836fba22b12fb3e94e8 sandcrawler SQL dump and upload updates 2021-12-08T03:10:23Z Bryan Newbold bnewbold@archive.org 2021-12-04T00:37:22Z urn:sha1:9a32daa502e2c729cf896ae5e7cb27a3aa6bb68d update fatcat_file SQL table schema, and add backfill notes 2021-12-08T03:10:23Z Bryan Newbold bnewbold@archive.org 2021-12-02T03:06:33Z urn:sha1:3f01f73563f40869c82b7ad3e21c4183fdee8206 update fatcat_file SQL table schema, and add backfill notes 2021-12-02T03:06:33Z Bryan Newbold bnewbold@archive.org 2021-12-02T03:06:33Z urn:sha1:4f02b47f57364195e7302ec80565ce51fd20048d commit old patch crawl notes 2021-12-02T03:06:00Z Bryan Newbold bnewbold@archive.org 2021-12-02T03:06:00Z urn:sha1:85a9c9008ab66680047fb151996c55566d56cbe3 Revert "pipenv: update deps" 2021-12-02T01:37:19Z Bryan Newbold bnewbold@archive.org 2021-12-02T01:37:10Z urn:sha1:6777004f20f742134105c18d6bae06d0ce362d50 This reverts commit 7a5b203dbb37958a452eb1be3bd1bf8ed94cbbce. There is a problem with `internetarchive` 2.2.0, so reverting for now.