sandcrawler/python, branch trawler [no description] https://git.bnewbold.net/sandcrawler/atom?h=trawler 2021-12-08T03:44:53+00:00 grobid: set a maximum file size (256 MByte) 2021-12-08T03:44:53+00:00 Bryan Newbold bnewbold@archive.org 2021-12-08T03:44:53+00:00 urn:sha1:89b5f51e57d3a0cc043640262e396e28297e7c00 worker: add kafka_group_suffix option 2021-12-08T03:10:23+00:00 Bryan Newbold bnewbold@archive.org 2021-12-08T03:09:54+00:00 urn:sha1:833f9bb5181419ca9f5af0f9ba0e2e047ee164d4 ingest tool: allow configuration of GROBID endpoint 2021-12-08T03:10:23+00:00 Bryan Newbold bnewbold@archive.org 2021-12-04T00:38:28+00:00 urn:sha1:5c82ee1b965e1f3901294c752d8b2d24c6bdc974 Revert "pipenv: update deps" 2021-12-02T01:37:19+00:00 Bryan Newbold bnewbold@archive.org 2021-12-02T01:37:10+00:00 urn:sha1:6777004f20f742134105c18d6bae06d0ce362d50 This reverts commit 7a5b203dbb37958a452eb1be3bd1bf8ed94cbbce. There is a problem with `internetarchive` 2.2.0, so reverting for now. pipenv: update deps 2021-12-02T01:14:20+00:00 Bryan Newbold bnewbold@archive.org 2021-12-02T01:14:20+00:00 urn:sha1:7a5b203dbb37958a452eb1be3bd1bf8ed94cbbce add CDX sha1hex lookup/fetch helper script 2021-11-30T23:29:41+00:00 Bryan Newbold bnewbold@archive.org 2021-11-30T23:29:41+00:00 urn:sha1:0328598e3b643edd0a2033ca97c607f596dfb092 codespell typos in python (comments) 2021-11-25T00:05:24+00:00 Bryan Newbold bnewbold@archive.org 2021-11-25T00:05:11+00:00 urn:sha1:dfd13be5a7ac87b8b6c186986624f97da02b8923 html_meta: actual typo in code (CSS selector) caught by codespell 2021-11-25T00:04:43+00:00 Bryan Newbold bnewbold@archive.org 2021-11-25T00:04:43+00:00 urn:sha1:a6cfb01063da8a5172d38d2da190a25e7d070993 ingest tool: new backfill mode 2021-11-17T00:16:14+00:00 Bryan Newbold bnewbold@archive.org 2021-11-17T00:16:14+00:00 urn:sha1:b4ca684c83d77a9fc6e7844ea8c45dfcb72aacb4 make fmt 2021-11-17T00:10:08+00:00 Bryan Newbold bnewbold@archive.org 2021-11-17T00:10:08+00:00 urn:sha1:9d8da0ec55a3d901bf7ffad6d86fd8cc08f89b6f