sandcrawler/python, branch master [no description] https://git.bnewbold.net/sandcrawler/atom?h=master 2023-01-05T03:38:16+00:00 pytest: skip warning in gwb 2023-01-05T03:38:16+00:00 Bryan Newbold bnewbold@archive.org 2023-01-05T03:38:16+00:00 urn:sha1:ebc7b6a74b7cc2297c9c291e74ef5466a4753b25 mypy lint fixes 2023-01-05T03:37:07+00:00 Bryan Newbold bnewbold@archive.org 2023-01-05T03:37:07+00:00 urn:sha1:5f73b6428f4b505880ef02429d57f11dc50d98e5 python-specific README file 2023-01-03T03:10:01+00:00 Bryan Newbold bnewbold@archive.org 2023-01-03T03:10:01+00:00 urn:sha1:e433990172c157707d92452652aefe2f21b6a4a0 bump python deps 2022-12-23T23:54:33+00:00 Bryan Newbold bnewbold@archive.org 2022-12-23T23:54:33+00:00 urn:sha1:b7e4629f3c84f35af5ad62346a9480bea957c719 bad pdf hash 2022-12-16T19:17:24+00:00 Bryan Newbold bnewbold@archive.org 2022-12-16T19:17:24+00:00 urn:sha1:b42103eb35f8eb55bda03facb8b14a366fd544c2 sandcrawler: try to handle weird CDX API response 2022-11-01T23:43:40+00:00 Bryan Newbold bnewbold@archive.org 2022-11-01T23:43:38+00:00 urn:sha1:dcc0fe1a61c6816e519cfad95ec12d8abe5ddd29 Hard to debug this because sentry is broken. ingest: more generic OJS support, including pre-prints 2022-10-25T01:35:04+00:00 Bryan Newbold bnewbold@archive.org 2022-10-25T01:35:02+00:00 urn:sha1:a90b604c189bc5655d4a050a9241dfe0b34dbc5b There were some '/article/view/' patterns which can also be, eg, '/preprint/view/'. ingest: more generic PDF fulltext URL patterns 2022-10-24T21:22:39+00:00 Bryan Newbold bnewbold@archive.org 2022-10-24T21:22:39+00:00 urn:sha1:d8f82f5836004d394a419574c50f0636369c94d7 ingest: another wall pattern, and check for walls in more places 2022-10-24T21:22:17+00:00 Bryan Newbold bnewbold@archive.org 2022-10-24T21:22:17+00:00 urn:sha1:5563cb5121c94efcf1819b915e7e7c602215a6e5 ingest: don't prefer WARC over SPN so strongly 2022-10-24T21:17:46+00:00 Bryan Newbold bnewbold@archive.org 2022-10-24T21:17:44+00:00 urn:sha1:4f0d10f4b38534eda673a8dfe28e3a58af9a8a8a We generally prefer an older WARC record over an SPN record, because the lookup is easier. But, this was causing problems with repeated ingest, so demote it. We may want to make this more configurable in the future, so things like HTML sub-resource lookups or bulk ingest won't prefer random new SPN captures.