sandcrawler/python, branch bnewbold-refactor-loggging [no description] https://git.bnewbold.net/sandcrawler/atom?h=bnewbold-refactor-loggging 2022-07-12T22:03:29+00:00 WIP: refactor logging calls in ingest pipelines 2022-07-12T22:03:29+00:00 Bryan Newbold bnewbold@archive.org 2022-07-12T22:03:29+00:00 urn:sha1:c15432c0ce52c48efabcd7e3221a5d625ef3e9d0 ingest: IEEE domain is blocking us 2022-07-07T20:17:49+00:00 Bryan Newbold bnewbold@archive.org 2022-07-07T20:17:49+00:00 urn:sha1:695a80a64f02f4c23bb938ecfffeef146344841f ingest: catch more ConnectionErrors (SPN, replay fetch, GROBID) 2022-05-16T22:02:02+00:00 Bryan Newbold bnewbold@archive.org 2022-05-16T22:02:02+00:00 urn:sha1:fcc5a1648d2e49e7002ca569ed668d3318a75584 ingest: skip arxiv.org DOIs, we already direct-ingest 2022-05-11T19:19:48+00:00 Bryan Newbold bnewbold@archive.org 2022-05-11T19:19:48+00:00 urn:sha1:1534ff4d05c6fca460e82b5707fe3fbdc3504e50 python make fmt 2022-05-05T18:21:35+00:00 Bryan Newbold bnewbold@archive.org 2022-05-05T18:21:35+00:00 urn:sha1:a0214959c10a5ecb794d78b189a767ac01c0af48 ingest spn2: fix tests 2022-05-05T18:21:29+00:00 Bryan Newbold bnewbold@archive.org 2022-05-05T18:21:29+00:00 urn:sha1:21ad5cd9942044939c8203dd076ea080b6d55a61 ingest: more loginwall patterns 2022-05-05T18:08:52+00:00 Bryan Newbold bnewbold@archive.org 2022-05-05T18:08:52+00:00 urn:sha1:1f9ca570bd168154a72adcd2454b992dbc7e8d0a ingest_tool: fix arg parsing 2022-05-04T00:35:52+00:00 Bryan Newbold bnewbold@archive.org 2022-05-04T00:35:52+00:00 urn:sha1:1ec661af75f37b3ae5031851f6c452039e08503c switch default kafka-broker host from wbgrp-svc263 to wbgrp-svc350 2022-05-04T00:12:48+00:00 Bryan Newbold bnewbold@archive.org 2022-05-04T00:12:48+00:00 urn:sha1:00ae74378413e87f230c88113ff8163a6f969d63 SPNv2: several fixes for prod throughput 2022-04-26T22:25:23+00:00 Bryan Newbold bnewbold@archive.org 2022-04-26T22:25:20+00:00 urn:sha1:8f9240fed272367669b535b1334e280c588a1791 Most importantly, for some API flags, if the value is not true-thy, do not set the flag at all. Setting any flag was resulting in screenshots and outlinks actually getting created/captured, which was a huge slowdown. Also, check per-user SPNv2 slots available, using API, before requesting an actual capture.