sandcrawler/python/tests, branch bnewbold-persist-grobid-errors [no description] https://git.bnewbold.net/sandcrawler/atom?h=bnewbold-persist-grobid-errors 2020-01-17T20:22:34+00:00 ingest: add URL blocklist feature 2020-01-17T20:22:34+00:00 Bryan Newbold bnewbold@archive.org 2020-01-17T20:12:05+00:00 urn:sha1:58f744e97c8f3f1a3472aa821f4518d7d139e850 And, temporarily, block zenodo and figshare. clarify ingest result schema and semantics 2020-01-15T21:54:02+00:00 Bryan Newbold bnewbold@archive.org 2020-01-15T21:54:02+00:00 urn:sha1:d06fd45e3c86cb080ad7724f3fc7575750a9cd69 add postgrest checks to test mocks 2020-01-15T01:15:22+00:00 Bryan Newbold bnewbold@archive.org 2020-01-15T01:15:22+00:00 urn:sha1:6a56b922ced9013c6d09027d771dc8c4fc80421e tests: don't use localhost as a responses mock host 2020-01-15T01:15:01+00:00 Bryan Newbold bnewbold@archive.org 2020-01-15T01:15:01+00:00 urn:sha1:af2ffc0cea2d03a86f3ecf6c8dd0bd106a19b851 SPNv2 doesn't support FTP; add a live test for non-revist FTP 2020-01-15T00:06:19+00:00 Bryan Newbold bnewbold@archive.org 2020-01-15T00:06:19+00:00 urn:sha1:4bb341270907f91b0475a7cdb00a7d280a80c06c more ftp status 226 support 2020-01-15T00:05:41+00:00 Bryan Newbold bnewbold@archive.org 2020-01-15T00:05:41+00:00 urn:sha1:ba6f16a02cfde0e4acb499c00b456b42472c0b00 add live tests for ftp, revisits 2020-01-14T23:53:00+00:00 Bryan Newbold bnewbold@archive.org 2020-01-14T23:53:00+00:00 urn:sha1:21599839802b8ef3a84ffe90855f7bceaaa12a0d more live tests (for regressions) 2020-01-11T00:04:13+00:00 Bryan Newbold bnewbold@archive.org 2020-01-11T00:04:13+00:00 urn:sha1:89abcd4da267665d363e558ab54ec3272d67c6e4 refactor ingest to a loop, allowing multiple hops 2020-01-10T01:31:08+00:00 Bryan Newbold bnewbold@archive.org 2020-01-10T01:31:08+00:00 urn:sha1:24185837a47f305757a5c783b95ca25b709f66e3 add (skipped) live tests for wayback services 2020-01-10T00:51:59+00:00 Bryan Newbold bnewbold@archive.org 2020-01-10T00:51:59+00:00 urn:sha1:00cf33a1c230c8ce5dcda41aba5dcc6a88264d46