sandcrawler/notes/ingest, branch trawler

sandcrawler/notes/ingest, branch trawler [no description] https://git.bnewbold.net/sandcrawler/atom?h=trawler 2021-12-02T03:06:00+00:00 commit old patch crawl notes 2021-12-02T03:06:00+00:00 Bryan Newbold bnewbold@archive.org 2021-12-02T03:06:00+00:00 urn:sha1:85a9c9008ab66680047fb151996c55566d56cbe3 daily OA crawl improvements/notes 2021-09-08T19:16:44+00:00 Bryan Newbold bnewbold@archive.org 2021-09-08T19:16:44+00:00 urn:sha1:ce25d59845083ca0beab98144b0c43bfc4254d6d OAI-PMH patch and ingest improvement notes 2021-09-04T01:34:33+00:00 Bryan Newbold bnewbold@archive.org 2021-09-04T01:34:33+00:00 urn:sha1:d749a7a6a1c1d439596c5d053daf904638b4dbc2 commit old patch crawl notes (dec 2020) 2021-09-04T01:33:23+00:00 Bryan Newbold bnewbold@archive.org 2021-09-04T01:33:23+00:00 urn:sha1:c147ad35fa8ec59b8c015f8badae67c525f65253 commit old arxiv ingest notes 2021-09-04T01:32:39+00:00 Bryan Newbold bnewbold@archive.org 2021-09-04T01:32:39+00:00 urn:sha1:c3cbab57fc5b27a5add399dd27dff0a91c9d9fa1 commit old patch notes (will rework) 2021-09-03T17:36:49+00:00 Bryan Newbold bnewbold@archive.org 2021-09-03T16:04:55+00:00 urn:sha1:f074a6aafd9af06866829d35555afe10286126fb MAG post-crawl stats (5m+ new PDFs crawled successfully) 2021-09-02T23:31:23+00:00 Bryan Newbold bnewbold@archive.org 2021-09-02T23:31:23+00:00 urn:sha1:ffd6cd86bb8a4756d123decaa5f2ef03428f208f MAG and OAI-PMH crawl/processing notes 2021-08-13T20:57:57+00:00 Bryan Newbold bnewbold@archive.org 2021-08-13T20:57:57+00:00 urn:sha1:eab9b929a05da3fa25f4bfaffd84bb0d7b219c73 2021-07 unpaywall crawl wrap-up notes 2021-07-30T16:33:24+00:00 Bryan Newbold bnewbold@archive.org 2021-07-30T16:33:24+00:00 urn:sha1:ccb2c72c170d6736af675734906c8957ee176a8b unpaywall 2021-07 crawl partial notes 2021-07-14T22:45:53+00:00 Bryan Newbold bnewbold@archive.org 2021-07-14T22:45:53+00:00 urn:sha1:23374f2d10914e06c67e7c0c1f9c37ba98e36eeb