fatcat/python/fatcat_tools/harvest, branch v0.3.3[no description]https://git.bnewbold.net/fatcat/atom?h=v0.3.32020-08-10T17:58:12Zharvest: datacite API yields HTTP 200 with broken JSON2020-08-10T17:58:12ZMartin Czyganmartin.czygan@gmail.com2020-08-10T17:55:14Zurn:sha1:e18d48642cecb55d9f2270f9048953a7b543472e
As a first step: log response body for debugging.
arxiv: do retry five times of HTTP 5032020-07-09T22:54:55ZMartin Czyganmartin.czygan@gmail.com2020-07-09T22:54:55Zurn:sha1:c403cb4a1f20bd056008f68f71b374bde1e089b5lint (flake8) tool python files2020-07-02T01:35:24ZBryan Newboldbnewbold@robocracy.org2020-07-02T01:35:24Zurn:sha1:30905f1effb33c3ef193d084120aa3fbd75d0b9bharvest: fail on HTTP 4002020-05-29T17:07:50ZMartin Czyganmartin.czygan@gmail.com2020-05-29T17:00:30Zurn:sha1:34a64b5d8c470ae2627458d791239cfc4d66d6b3
In the past harvest of datacite resulted in occasional HTTP 400.
Meanwhile, various API bugs have been fixed (most recently:
https://github.com/datacite/lupo/pull/537,
https://github.com/datacite/datacite/issues/1038). Downside of ignoring
this error was that state lives in kafka, which has limited support for
deletion of arbitrary messages from a topic.
rename HarvestState.next() to HarvestState.next_span()2020-05-27T02:09:55ZBryan Newboldbnewbold@robocracy.org2020-05-27T02:01:28Zurn:sha1:670aed3800873869550b477846f48cb2b4193005
"span" short for "timespan" to harvest; there may be a better name to
use.
Motivation for this is to work around a pylint erorr that .next() was
not callable. This might be a bug with pylint, but .next() is also a
very generic name.
HACK: skip pylint errors on lines that seem to be fine2020-05-22T23:52:40ZBryan Newboldbnewbold@robocracy.org2020-05-22T23:52:38Zurn:sha1:99dfb5031511eb730c3397b18f2bd6e537b67e9a
It seems to be an inadvertantly ugraded version of pylint saying that
these lines are not-callable.
crossref: switch from index-date to update-date2020-03-31T04:23:11ZBryan Newboldbnewbold@robocracy.org2020-03-31T03:56:04Zurn:sha1:851c40143d44a73a92ff2c9556b3a63f29668c2d
This goes against what the API docs recommend, but we are currently far
behind on updates and need to catch up. Other than what the docs say,
this seems to be consistent with the behavior we want.
crossref: longer comment about crossref API date fields2020-03-31T03:55:44ZBryan Newboldbnewbold@robocracy.org2020-03-31T03:55:44Zurn:sha1:98933a068ec3d918deb0e7dff30aed517ca515d9Merge pull request #53 from EdwardBetts/spelling2020-03-27T23:50:08Zbnewboldbnewbold@archive.org2020-03-27T23:50:08Zurn:sha1:98abe2e751187aa7c2e751b355ffb56d9b1f8c6a
Correct spelling mistakesCorrect spelling mistakes2020-03-27T21:25:54ZEdward Bettsedward@4angle.com2020-03-27T21:25:54Zurn:sha1:94710b2803780ab16fb30b79010f8e27cf115512