fatcat/python/fatcat_tools/harvest, branch v0.4.0[no description]https://git.bnewbold.net/fatcat/atom?h=v0.4.02021-09-09T19:20:19Zpubmed: workaround a networking issue2021-09-09T19:20:19ZMartin Czyganmartin.czygan@gmail.com2021-09-09T18:33:35Zurn:sha1:23c9206670bdad61d09d89622393e16e5f11fdca
use an http proxy (https://github.com/miku/ftpup) to fetch files from
FTP, keep some retry logic; also, hardcoding the proxy path as this
should be a temporary workaround
pubmed: add option to ftp download with lftp2021-09-08T20:02:48ZMartin Czyganmartin.czygan@gmail.com2021-08-30T21:23:28Zurn:sha1:6d9d67c9c4d1a0b208fc2056ab485a1c8d21e100
lftp is a classic command line ftp client, and we hope that its retry
capabilities are enough of a workaround for the current networking issue
pubmed harvester: add basic retry logic2021-08-20T20:32:19ZMartin Czyganmartin.czygan@gmail.com2021-08-20T20:32:19Zurn:sha1:a4352a003a9fc7085638268ff00c05e305c519f5
Related to a previous issue with seemingly random EOFError from FTP
connections, this patch wrap "ftpretr" helper function with a basic
retry.
Refs: fatcat-workers/issues/92151, fatcat-workers/issues/91102
pubmed: update docs2021-07-16T23:38:29ZMartin Czyganmartin.czygan@gmail.com2021-07-16T23:38:29Zurn:sha1:0202f5f9d0c508e2c4cc4af6a8b22bd624bcbd0bpubmed: do not fail when accessing missing file2021-07-16T23:29:16ZMartin Czyganmartin.czygan@gmail.com2021-07-16T23:29:16Zurn:sha1:6056dcc93fa8111a04d76a7af5bcddb12704cb96
after a sync gap (e.g. 06/07 2021) harvester wanted to fetch a file,
that was not on the server (any more) - do not fail in this case
we'll need to backfill missing records via full data dump
pubmed: reconnect on error2021-07-16T21:45:48ZMartin Czyganmartin.czygan@gmail.com2021-07-16T12:42:30Zurn:sha1:47c98540083da302802b54a70f77fb8abc69b4de
ftp retrieval would run but fail with EOFError on
/pubmed/updatefiles/pubmed21n1328_stats.html - not able to find the root
cause; using a fresh client, the exact same file would work just
fine. So when we retry, we reconnect on failure.
Refs: sentry #91102.
small python lint fixes (no behavior change)2021-05-25T23:30:09ZBryan Newboldbnewbold@robocracy.org2021-05-25T23:30:09Zurn:sha1:7aa19b92f0e7808a341aec8ae17485408dfae68charvest: datacite API yields HTTP 200 with broken JSON2020-08-10T17:58:12ZMartin Czyganmartin.czygan@gmail.com2020-08-10T17:55:14Zurn:sha1:e18d48642cecb55d9f2270f9048953a7b543472e
As a first step: log response body for debugging.
arxiv: do retry five times of HTTP 5032020-07-09T22:54:55ZMartin Czyganmartin.czygan@gmail.com2020-07-09T22:54:55Zurn:sha1:c403cb4a1f20bd056008f68f71b374bde1e089b5lint (flake8) tool python files2020-07-02T01:35:24ZBryan Newboldbnewbold@robocracy.org2020-07-02T01:35:24Zurn:sha1:30905f1effb33c3ef193d084120aa3fbd75d0b9b