diff options
| author | Bryan Newbold <bnewbold@archive.org> | 2020-10-17 17:20:58 -0700 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@archive.org> | 2020-10-17 17:21:02 -0700 |
| commit | cc26ea975e29eefa2e2d3565c55ba0ac0a491bb7 (patch) | |
| tree | 748b6b68d3d4fcd038792f24fc1d0cc718bc0789 | |
| parent | c8a7683f1ac1f42bd9e4c312ae54c792b01392ec (diff) | |
| download | sandcrawler-cc26ea975e29eefa2e2d3565c55ba0ac0a491bb7.tar.gz sandcrawler-cc26ea975e29eefa2e2d3565c55ba0ac0a491bb7.zip | |
ingest: experimentally reduce CDX API retry delay
This code path is only working about 1/7 times in production. Going to
try with a much shorter retry delay and see if we get no success with
that. Considering also just disabling this attempt all together and
relying on retries after hours/days.
| -rw-r--r-- | python/sandcrawler/ia.py | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/python/sandcrawler/ia.py b/python/sandcrawler/ia.py index 60e3d9a..ea29e67 100644 --- a/python/sandcrawler/ia.py +++ b/python/sandcrawler/ia.py @@ -978,7 +978,7 @@ class SavePageNowClient: url=spn_result.terminal_url, datetime=spn_result.terminal_dt, filter_status_code=filter_status_code, - retry_sleep=10.0, + retry_sleep=2.0, ) except KeyError as ke: print("CDX KeyError: {}".format(ke), file=sys.stderr) |