aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@archive.org>2020-10-17 17:20:58 -0700
committerBryan Newbold <bnewbold@archive.org>2020-10-17 17:21:02 -0700
commitcc26ea975e29eefa2e2d3565c55ba0ac0a491bb7 (patch)
tree748b6b68d3d4fcd038792f24fc1d0cc718bc0789
parentc8a7683f1ac1f42bd9e4c312ae54c792b01392ec (diff)
downloadsandcrawler-cc26ea975e29eefa2e2d3565c55ba0ac0a491bb7.tar.gz
sandcrawler-cc26ea975e29eefa2e2d3565c55ba0ac0a491bb7.zip
ingest: experimentally reduce CDX API retry delay
This code path is only working about 1/7 times in production. Going to try with a much shorter retry delay and see if we get no success with that. Considering also just disabling this attempt all together and relying on retries after hours/days.
-rw-r--r--python/sandcrawler/ia.py2
1 files changed, 1 insertions, 1 deletions
diff --git a/python/sandcrawler/ia.py b/python/sandcrawler/ia.py
index 60e3d9a..ea29e67 100644
--- a/python/sandcrawler/ia.py
+++ b/python/sandcrawler/ia.py
@@ -978,7 +978,7 @@ class SavePageNowClient:
url=spn_result.terminal_url,
datetime=spn_result.terminal_dt,
filter_status_code=filter_status_code,
- retry_sleep=10.0,
+ retry_sleep=2.0,
)
except KeyError as ke:
print("CDX KeyError: {}".format(ke), file=sys.stderr)