diff options
| author | Bryan Newbold <bnewbold@archive.org> | 2021-11-16 00:24:13 -0800 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@archive.org> | 2021-11-16 00:24:15 -0800 |
| commit | e323b390bf6790edfd8a4d0683642a480e62c743 (patch) | |
| tree | b587248bccfa3bffc122bc53dc2412bb15d2f636 /python | |
| parent | bd8b0b4d0d1fb04e34002dd3da91d83b922d0991 (diff) | |
| download | sandcrawler-e323b390bf6790edfd8a4d0683642a480e62c743.tar.gz sandcrawler-e323b390bf6790edfd8a4d0683642a480e62c743.zip | |
SPNv2: make 'resources' optional
This was always present previously. A change was made to SPNv2 API
recently that borked it a bit, though in theory should be present on new
captures. I'm not seeing it for some captures, so pushing this work
around. It seems like we don't actually use this field anyways, at least
for ingest pipeline.
Diffstat (limited to 'python')
| -rw-r--r-- | python/sandcrawler/ia.py | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/python/sandcrawler/ia.py b/python/sandcrawler/ia.py index d334d24..dc9aae5 100644 --- a/python/sandcrawler/ia.py +++ b/python/sandcrawler/ia.py @@ -1101,7 +1101,7 @@ class SavePageNowClient: request_url, final_json["original_url"], final_json["timestamp"], - final_json["resources"], + final_json.get("resources") or None, ) else: if final_json["status"] == "pending": |