diff options
| author | Bryan Newbold <bnewbold@archive.org> | 2020-01-02 18:10:41 -0800 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@archive.org> | 2020-01-02 18:13:03 -0800 |
| commit | 34f8d3d32845774d7fc8690350084eabff54e44f (patch) | |
| tree | f016f7449e8eef538a4ba367fe0f5481742e2cb8 | |
| parent | ddf56878e867e3556cb9c6cb6df6e60cff4e21fa (diff) | |
| download | sandcrawler-34f8d3d32845774d7fc8690350084eabff54e44f.tar.gz sandcrawler-34f8d3d32845774d7fc8690350084eabff54e44f.zip | |
TODO: check out python ratelimit library
| -rw-r--r-- | python/TODO | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/python/TODO b/python/TODO index 89cec83..58a463f 100644 --- a/python/TODO +++ b/python/TODO @@ -4,4 +4,4 @@ ingest crawler: - remove most SPNv1/v2 path selection - landing page + fulltext hops only (short recursion depth) - use wayback client library instead of requests to fetch content - +- https://pypi.org/project/ratelimit/ |