aboutsummaryrefslogtreecommitdiffstats
path: root/python/TODO
blob: 58a463f8211170199cba2fff1bca30f9fb0d3ed7 (plain)
1
2
3
4
5
6
7

ingest crawler:
- SPNv2 only
    - remove most SPNv1/v2 path selection
- landing page + fulltext hops only (short recursion depth)
- use wayback client library instead of requests to fetch content
- https://pypi.org/project/ratelimit/