index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
sandcrawler
/
ia.py
Commit message (
Expand
)
Author
Age
Files
Lines
*
SPNv2: make 'resources' optional
Bryan Newbold
2021-11-16
1
-1
/
+1
*
IA (wayback): actually use an HTTP session for replay fetches
Bryan Newbold
2021-11-03
1
-2
/
+3
*
make fmt (black 21.9b0)
Bryan Newbold
2021-10-27
1
-257
/
+354
*
fix type annotations for petabox body fetch helper
Bryan Newbold
2021-10-26
1
-1
/
+1
*
lint collection membership (last lint for now)
Bryan Newbold
2021-10-26
1
-7
/
+7
*
more progress on type annotations and linting
Bryan Newbold
2021-10-26
1
-2
/
+2
*
ia: more tweaks to delicate code to satisfy type checker
Bryan Newbold
2021-10-26
1
-10
/
+12
*
ia helpers: enforce max_redirects count correctly
Bryan Newbold
2021-10-26
1
-1
/
+1
*
set CDX request params are str, not int or datetime
Bryan Newbold
2021-10-26
1
-3
/
+6
*
bugfix: was setting 'from' parameter as a tuple, not a string
Bryan Newbold
2021-10-26
1
-1
/
+1
*
start type annotating IA helper code
Bryan Newbold
2021-10-26
1
-37
/
+65
*
flake8 clean (with current settings)
Bryan Newbold
2021-10-26
1
-2
/
+2
*
start handling trivial lint cleanups: unused imports, 'is None', etc
Bryan Newbold
2021-10-26
1
-15
/
+14
*
make fmt
Bryan Newbold
2021-10-26
1
-68
/
+124
*
python: isort all imports
Bryan Newbold
2021-10-26
1
-10
/
+12
*
move SPNv2 'simple_get' logic to SPN client
Bryan Newbold
2021-10-15
1
-2
/
+29
*
spn: avoid 'None' job_id
Bryan Newbold
2021-10-11
1
-2
/
+2
*
ingest CDX lookup: weigh year+month of capture against in-petabox-or-not
Bryan Newbold
2021-09-30
1
-0
/
+1
*
tune SPN CDX retry/wait depending on mode (priority vs daily)
Bryan Newbold
2021-09-30
1
-2
/
+4
*
crawl: SPN2 non-200 success code path
Bryan Newbold
2021-07-13
1
-11
/
+25
*
crawl: SPN self-redirect hack
Bryan Newbold
2021-07-13
1
-0
/
+9
*
crawl: small comment updates
Bryan Newbold
2021-07-13
1
-3
/
+6
*
ia CDX: handle bad CDX rows
Bryan Newbold
2021-01-05
1
-2
/
+4
*
spn: more status codes
Bryan Newbold
2020-12-21
1
-1
/
+2
*
handle more wayback error conditions
Bryan Newbold
2020-11-20
1
-0
/
+6
*
spn 'forbidden' status code
Bryan Newbold
2020-11-12
1
-1
/
+1
*
spn2-internal-server-error is a problem with remote server, not SPN2
Bryan Newbold
2020-11-08
1
-0
/
+2
*
ingest: better non-full URL fixup
Bryan Newbold
2020-11-08
1
-4
/
+3
*
direct some more warnings to sys.stderr, not stdout
Bryan Newbold
2020-11-08
1
-2
/
+2
*
html: handle no-capture for sub-resources
Bryan Newbold
2020-11-08
1
-0
/
+3
*
ingest: fix null-body case
Bryan Newbold
2020-11-08
1
-0
/
+2
*
ia: use newer gwb (petabox) loading class
Bryan Newbold
2020-11-04
1
-5
/
+8
*
move transfer encoding helper to sandcrawler/ia.py
Bryan Newbold
2020-11-03
1
-1
/
+26
*
ingest: tweak debug printing alignment
Bryan Newbold
2020-11-03
1
-8
/
+7
*
cdx: fix 'closest' support
Bryan Newbold
2020-11-03
1
-3
/
+2
*
cdx: add support for 'closest' time parameter
Bryan Newbold
2020-10-30
1
-3
/
+9
*
ingest: decrease CDX timeout retries again
Bryan Newbold
2020-10-22
1
-1
/
+1
*
differential wayback-error from wayback-content-error
Bryan Newbold
2020-10-21
1
-10
/
+13
*
SPN CDX delay now seems reasonable; increase to 40sec to catch most
Bryan Newbold
2020-10-19
1
-1
/
+1
*
CDX: when retrying, do so every 3 seconds up to limit
Bryan Newbold
2020-10-19
1
-5
/
+9
*
SPN: more verbose status logging
Bryan Newbold
2020-10-19
1
-0
/
+4
*
CDX: revert post-SPN CDX lookup retry to 10 seconds
Bryan Newbold
2020-10-19
1
-1
/
+1
*
ingest: catch wayback-fail-after-SPN as separate status
Bryan Newbold
2020-10-19
1
-4
/
+17
*
SPN: better log line when starting a request
Bryan Newbold
2020-10-19
1
-0
/
+1
*
SPN: look for non-200 CDX responses
Bryan Newbold
2020-10-19
1
-1
/
+1
*
SPN: better check for partial URLs returned
Bryan Newbold
2020-10-19
1
-2
/
+2
*
CDX fetch: more permissive fuzzy/normalization check
Bryan Newbold
2020-10-19
1
-3
/
+9
*
ingest: experimentally reduce CDX API retry delay
Bryan Newbold
2020-10-17
1
-1
/
+1
*
ingest: handle cookieAbsent and partial SPNv2 URL reponse cases better
Bryan Newbold
2020-10-17
1
-0
/
+31
*
store no-capture URLs in terminal_url
Bryan Newbold
2020-10-12
1
-1
/
+1
[next]