index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
sandcrawler
/
ia.py
Commit message (
Expand
)
Author
Age
Files
Lines
*
crawl: SPN2 non-200 success code path
Bryan Newbold
2021-07-13
1
-11
/
+25
*
crawl: SPN self-redirect hack
Bryan Newbold
2021-07-13
1
-0
/
+9
*
crawl: small comment updates
Bryan Newbold
2021-07-13
1
-3
/
+6
*
ia CDX: handle bad CDX rows
Bryan Newbold
2021-01-05
1
-2
/
+4
*
spn: more status codes
Bryan Newbold
2020-12-21
1
-1
/
+2
*
handle more wayback error conditions
Bryan Newbold
2020-11-20
1
-0
/
+6
*
spn 'forbidden' status code
Bryan Newbold
2020-11-12
1
-1
/
+1
*
spn2-internal-server-error is a problem with remote server, not SPN2
Bryan Newbold
2020-11-08
1
-0
/
+2
*
ingest: better non-full URL fixup
Bryan Newbold
2020-11-08
1
-4
/
+3
*
direct some more warnings to sys.stderr, not stdout
Bryan Newbold
2020-11-08
1
-2
/
+2
*
html: handle no-capture for sub-resources
Bryan Newbold
2020-11-08
1
-0
/
+3
*
ingest: fix null-body case
Bryan Newbold
2020-11-08
1
-0
/
+2
*
ia: use newer gwb (petabox) loading class
Bryan Newbold
2020-11-04
1
-5
/
+8
*
move transfer encoding helper to sandcrawler/ia.py
Bryan Newbold
2020-11-03
1
-1
/
+26
*
ingest: tweak debug printing alignment
Bryan Newbold
2020-11-03
1
-8
/
+7
*
cdx: fix 'closest' support
Bryan Newbold
2020-11-03
1
-3
/
+2
*
cdx: add support for 'closest' time parameter
Bryan Newbold
2020-10-30
1
-3
/
+9
*
ingest: decrease CDX timeout retries again
Bryan Newbold
2020-10-22
1
-1
/
+1
*
differential wayback-error from wayback-content-error
Bryan Newbold
2020-10-21
1
-10
/
+13
*
SPN CDX delay now seems reasonable; increase to 40sec to catch most
Bryan Newbold
2020-10-19
1
-1
/
+1
*
CDX: when retrying, do so every 3 seconds up to limit
Bryan Newbold
2020-10-19
1
-5
/
+9
*
SPN: more verbose status logging
Bryan Newbold
2020-10-19
1
-0
/
+4
*
CDX: revert post-SPN CDX lookup retry to 10 seconds
Bryan Newbold
2020-10-19
1
-1
/
+1
*
ingest: catch wayback-fail-after-SPN as separate status
Bryan Newbold
2020-10-19
1
-4
/
+17
*
SPN: better log line when starting a request
Bryan Newbold
2020-10-19
1
-0
/
+1
*
SPN: look for non-200 CDX responses
Bryan Newbold
2020-10-19
1
-1
/
+1
*
SPN: better check for partial URLs returned
Bryan Newbold
2020-10-19
1
-2
/
+2
*
CDX fetch: more permissive fuzzy/normalization check
Bryan Newbold
2020-10-19
1
-3
/
+9
*
ingest: experimentally reduce CDX API retry delay
Bryan Newbold
2020-10-17
1
-1
/
+1
*
ingest: handle cookieAbsent and partial SPNv2 URL reponse cases better
Bryan Newbold
2020-10-17
1
-0
/
+31
*
store no-capture URLs in terminal_url
Bryan Newbold
2020-10-12
1
-1
/
+1
*
Revert "ingest: reduce CDX retry_sleep to 3.0 sec (after SPN)"
Bryan Newbold
2020-08-11
1
-1
/
+1
*
ingest: reduce CDX retry_sleep to 3.0 sec (after SPN)
Bryan Newbold
2020-08-11
1
-1
/
+1
*
refactor: force_get -> force_simple_get
Bryan Newbold
2020-08-11
1
-5
/
+5
*
spn2: skip js behavior (experiment)
Bryan Newbold
2020-08-05
1
-0
/
+1
*
SPN2: ensure not fetching outlinks
Bryan Newbold
2020-08-05
1
-0
/
+1
*
use new SPNv2 'skip_first_archive' param
Bryan Newbold
2020-07-22
1
-0
/
+1
*
report revisit non-200 as a WaybackError
Bryan Newbold
2020-06-26
1
-7
/
+7
*
pdf: mypy and typo fixes
Bryan Newbold
2020-06-17
1
-1
/
+1
*
ingest: quick hack to capture CNKI outlinks
Bryan Newbold
2020-04-13
1
-2
/
+9
*
ia: set User-Agent for replay fetch from wayback
Bryan Newbold
2020-03-29
1
-0
/
+5
*
ingest: better spn2 pending error code
Bryan Newbold
2020-03-27
1
-0
/
+2
*
ia: more conservative use of clean_url()
Bryan Newbold
2020-03-24
1
-3
/
+5
*
ingest: clean_url() in more places
Bryan Newbold
2020-03-23
1
-1
/
+4
*
ingest: log every URL (from ia code side)
Bryan Newbold
2020-03-18
1
-0
/
+1
*
implement (unused) force_get flag for SPN2
Bryan Newbold
2020-03-18
1
-3
/
+4
*
work around local redirect (resource.location)
Bryan Newbold
2020-03-17
1
-1
/
+6
*
ia: catch wayback ChunkedEncodingError
Bryan Newbold
2020-03-05
1
-0
/
+3
*
fix warc_offset -> offset
Bryan Newbold
2020-02-24
1
-1
/
+1
*
ingest: handle broken revisit records
Bryan Newbold
2020-02-24
1
-1
/
+4
[next]