index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
sandcrawler
/
ia.py
Commit message (
Expand
)
Author
Age
Files
Lines
*
SPN: more verbose status logging
Bryan Newbold
2020-10-19
1
-0
/
+4
*
CDX: revert post-SPN CDX lookup retry to 10 seconds
Bryan Newbold
2020-10-19
1
-1
/
+1
*
ingest: catch wayback-fail-after-SPN as separate status
Bryan Newbold
2020-10-19
1
-4
/
+17
*
SPN: better log line when starting a request
Bryan Newbold
2020-10-19
1
-0
/
+1
*
SPN: look for non-200 CDX responses
Bryan Newbold
2020-10-19
1
-1
/
+1
*
SPN: better check for partial URLs returned
Bryan Newbold
2020-10-19
1
-2
/
+2
*
CDX fetch: more permissive fuzzy/normalization check
Bryan Newbold
2020-10-19
1
-3
/
+9
*
ingest: experimentally reduce CDX API retry delay
Bryan Newbold
2020-10-17
1
-1
/
+1
*
ingest: handle cookieAbsent and partial SPNv2 URL reponse cases better
Bryan Newbold
2020-10-17
1
-0
/
+31
*
store no-capture URLs in terminal_url
Bryan Newbold
2020-10-12
1
-1
/
+1
*
Revert "ingest: reduce CDX retry_sleep to 3.0 sec (after SPN)"
Bryan Newbold
2020-08-11
1
-1
/
+1
*
ingest: reduce CDX retry_sleep to 3.0 sec (after SPN)
Bryan Newbold
2020-08-11
1
-1
/
+1
*
refactor: force_get -> force_simple_get
Bryan Newbold
2020-08-11
1
-5
/
+5
*
spn2: skip js behavior (experiment)
Bryan Newbold
2020-08-05
1
-0
/
+1
*
SPN2: ensure not fetching outlinks
Bryan Newbold
2020-08-05
1
-0
/
+1
*
use new SPNv2 'skip_first_archive' param
Bryan Newbold
2020-07-22
1
-0
/
+1
*
report revisit non-200 as a WaybackError
Bryan Newbold
2020-06-26
1
-7
/
+7
*
pdf: mypy and typo fixes
Bryan Newbold
2020-06-17
1
-1
/
+1
*
ingest: quick hack to capture CNKI outlinks
Bryan Newbold
2020-04-13
1
-2
/
+9
*
ia: set User-Agent for replay fetch from wayback
Bryan Newbold
2020-03-29
1
-0
/
+5
*
ingest: better spn2 pending error code
Bryan Newbold
2020-03-27
1
-0
/
+2
*
ia: more conservative use of clean_url()
Bryan Newbold
2020-03-24
1
-3
/
+5
*
ingest: clean_url() in more places
Bryan Newbold
2020-03-23
1
-1
/
+4
*
ingest: log every URL (from ia code side)
Bryan Newbold
2020-03-18
1
-0
/
+1
*
implement (unused) force_get flag for SPN2
Bryan Newbold
2020-03-18
1
-3
/
+4
*
work around local redirect (resource.location)
Bryan Newbold
2020-03-17
1
-1
/
+6
*
ia: catch wayback ChunkedEncodingError
Bryan Newbold
2020-03-05
1
-0
/
+3
*
fix warc_offset -> offset
Bryan Newbold
2020-02-24
1
-1
/
+1
*
ingest: handle broken revisit records
Bryan Newbold
2020-02-24
1
-1
/
+4
*
ingest: treat CDX lookup error as a wayback-error
Bryan Newbold
2020-02-24
1
-1
/
+4
*
fetch_petabox_body: allow non-200 status code fetches
Bryan Newbold
2020-02-24
1
-2
/
+10
*
allow fuzzy revisit matches
Bryan Newbold
2020-02-24
1
-1
/
+26
*
ingest: more revisit fixes
Bryan Newbold
2020-02-22
1
-4
/
+4
*
ia: improve warc/revisit implementation
Bryan Newbold
2020-02-22
1
-26
/
+46
*
cdx: handle empty/null CDX response
Bryan Newbold
2020-02-22
1
-0
/
+2
*
filter out CDX rows missing WARC playback fields
Bryan Newbold
2020-02-19
1
-0
/
+4
*
X-Archive-Src more robust than X-Archive-Redirect-Reason
Bryan Newbold
2020-02-18
1
-2
/
+3
*
wayback: on bad redirects, log instead of assert
Bryan Newbold
2020-02-18
1
-2
/
+13
*
attempt to work around corrupt ARC files from alexa issue
Bryan Newbold
2020-02-18
1
-0
/
+5
*
handle alternative dt format in WARC headers
Bryan Newbold
2020-02-05
1
-2
/
+4
*
decrease SPNv2 polling timeout to 3 minutes
Bryan Newbold
2020-02-05
1
-2
/
+2
*
improvements to reliability from prod testing
Bryan Newbold
2020-02-03
1
-5
/
+11
*
hack-y backoff ingest attempt
Bryan Newbold
2020-02-03
1
-2
/
+11
*
wayback: try to resolve HTTPException due to many HTTP headers
Bryan Newbold
2020-02-02
1
-1
/
+9
*
fix WaybackError exception formating
Bryan Newbold
2020-01-28
1
-1
/
+1
*
fix elif syntax error
Bryan Newbold
2020-01-28
1
-1
/
+1
*
clarify petabox fetch behavior
Bryan Newbold
2020-01-28
1
-3
/
+6
*
wayback: replay redirects have X-Archive-Redirect-Reason
Bryan Newbold
2020-01-21
1
-2
/
+4
*
handle UnicodeDecodeError in the other GET instance
Bryan Newbold
2020-01-15
1
-0
/
+2
*
increase SPNv2 polling timeout to 4 minutes
Bryan Newbold
2020-01-15
1
-1
/
+3
[next]