index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
persist grobid: actually, status_code is required
Bryan Newbold
2020-01-21
2
-3
/
+10
*
ingest: check for null-body before file_meta
Bryan Newbold
2020-01-21
1
-0
/
+3
*
wayback: replay redirects have X-Archive-Redirect-Reason
Bryan Newbold
2020-01-21
1
-2
/
+4
*
persist: work around GROBID timeouts with no status_code
Bryan Newbold
2020-01-21
2
-3
/
+3
*
grobid: fix error_msg typo; set status_code for timeouts
Bryan Newbold
2020-01-21
1
-1
/
+2
*
add 200 second timeout to GROBID requests
Bryan Newbold
2020-01-17
1
-8
/
+15
*
add SKIP log line for skip-url-blocklist path
Bryan Newbold
2020-01-17
1
-0
/
+1
*
ingest: add URL blocklist feature
Bryan Newbold
2020-01-17
2
-4
/
+49
*
handle UnicodeDecodeError in the other GET instance
Bryan Newbold
2020-01-15
1
-0
/
+2
*
increase SPNv2 polling timeout to 4 minutes
Bryan Newbold
2020-01-15
1
-1
/
+3
*
make failed replay fetch an error, not assert error
Bryan Newbold
2020-01-15
1
-1
/
+2
*
improve sentry reporting with 'release' git hash
Bryan Newbold
2020-01-15
2
-2
/
+5
*
wayback replay: catch UnicodeDecodeError
Bryan Newbold
2020-01-15
1
-0
/
+2
*
persist: fix dupe field copying
Bryan Newbold
2020-01-15
1
-1
/
+8
*
persist worker: implement updated ingest result semantics
Bryan Newbold
2020-01-15
2
-12
/
+17
*
clarify ingest result schema and semantics
Bryan Newbold
2020-01-15
3
-7
/
+32
*
pass through revisit_cdx
Bryan Newbold
2020-01-15
2
-5
/
+21
*
fix revisit resolution
Bryan Newbold
2020-01-15
1
-4
/
+12
*
add postgrest checks to test mocks
Bryan Newbold
2020-01-14
1
-1
/
+9
*
tests: don't use localhost as a responses mock host
Bryan Newbold
2020-01-14
2
-6
/
+6
*
bulk ingest file request topic support
Bryan Newbold
2020-01-14
1
-1
/
+7
*
ingest: sketch out more of how 'existing' path would work
Bryan Newbold
2020-01-14
1
-8
/
+22
*
ingest: check existing GROBID; also push results to sink
Bryan Newbold
2020-01-14
1
-4
/
+22
*
ingest persist skips 'existing' ingest results
Bryan Newbold
2020-01-14
1
-0
/
+3
*
grobid-to-kafka support in ingest worker
Bryan Newbold
2020-01-14
1
-0
/
+6
*
grobid worker fixes for newer ia lib refactors
Bryan Newbold
2020-01-14
1
-3
/
+9
*
small fixups to SandcrawlerPostgrestClient
Bryan Newbold
2020-01-14
2
-1
/
+11
*
filter out archive.org and web.archive.org (until implemented)
Bryan Newbold
2020-01-14
1
-1
/
+12
*
SPNv2 doesn't support FTP; add a live test for non-revist FTP
Bryan Newbold
2020-01-14
2
-0
/
+26
*
more ftp status 226 support
Bryan Newbold
2020-01-14
5
-9
/
+23
*
add live tests for ftp, revisits
Bryan Newbold
2020-01-14
1
-1
/
+36
*
basic FTP ingest support; revist record resolution
Bryan Newbold
2020-01-14
2
-35
/
+78
*
arabesque2ingestrequest: ingest type flag
Bryan Newbold
2020-01-14
1
-1
/
+4
*
better print() output
Bryan Newbold
2020-01-10
2
-4
/
+4
*
fix trivial typo
Bryan Newbold
2020-01-10
1
-1
/
+1
*
hack/workaround for protocols.io octet PDFs
Bryan Newbold
2020-01-10
1
-2
/
+4
*
html extract: protocols.io, fix americanarchivist
Bryan Newbold
2020-01-10
1
-1
/
+7
*
fix redirect replay fetch method
Bryan Newbold
2020-01-10
1
-1
/
+4
*
limit length of error messages
Bryan Newbold
2020-01-10
1
-4
/
+4
*
handle SPNv2-then-CDX lookup failures
Bryan Newbold
2020-01-10
1
-6
/
+23
*
SPNv2 hack specifically for elsevier lookups
Bryan Newbold
2020-01-10
1
-0
/
+15
*
add support for redirect lookups from replay
Bryan Newbold
2020-01-10
1
-9
/
+69
*
more general ingest teaks and affordances
Bryan Newbold
2020-01-10
2
-15
/
+42
*
add sleep-and-retry workaround for CDX after SPNv2
Bryan Newbold
2020-01-10
1
-1
/
+9
*
more ingest HTML extraction hacks
Bryan Newbold
2020-01-10
1
-6
/
+46
*
more live tests (for regressions)
Bryan Newbold
2020-01-10
2
-0
/
+42
*
disable CDX best lookup 'collapse'; leave comment
Bryan Newbold
2020-01-10
1
-1
/
+3
*
hack: reverse sort of CDX exact seems broken with SPNv2 results
Bryan Newbold
2020-01-10
1
-1
/
+1
*
many publisher-specific ingest improvements
Bryan Newbold
2020-01-10
1
-4
/
+96
*
improve ingest robustness (for legacy requests)
Bryan Newbold
2020-01-10
1
-6
/
+12
[prev]
[next]