index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
sandcrawler
/
ingest.py
Commit message (
Expand
)
Author
Age
Files
Lines
*
add SKIP log line for skip-url-blocklist path
Bryan Newbold
2020-01-17
1
-0
/
+1
*
ingest: add URL blocklist feature
Bryan Newbold
2020-01-17
1
-4
/
+32
*
clarify ingest result schema and semantics
Bryan Newbold
2020-01-15
1
-4
/
+11
*
pass through revisit_cdx
Bryan Newbold
2020-01-15
1
-0
/
+3
*
ingest: sketch out more of how 'existing' path would work
Bryan Newbold
2020-01-14
1
-8
/
+22
*
ingest: check existing GROBID; also push results to sink
Bryan Newbold
2020-01-14
1
-4
/
+22
*
filter out archive.org and web.archive.org (until implemented)
Bryan Newbold
2020-01-14
1
-1
/
+12
*
basic FTP ingest support; revist record resolution
Bryan Newbold
2020-01-14
1
-1
/
+1
*
better print() output
Bryan Newbold
2020-01-10
1
-1
/
+1
*
fix trivial typo
Bryan Newbold
2020-01-10
1
-1
/
+1
*
hack/workaround for protocols.io octet PDFs
Bryan Newbold
2020-01-10
1
-2
/
+4
*
limit length of error messages
Bryan Newbold
2020-01-10
1
-4
/
+4
*
more general ingest teaks and affordances
Bryan Newbold
2020-01-10
1
-10
/
+24
*
improve ingest robustness (for legacy requests)
Bryan Newbold
2020-01-10
1
-6
/
+12
*
support forwarding url types other than pdf_url
Bryan Newbold
2020-01-09
1
-4
/
+5
*
refactor ingest to a loop, allowing multiple hops
Bryan Newbold
2020-01-09
1
-25
/
+48
*
lots of progress on wayback refactoring
Bryan Newbold
2020-01-09
1
-11
/
+15
*
wrap up basic (locally testable) ingest refactor
Bryan Newbold
2020-01-09
1
-159
/
+196
*
remove SPNv1 code paths
Bryan Newbold
2020-01-07
1
-30
/
+24
*
refactor: use print(..., file=sys.stderr)
Bryan Newbold
2019-12-18
1
-4
/
+4
*
handle wayback fetch redirect loop in ingest code
Bryan Newbold
2019-11-14
1
-2
/
+5
*
handle WaybackError during ingest
Bryan Newbold
2019-11-14
1
-0
/
+4
*
start of hrmars.com ingest support
Bryan Newbold
2019-11-14
1
-2
/
+5
*
treat failure to get terminal capture as a SavePageNowError
Bryan Newbold
2019-11-13
1
-1
/
+1
*
handle wayback client return status correctly
Bryan Newbold
2019-11-13
1
-2
/
+2
*
allow way more errors in SPN path
Bryan Newbold
2019-11-13
1
-2
/
+11
*
fix lint errors
Bryan Newbold
2019-11-13
1
-1
/
+1
*
improve ingest worker remote failure behavior
Bryan Newbold
2019-11-13
1
-5
/
+12
*
rename FileIngestWorker
Bryan Newbold
2019-11-13
1
-5
/
+9
*
more progress on file ingest
Bryan Newbold
2019-11-13
1
-10
/
+37
*
much progress on file ingest path
Bryan Newbold
2019-10-22
1
-0
/
+150