index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
sandcrawler
/
ingest.py
Commit message (
Expand
)
Author
Age
Files
Lines
*
additional loginwall patterns
Bryan Newbold
2020-08-11
1
-0
/
+2
*
ingest: actually use force_get flag with SPN
Bryan Newbold
2020-08-11
1
-0
/
+13
*
check for simple URL patterns that are usually paywalls or loginwalls
Bryan Newbold
2020-08-11
1
-0
/
+11
*
ingest: check for URL blocklist and cookie URL patterns on every hop
Bryan Newbold
2020-08-11
1
-0
/
+13
*
refactor: force_get -> force_simple_get
Bryan Newbold
2020-08-11
1
-3
/
+3
*
add hkvalidate.perfdrive.com to domain blocklist
Bryan Newbold
2020-08-08
1
-0
/
+3
*
pdfextract support in ingest worker
Bryan Newbold
2020-06-25
1
-1
/
+35
*
workers: refactor to pass key to process()
Bryan Newbold
2020-06-17
1
-2
/
+2
*
ingest: don't 'want' non-PDF ingest
Bryan Newbold
2020-04-30
1
-0
/
+5
*
timeout message implementation for GROBID and ingest workers
Bryan Newbold
2020-04-27
1
-0
/
+9
*
ingest: block another large domain (and DOI prefix)
Bryan Newbold
2020-03-27
1
-0
/
+2
*
ingest: clean_url() in more places
Bryan Newbold
2020-03-23
1
-0
/
+1
*
implement (unused) force_get flag for SPN2
Bryan Newbold
2020-03-18
1
-1
/
+15
*
url cleaning (canonicalization) for ingest base_url
Bryan Newbold
2020-03-10
1
-2
/
+6
*
ingest: make content-decoding more robust
Bryan Newbold
2020-03-03
1
-1
/
+2
*
make gzip content-encoding path more robust
Bryan Newbold
2020-03-03
1
-1
/
+10
*
ingest: crude content-encoding support
Bryan Newbold
2020-03-02
1
-1
/
+19
*
ingest: add force_recrawl flag to skip historical wayback lookup
Bryan Newbold
2020-03-02
1
-3
/
+5
*
remove protocols.io octet-stream hack
Bryan Newbold
2020-03-02
1
-6
/
+2
*
ingest: narrow xhtml filter
Bryan Newbold
2020-02-25
1
-1
/
+1
*
ingest: include better terminal URL/status_code/dt
Bryan Newbold
2020-02-22
1
-0
/
+8
*
ingest: skip more non-pdf, non-paper domains
Bryan Newbold
2020-02-22
1
-0
/
+9
*
block springer page-one domain
Bryan Newbold
2020-01-28
1
-0
/
+3
*
re-enable figshare and zenodo crawling
Bryan Newbold
2020-01-21
1
-8
/
+0
*
ingest: check for null-body before file_meta
Bryan Newbold
2020-01-21
1
-0
/
+3
*
add SKIP log line for skip-url-blocklist path
Bryan Newbold
2020-01-17
1
-0
/
+1
*
ingest: add URL blocklist feature
Bryan Newbold
2020-01-17
1
-4
/
+32
*
clarify ingest result schema and semantics
Bryan Newbold
2020-01-15
1
-4
/
+11
*
pass through revisit_cdx
Bryan Newbold
2020-01-15
1
-0
/
+3
*
ingest: sketch out more of how 'existing' path would work
Bryan Newbold
2020-01-14
1
-8
/
+22
*
ingest: check existing GROBID; also push results to sink
Bryan Newbold
2020-01-14
1
-4
/
+22
*
filter out archive.org and web.archive.org (until implemented)
Bryan Newbold
2020-01-14
1
-1
/
+12
*
basic FTP ingest support; revist record resolution
Bryan Newbold
2020-01-14
1
-1
/
+1
*
better print() output
Bryan Newbold
2020-01-10
1
-1
/
+1
*
fix trivial typo
Bryan Newbold
2020-01-10
1
-1
/
+1
*
hack/workaround for protocols.io octet PDFs
Bryan Newbold
2020-01-10
1
-2
/
+4
*
limit length of error messages
Bryan Newbold
2020-01-10
1
-4
/
+4
*
more general ingest teaks and affordances
Bryan Newbold
2020-01-10
1
-10
/
+24
*
improve ingest robustness (for legacy requests)
Bryan Newbold
2020-01-10
1
-6
/
+12
*
support forwarding url types other than pdf_url
Bryan Newbold
2020-01-09
1
-4
/
+5
*
refactor ingest to a loop, allowing multiple hops
Bryan Newbold
2020-01-09
1
-25
/
+48
*
lots of progress on wayback refactoring
Bryan Newbold
2020-01-09
1
-11
/
+15
*
wrap up basic (locally testable) ingest refactor
Bryan Newbold
2020-01-09
1
-159
/
+196
*
remove SPNv1 code paths
Bryan Newbold
2020-01-07
1
-30
/
+24
*
refactor: use print(..., file=sys.stderr)
Bryan Newbold
2019-12-18
1
-4
/
+4
*
handle wayback fetch redirect loop in ingest code
Bryan Newbold
2019-11-14
1
-2
/
+5
*
handle WaybackError during ingest
Bryan Newbold
2019-11-14
1
-0
/
+4
*
start of hrmars.com ingest support
Bryan Newbold
2019-11-14
1
-2
/
+5
*
treat failure to get terminal capture as a SavePageNowError
Bryan Newbold
2019-11-13
1
-1
/
+1
*
handle wayback client return status correctly
Bryan Newbold
2019-11-13
1
-2
/
+2
[next]