index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
Commit message (
Expand
)
Author
Age
Files
Lines
*
ingest: disable soft404 and non-hit SPNv2 retries
Bryan Newbold
2020-10-19
1
-4
/
+5
*
CDX: revert post-SPN CDX lookup retry to 10 seconds
Bryan Newbold
2020-10-19
1
-1
/
+1
*
ingest: catch wayback-fail-after-SPN as separate status
Bryan Newbold
2020-10-19
1
-4
/
+17
*
SPN: better log line when starting a request
Bryan Newbold
2020-10-19
1
-0
/
+1
*
SPN: look for non-200 CDX responses
Bryan Newbold
2020-10-19
1
-1
/
+1
*
SPN: better check for partial URLs returned
Bryan Newbold
2020-10-19
1
-2
/
+2
*
CDX fetch: more permissive fuzzy/normalization check
Bryan Newbold
2020-10-19
1
-3
/
+9
*
ingest: experimentally reduce CDX API retry delay
Bryan Newbold
2020-10-17
1
-1
/
+1
*
ingest: handle cookieAbsent and partial SPNv2 URL reponse cases better
Bryan Newbold
2020-10-17
1
-0
/
+31
*
and another sha1
Bryan Newbold
2020-10-13
1
-0
/
+1
*
another day, another bad PDF sha1
Bryan Newbold
2020-10-13
1
-0
/
+1
*
store no-capture URLs in terminal_url
Bryan Newbold
2020-10-12
2
-3
/
+3
*
another bad PDF sha1
Bryan Newbold
2020-10-11
1
-0
/
+1
*
yet more bad sha1 PDFs to skip
Bryan Newbold
2020-10-10
1
-0
/
+20
*
ingest: small bugfix to print pdfextract status on SUCCESS
Bryan Newbold
2020-09-17
1
-1
/
+1
*
more bad PDF sha1
Bryan Newbold
2020-09-17
1
-0
/
+2
*
yet another broken PDF (sha1)
Bryan Newbold
2020-09-16
1
-0
/
+1
*
html: handle JMIR URL pattern
Bryan Newbold
2020-09-15
1
-0
/
+6
*
skip citation_pdf_url if it is a link loop
Bryan Newbold
2020-09-14
1
-2
/
+8
*
html parse: add another generic fulltext pattern
Bryan Newbold
2020-09-14
1
-1
/
+10
*
ingest: treat text/xml as XHTML in pdf ingest
Bryan Newbold
2020-09-14
1
-1
/
+1
*
more bad SHA1 PDF
Bryan Newbold
2020-09-02
1
-0
/
+2
*
another bad PDF sha1
Bryan Newbold
2020-09-01
1
-0
/
+1
*
another bad PDF sha1
Bryan Newbold
2020-08-24
1
-0
/
+1
*
html: handle embed with mangled 'src' attribute
Bryan Newbold
2020-08-24
1
-1
/
+1
*
another bad PDF sha1
Bryan Newbold
2020-08-17
1
-0
/
+1
*
another bad PDF sha1
Bryan Newbold
2020-08-15
1
-0
/
+1
*
more bad sha1
Bryan Newbold
2020-08-14
1
-0
/
+1
*
yet more bad PDF sha1
Bryan Newbold
2020-08-14
1
-0
/
+2
*
more bad SHA1
Bryan Newbold
2020-08-13
1
-0
/
+2
*
yet another PDF sha1
Bryan Newbold
2020-08-12
1
-0
/
+1
*
another bad sha1; maybe the last for this batch?
Bryan Newbold
2020-08-12
1
-0
/
+1
*
more bad sha1
Bryan Newbold
2020-08-11
1
-0
/
+2
*
additional loginwall patterns
Bryan Newbold
2020-08-11
1
-0
/
+2
*
more SHA1
Bryan Newbold
2020-08-11
1
-0
/
+2
*
Revert "ingest: reduce CDX retry_sleep to 3.0 sec (after SPN)"
Bryan Newbold
2020-08-11
1
-1
/
+1
*
ingest: reduce CDX retry_sleep to 3.0 sec (after SPN)
Bryan Newbold
2020-08-11
1
-1
/
+1
*
ingest: actually use force_get flag with SPN
Bryan Newbold
2020-08-11
1
-0
/
+13
*
check for simple URL patterns that are usually paywalls or loginwalls
Bryan Newbold
2020-08-11
2
-0
/
+29
*
ingest: check for URL blocklist and cookie URL patterns on every hop
Bryan Newbold
2020-08-11
1
-0
/
+13
*
refactor: force_get -> force_simple_get
Bryan Newbold
2020-08-11
2
-8
/
+8
*
html: extract eprints PDF url (eg, ub.uni-heidelberg.de)
Bryan Newbold
2020-08-11
1
-0
/
+2
*
extract PDF urls for e-periodica.ch
Bryan Newbold
2020-08-10
1
-0
/
+6
*
more bad sha1
Bryan Newbold
2020-08-10
1
-0
/
+2
*
another bad PDF sha1
Bryan Newbold
2020-08-10
1
-0
/
+1
*
add hkvalidate.perfdrive.com to domain blocklist
Bryan Newbold
2020-08-08
1
-0
/
+3
*
fix tests passing str as HTML
Bryan Newbold
2020-08-08
1
-3
/
+3
*
add more HTML extraction tricks
Bryan Newbold
2020-08-08
1
-2
/
+29
*
rwth-aachen.de HTML extract, and a generic URL guess method
Bryan Newbold
2020-08-08
1
-0
/
+15
*
another PDF hash to skip
Bryan Newbold
2020-08-08
1
-0
/
+1
[next]