index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
|
persist PDF extraction in ingest pipeline
Bryan Newbold
2020-10-20
1
-4
/
+16
*
|
ingest: add a cdx-error slowdown delay
Bryan Newbold
2020-10-19
1
-0
/
+3
*
|
SPN CDX delay now seems reasonable; increase to 40sec to catch most
Bryan Newbold
2020-10-19
1
-1
/
+1
*
|
ingest: fix old_failure datetime
Bryan Newbold
2020-10-19
1
-1
/
+1
*
|
CDX: when retrying, do so every 3 seconds up to limit
Bryan Newbold
2020-10-19
1
-5
/
+9
*
|
ingest: try SPNv2 for no-capture and old failures
Bryan Newbold
2020-10-19
1
-1
/
+5
*
|
SPN: more verbose status logging
Bryan Newbold
2020-10-19
1
-0
/
+4
*
|
ingest: disable soft404 and non-hit SPNv2 retries
Bryan Newbold
2020-10-19
1
-4
/
+5
*
|
CDX: revert post-SPN CDX lookup retry to 10 seconds
Bryan Newbold
2020-10-19
1
-1
/
+1
*
|
ingest: catch wayback-fail-after-SPN as separate status
Bryan Newbold
2020-10-19
1
-4
/
+17
*
|
SPN: better log line when starting a request
Bryan Newbold
2020-10-19
1
-0
/
+1
*
|
SPN: look for non-200 CDX responses
Bryan Newbold
2020-10-19
1
-1
/
+1
*
|
SPN: better check for partial URLs returned
Bryan Newbold
2020-10-19
1
-2
/
+2
*
|
CDX fetch: more permissive fuzzy/normalization check
Bryan Newbold
2020-10-19
1
-3
/
+9
*
|
ingest: experimentally reduce CDX API retry delay
Bryan Newbold
2020-10-17
1
-1
/
+1
*
|
notes on 2020-09 re-ingest passes
Bryan Newbold
2020-10-17
1
-0
/
+197
*
|
OA DOIs: partial notes
Bryan Newbold
2020-10-17
1
-0
/
+218
*
|
notes/status on daily ingest
Bryan Newbold
2020-10-17
1
-0
/
+193
*
|
update SQL ingest monitoring commands to be past-month by default
Bryan Newbold
2020-10-17
1
-5
/
+5
*
|
ingest: handle cookieAbsent and partial SPNv2 URL reponse cases better
Bryan Newbold
2020-10-17
1
-0
/
+31
*
|
and another sha1
Bryan Newbold
2020-10-13
1
-0
/
+1
*
|
another day, another bad PDF sha1
Bryan Newbold
2020-10-13
1
-0
/
+1
*
|
store no-capture URLs in terminal_url
Bryan Newbold
2020-10-12
3
-3
/
+39
*
|
start 2020-10 ingest notes
Bryan Newbold
2020-10-11
1
-0
/
+42
*
|
update unpaywall 2020-04 notes
Bryan Newbold
2020-10-11
1
-0
/
+32
*
|
OAI-PMH ingest progress timestamps
Bryan Newbold
2020-10-11
1
-0
/
+13
*
|
another bad PDF sha1
Bryan Newbold
2020-10-11
1
-0
/
+1
*
|
yet more bad sha1 PDFs to skip
Bryan Newbold
2020-10-10
1
-0
/
+20
*
|
notes on file_meta task (from august)
Bryan Newbold
2020-10-01
1
-0
/
+66
*
|
dump_file_meta helper
Bryan Newbold
2020-10-01
1
-0
/
+12
*
|
update README (public)
Bryan Newbold
2020-10-01
1
-17
/
+27
*
|
ingest: small bugfix to print pdfextract status on SUCCESS
Bryan Newbold
2020-09-17
1
-1
/
+1
*
|
more bad PDF sha1
Bryan Newbold
2020-09-17
1
-0
/
+2
*
|
yet another broken PDF (sha1)
Bryan Newbold
2020-09-16
1
-0
/
+1
*
|
html: handle JMIR URL pattern
Bryan Newbold
2020-09-15
1
-0
/
+6
*
|
updated sandcrawler-db stats
Bryan Newbold
2020-09-15
2
-6
/
+346
*
|
skip citation_pdf_url if it is a link loop
Bryan Newbold
2020-09-14
1
-2
/
+8
*
|
html parse: add another generic fulltext pattern
Bryan Newbold
2020-09-14
1
-1
/
+10
*
|
ingest: treat text/xml as XHTML in pdf ingest
Bryan Newbold
2020-09-14
1
-1
/
+1
*
|
OAI-PMH ingest notes
Bryan Newbold
2020-09-03
1
-0
/
+232
*
|
daily ingest notes
Bryan Newbold
2020-09-02
1
-0
/
+202
*
|
follow-up notes on processing 'holes'
Bryan Newbold
2020-09-02
1
-0
/
+19
*
|
unpaywall ingest follow-up
Bryan Newbold
2020-09-02
1
-0
/
+115
*
|
more bad SHA1 PDF
Bryan Newbold
2020-09-02
1
-0
/
+2
*
|
another bad PDF sha1
Bryan Newbold
2020-09-01
1
-0
/
+1
*
|
another bad PDF sha1
Bryan Newbold
2020-08-24
1
-0
/
+1
*
|
html: handle embed with mangled 'src' attribute
Bryan Newbold
2020-08-24
1
-1
/
+1
*
|
WIP weekly re-ingest script
Bryan Newbold
2020-08-17
2
-0
/
+97
*
|
another bad PDF sha1
Bryan Newbold
2020-08-17
1
-0
/
+1
*
|
another bad PDF sha1
Bryan Newbold
2020-08-15
1
-0
/
+1
[prev]
[next]