index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
Commit message (
Expand
)
Author
Age
Files
Lines
*
html: worldscientific PDF URL extraction
Bryan Newbold
2022-10-24
1
-0
/
+16
*
html: pubpub platform detection
Bryan Newbold
2022-10-24
1
-0
/
+2
*
persist: skip huge URLs
Bryan Newbold
2022-09-28
1
-0
/
+4
*
filesets: handle unknown file sizes (mypy lint fix)
Bryan Newbold
2022-09-28
1
-1
/
+1
*
update oai-pmh ingest request transform script
Bryan Newbold
2022-09-28
1
-2
/
+38
*
pytest: supress another deprecationwarning
Bryan Newbold
2022-09-14
1
-0
/
+1
*
spn2: fix tests by not retrying on HTTP 500
Bryan Newbold
2022-09-14
1
-1
/
+3
*
catch poppler 'ValueError' when parsing PDFs
Bryan Newbold
2022-09-14
1
-1
/
+2
*
bad PDF sha1
Bryan Newbold
2022-09-12
1
-0
/
+4
*
bad PDF sha1
Bryan Newbold
2022-09-11
1
-0
/
+2
*
another bad PDF sha1
Bryan Newbold
2022-09-09
1
-0
/
+1
*
yet more bad PDF hashes
Bryan Newbold
2022-09-08
1
-0
/
+4
*
pipenv: removed unused deps; re-lock deps
Bryan Newbold
2022-09-07
2
-783
/
+767
*
html ingest: handle TEI-XML parse error
Bryan Newbold
2022-07-28
1
-1
/
+4
*
yet another bad PDF sha1
Bryan Newbold
2022-07-27
1
-0
/
+1
*
CDX: skip sha-256 digests
Bryan Newbold
2022-07-25
1
-1
/
+5
*
yet another bad SHA1 PDF hash
Bryan Newbold
2022-07-24
1
-0
/
+1
*
ingest: bump max-hops from 6 to 8
Bryan Newbold
2022-07-20
1
-1
/
+1
*
ingest: more PDF fulltext tricks
Bryan Newbold
2022-07-20
2
-0
/
+36
*
ingest: more PDF fulltext URL patterns
Bryan Newbold
2022-07-20
1
-0
/
+42
*
doaj and unpaywall transforms: more domains to skip
Bryan Newbold
2022-07-20
2
-3
/
+1
*
ingest: record bad GZIP transfer decode, instead of crashing (HTML)
Bryan Newbold
2022-07-18
1
-1
/
+4
*
make fmt
Bryan Newbold
2022-07-18
1
-1
/
+0
*
cdx: tweak CDX lookups and resolution (sort)
Bryan Newbold
2022-07-16
1
-4
/
+7
*
html ingest: allow fuzzy CDX sha1 match based on encoding/not-encoding
Bryan Newbold
2022-07-16
1
-3
/
+10
*
HTML: no longer extracting citation_pdf_url in main extract function
Bryan Newbold
2022-07-16
1
-24
/
+0
*
html: mangled JSON-in-URL pattern
Bryan Newbold
2022-07-15
1
-0
/
+1
*
html: remove old citation_pdf_url code path
Bryan Newbold
2022-07-15
1
-32
/
+1
*
wayback: use same 5xx/4xx-allowing tricks for replay body fetch as for replay...
Bryan Newbold
2022-07-15
1
-7
/
+7
*
cdx api: add another allowable URL fuzzy-match pattern (double slashes)
Bryan Newbold
2022-07-15
1
-0
/
+9
*
ingest: more bogus domain patterns
Bryan Newbold
2022-07-15
1
-0
/
+3
*
spn2: handle case of re-attempting a recent crawl (race condition)
Bryan Newbold
2022-07-15
1
-0
/
+14
*
html: fulltext URL prefixes to skip; also fix broken pattern matching
Bryan Newbold
2022-07-15
1
-4
/
+19
*
row2json script: fix argument type
Bryan Newbold
2022-07-15
1
-1
/
+1
*
row2json script: add flag to enable recrawling
Bryan Newbold
2022-07-15
1
-1
/
+8
*
ingest: another form of cookie block URL
Bryan Newbold
2022-07-15
1
-0
/
+2
*
HTML ingest: most sub-resource patterns to skip
Bryan Newbold
2022-07-15
1
-1
/
+13
*
cdx lookups: prioritize truely exact URL matches
Bryan Newbold
2022-07-14
1
-0
/
+1
*
ingest: handle another type of wayback redirect
Bryan Newbold
2022-07-14
1
-2
/
+5
*
yet another bad PDF
Bryan Newbold
2022-07-13
1
-0
/
+1
*
wayback fetch: handle upstream 5xx replays
Bryan Newbold
2022-07-13
1
-4
/
+15
*
shorten default HTTP backoff factor
Bryan Newbold
2022-07-13
1
-1
/
+1
*
ingest: random site PDF link pattern
Bryan Newbold
2022-07-12
1
-0
/
+7
*
ingest: doaj.org article landing page access links
Bryan Newbold
2022-07-12
2
-1
/
+12
*
ingest: IEEE domain is blocking us
Bryan Newbold
2022-07-07
1
-1
/
+2
*
ingest: catch more ConnectionErrors (SPN, replay fetch, GROBID)
Bryan Newbold
2022-05-16
2
-4
/
+19
*
ingest: skip arxiv.org DOIs, we already direct-ingest
Bryan Newbold
2022-05-11
1
-0
/
+1
*
python make fmt
Bryan Newbold
2022-05-05
1
-3
/
+1
*
ingest spn2: fix tests
Bryan Newbold
2022-05-05
4
-6
/
+108
*
ingest: more loginwall patterns
Bryan Newbold
2022-05-05
1
-0
/
+3
[next]