index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
sandcrawler
Commit message (
Expand
)
Author
Age
Files
Lines
*
wayback: use same 5xx/4xx-allowing tricks for replay body fetch as for replay...
Bryan Newbold
2022-07-15
1
-7
/
+7
*
cdx api: add another allowable URL fuzzy-match pattern (double slashes)
Bryan Newbold
2022-07-15
1
-0
/
+9
*
ingest: more bogus domain patterns
Bryan Newbold
2022-07-15
1
-0
/
+3
*
spn2: handle case of re-attempting a recent crawl (race condition)
Bryan Newbold
2022-07-15
1
-0
/
+14
*
html: fulltext URL prefixes to skip; also fix broken pattern matching
Bryan Newbold
2022-07-15
1
-4
/
+19
*
ingest: another form of cookie block URL
Bryan Newbold
2022-07-15
1
-0
/
+2
*
HTML ingest: most sub-resource patterns to skip
Bryan Newbold
2022-07-15
1
-1
/
+13
*
cdx lookups: prioritize truely exact URL matches
Bryan Newbold
2022-07-14
1
-0
/
+1
*
ingest: handle another type of wayback redirect
Bryan Newbold
2022-07-14
1
-2
/
+5
*
yet another bad PDF
Bryan Newbold
2022-07-13
1
-0
/
+1
*
wayback fetch: handle upstream 5xx replays
Bryan Newbold
2022-07-13
1
-4
/
+15
*
shorten default HTTP backoff factor
Bryan Newbold
2022-07-13
1
-1
/
+1
*
ingest: random site PDF link pattern
Bryan Newbold
2022-07-12
1
-0
/
+7
*
ingest: doaj.org article landing page access links
Bryan Newbold
2022-07-12
2
-1
/
+12
*
ingest: IEEE domain is blocking us
Bryan Newbold
2022-07-07
1
-1
/
+2
*
ingest: catch more ConnectionErrors (SPN, replay fetch, GROBID)
Bryan Newbold
2022-05-16
2
-4
/
+19
*
ingest: skip arxiv.org DOIs, we already direct-ingest
Bryan Newbold
2022-05-11
1
-0
/
+1
*
ingest spn2: fix tests
Bryan Newbold
2022-05-05
2
-1
/
+2
*
ingest: more loginwall patterns
Bryan Newbold
2022-05-05
1
-0
/
+3
*
SPNv2: several fixes for prod throughput
Bryan Newbold
2022-04-26
1
-11
/
+34
*
make fmt
Bryan Newbold
2022-04-26
1
-2
/
+5
*
block isiarticles.com from future PDF crawls
Bryan Newbold
2022-04-20
1
-0
/
+2
*
ingest: drive.google.com ingest support
Bryan Newbold
2022-04-04
1
-0
/
+8
*
filesets: fix archive.org path naming
Bryan Newbold
2022-03-29
1
-7
/
+8
*
bugfix: sha1/md5 typo
Bryan Newbold
2022-03-23
1
-1
/
+1
*
file ingest: don't 'backoff' on spn2 backoff error
Bryan Newbold
2022-03-22
2
-0
/
+8
*
small lint/typo/fmt fixes
Bryan Newbold
2022-02-24
3
-5
/
+5
*
another bad PDF sha1
Bryan Newbold
2022-02-23
1
-0
/
+1
*
ingest: fix mistakenly commented except block (?)
Bryan Newbold
2022-02-18
1
-4
/
+3
*
ingest: handle more fileset failure modes
Bryan Newbold
2022-02-18
2
-3
/
+30
*
yet another bad PDF sha1
Bryan Newbold
2022-02-08
1
-0
/
+1
*
sandcrawler: additional extracts, mostly OJS
Bryan Newbold
2022-01-13
1
-1
/
+23
*
filesets: more figshare URL patterns
Bryan Newbold
2022-01-13
1
-0
/
+13
*
fileset ingest: better verification of resources
Bryan Newbold
2022-01-13
1
-7
/
+23
*
ingest: PDF pattern for integrityresjournals.org
Bryan Newbold
2022-01-13
1
-0
/
+8
*
null-body -> empty-blob
Bryan Newbold
2022-01-13
3
-4
/
+8
*
spn: handle blocked-url (etc) better
Bryan Newbold
2022-01-11
1
-0
/
+10
*
filesets: handle weird figshare link-only case better
Bryan Newbold
2021-12-16
1
-1
/
+4
*
lint ('not in')
Bryan Newbold
2021-12-15
1
-2
/
+2
*
more fileset ingest tweaks
Bryan Newbold
2021-12-15
2
-0
/
+7
*
fileset ingest: more requests timeouts, sessions
Bryan Newbold
2021-12-15
3
-37
/
+68
*
fileset ingest: create tmp subdirectories if needed
Bryan Newbold
2021-12-15
1
-0
/
+5
*
fileset ingest: configure IA session from env
Bryan Newbold
2021-12-15
1
-1
/
+6
*
fileset ingest: actually use spn2 CLI flag
Bryan Newbold
2021-12-11
2
-3
/
+4
*
grobid: set a maximum file size (256 MByte)
Bryan Newbold
2021-12-07
1
-0
/
+8
*
codespell typos in python (comments)
Bryan Newbold
2021-11-24
4
-4
/
+4
*
html_meta: actual typo in code (CSS selector) caught by codespell
Bryan Newbold
2021-11-24
1
-1
/
+1
*
make fmt
Bryan Newbold
2021-11-16
1
-1
/
+1
*
SPNv2: make 'resources' optional
Bryan Newbold
2021-11-16
1
-1
/
+1
*
grobid: handle XML parsing errors, and have them recorded in sandcrawler-db
Bryan Newbold
2021-11-12
1
-1
/
+5
[next]