index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
...
|
*
improve test running and config
Bryan Newbold
2020-10-29
3
-3
/
+7
|
*
misc: type annotations, fix parse_cdx_datetime
Bryan Newbold
2020-10-29
1
-14
/
+18
|
*
html: initial ingest implementation
Bryan Newbold
2020-10-29
1
-0
/
+193
|
*
html: more metadata tests
Bryan Newbold
2020-10-29
2
-0
/
+2453
|
*
pipenv: braveblocker, dynaconf, sentry-sdk
Bryan Newbold
2020-10-29
2
-29
/
+34
|
*
html: more biblio selectors; resource extraction
Bryan Newbold
2020-10-29
1
-0
/
+102
|
*
HTML meta: more from online hunting/research
Bryan Newbold
2020-10-27
1
-3
/
+54
|
*
HTML metadata: fix type warnings
Bryan Newbold
2020-10-27
2
-2
/
+5
|
*
grobid2json: fix deprecation warning
Bryan Newbold
2020-10-27
1
-1
/
+1
|
*
pylint: fix error on selectolax import
Bryan Newbold
2020-10-27
1
-0
/
+1
|
*
pytest: remove unknown 'ignore' from config file
Bryan Newbold
2020-10-27
1
-2
/
+0
|
*
new dependencies for HTML metadata parsing
Bryan Newbold
2020-10-27
2
-926
/
+184
|
*
start HTML metadata extraction code
Bryan Newbold
2020-10-27
6
-0
/
+2858
*
|
unpaywall oct 2020 crawl notes
Bryan Newbold
2020-11-02
1
-45
/
+82
*
|
more notes on unpaywall ingest from last week
Bryan Newbold
2020-10-27
1
-0
/
+73
*
|
kafka topics for fatcat -> scholar pipeline
Bryan Newbold
2020-10-27
1
-0
/
+19
|
/
*
ingest: skip JSTOR DOI prefixes
Bryan Newbold
2020-10-23
1
-0
/
+3
*
Revert "reimplement worker timeout with multiprocessing"
Bryan Newbold
2020-10-22
1
-17
/
+23
*
ingest: decrease CDX timeout retries again
Bryan Newbold
2020-10-22
1
-1
/
+1
*
make: coverage report and HTML in one go
Bryan Newbold
2020-10-22
2
-5
/
+2
*
reimplement worker timeout with multiprocessing
Bryan Newbold
2020-10-22
1
-23
/
+17
*
SQL: update weekly/quarterly ingest retry scripts
Bryan Newbold
2020-10-21
5
-18
/
+119
*
sql stats: larger limits (more complete lists)
Bryan Newbold
2020-10-21
1
-8
/
+8
*
ingest: fix WaybackContentError typo
Bryan Newbold
2020-10-21
1
-1
/
+1
*
ingest: add a check for blocked-cookie before trying PDF url extraction
Bryan Newbold
2020-10-21
1
-0
/
+11
*
differential wayback-error from wayback-content-error
Bryan Newbold
2020-10-21
7
-18
/
+22
*
persist PDF extraction in ingest pipeline
Bryan Newbold
2020-10-20
1
-4
/
+16
*
ingest: add a cdx-error slowdown delay
Bryan Newbold
2020-10-19
1
-0
/
+3
*
SPN CDX delay now seems reasonable; increase to 40sec to catch most
Bryan Newbold
2020-10-19
1
-1
/
+1
*
ingest: fix old_failure datetime
Bryan Newbold
2020-10-19
1
-1
/
+1
*
CDX: when retrying, do so every 3 seconds up to limit
Bryan Newbold
2020-10-19
1
-5
/
+9
*
ingest: try SPNv2 for no-capture and old failures
Bryan Newbold
2020-10-19
1
-1
/
+5
*
SPN: more verbose status logging
Bryan Newbold
2020-10-19
1
-0
/
+4
*
ingest: disable soft404 and non-hit SPNv2 retries
Bryan Newbold
2020-10-19
1
-4
/
+5
*
CDX: revert post-SPN CDX lookup retry to 10 seconds
Bryan Newbold
2020-10-19
1
-1
/
+1
*
ingest: catch wayback-fail-after-SPN as separate status
Bryan Newbold
2020-10-19
1
-4
/
+17
*
SPN: better log line when starting a request
Bryan Newbold
2020-10-19
1
-0
/
+1
*
SPN: look for non-200 CDX responses
Bryan Newbold
2020-10-19
1
-1
/
+1
*
SPN: better check for partial URLs returned
Bryan Newbold
2020-10-19
1
-2
/
+2
*
CDX fetch: more permissive fuzzy/normalization check
Bryan Newbold
2020-10-19
1
-3
/
+9
*
ingest: experimentally reduce CDX API retry delay
Bryan Newbold
2020-10-17
1
-1
/
+1
*
notes on 2020-09 re-ingest passes
Bryan Newbold
2020-10-17
1
-0
/
+197
*
OA DOIs: partial notes
Bryan Newbold
2020-10-17
1
-0
/
+218
*
notes/status on daily ingest
Bryan Newbold
2020-10-17
1
-0
/
+193
*
update SQL ingest monitoring commands to be past-month by default
Bryan Newbold
2020-10-17
1
-5
/
+5
*
ingest: handle cookieAbsent and partial SPNv2 URL reponse cases better
Bryan Newbold
2020-10-17
1
-0
/
+31
*
and another sha1
Bryan Newbold
2020-10-13
1
-0
/
+1
*
another day, another bad PDF sha1
Bryan Newbold
2020-10-13
1
-0
/
+1
*
store no-capture URLs in terminal_url
Bryan Newbold
2020-10-12
3
-3
/
+39
*
start 2020-10 ingest notes
Bryan Newbold
2020-10-11
1
-0
/
+42
[prev]
[next]