index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
|
ingest: check URL blocklist again after redirects
Bryan Newbold
2021-09-03
1
-0
/
+7
*
|
OAI-PMH patch and ingest improvement notes
Bryan Newbold
2021-09-03
2
-204
/
+1578
*
|
commit old patch crawl notes (dec 2020)
Bryan Newbold
2021-09-03
1
-0
/
+1
*
|
commit old arxiv ingest notes
Bryan Newbold
2021-09-03
1
-0
/
+12
*
|
kafka re-balancing tweaks
Bryan Newbold
2021-09-03
1
-1
/
+2
*
|
refactor and expand wall/block/cookie URL patterns
Bryan Newbold
2021-09-03
2
-6
/
+39
*
|
HTML ingest: several more PDF fulltext URL patterns
Bryan Newbold
2021-09-03
1
-0
/
+87
*
|
HTML ingest: skip noisy print() statement
Bryan Newbold
2021-09-03
1
-1
/
+1
*
|
commit old patch notes (will rework)
Bryan Newbold
2021-09-03
1
-0
/
+110
*
|
MAG post-crawl stats (5m+ new PDFs crawled successfully)
Bryan Newbold
2021-09-02
1
-0
/
+124
*
|
HTML ingest: more meta-URI prefixes
Bryan Newbold
2021-08-24
1
-2
/
+8
*
|
html ingest: detect some blog platforms, and allow lower wordcount threshold
Bryan Newbold
2021-08-16
1
-0
/
+6
*
|
html ingest: detect domain homepage (no path) as special case
Bryan Newbold
2021-08-16
1
-0
/
+8
*
|
html ingest: skip 'about:blank'
Bryan Newbold
2021-08-16
1
-0
/
+3
*
|
MAG and OAI-PMH crawl/processing notes
Bryan Newbold
2021-08-13
2
-0
/
+480
*
|
2021-07 unpaywall crawl wrap-up notes
Bryan Newbold
2021-07-30
1
-12
/
+108
*
|
more bad PDF hashes
Bryan Newbold
2021-07-26
1
-0
/
+2
*
|
ingest: fix postgrest lookup bug (double get of GROBID)
Bryan Newbold
2021-07-26
1
-1
/
+1
*
|
reingest: skip spn2 'unknown' errors
Bryan Newbold
2021-07-21
2
-0
/
+2
*
|
more blocked-cookie patterns; fix old typo
Bryan Newbold
2021-07-14
1
-2
/
+2
*
|
unpaywall 2021-07 crawl partial notes
Bryan Newbold
2021-07-14
1
-0
/
+224
*
|
CI: wget used in pig CI scripts
Bryan Newbold
2021-07-13
1
-1
/
+1
*
|
CI: new sbt debian repository
Bryan Newbold
2021-07-13
1
-4
/
+3
*
|
Revert "CI: sbt bintray is gone, but ubuntu focal version should work"
Bryan Newbold
2021-07-13
1
-0
/
+5
*
|
CI: sbt bintray is gone, but ubuntu focal version should work
Bryan Newbold
2021-07-13
1
-5
/
+0
*
|
another bad PDF sha1
Bryan Newbold
2021-07-13
1
-0
/
+1
*
|
crawl: SPN2 non-200 success code path
Bryan Newbold
2021-07-13
1
-11
/
+25
*
|
crawl: SPN self-redirect hack
Bryan Newbold
2021-07-13
1
-0
/
+9
*
|
crawl: small comment updates
Bryan Newbold
2021-07-13
1
-3
/
+6
*
|
another lowercase DOI in an (unused?) script
Bryan Newbold
2021-07-13
1
-1
/
+1
*
|
gitignore: samples/
Bryan Newbold
2021-07-13
1
-0
/
+1
*
|
add crossref postgrest fetch support to python db helpers
Bryan Newbold
2021-06-02
1
-0
/
+9
*
|
crossref DB proposal, and include in SQL schema
Bryan Newbold
2021-06-02
2
-0
/
+93
*
|
notes on large-domain ingest tweaks
Bryan Newbold
2021-05-27
1
-0
/
+480
*
|
2021-04 unpaywall crawl notes
Bryan Newbold
2021-05-27
1
-0
/
+368
*
|
python Makefile: fix test/*.py linting with newer pylint
Bryan Newbold
2021-05-24
1
-1
/
+1
*
|
ingest: fix html PDF extraction exception catch behavior
Bryan Newbold
2021-05-24
1
-3
/
+2
*
|
ingest PDF extraction updates
Bryan Newbold
2021-05-21
3
-2
/
+74
*
|
better OSF preprint download re-writing
Bryan Newbold
2021-05-21
1
-6
/
+23
*
|
html ingest: remove whitespace around relative URLs (eg, for d-lib)
Bryan Newbold
2021-05-21
1
-1
/
+1
*
|
add cdx_collection.py python script (from scratch repo)
Bryan Newbold
2021-05-04
1
-0
/
+80
*
|
sql: do periodically retry spn2-wayback-error
Bryan Newbold
2021-04-27
2
-2
/
+0
*
|
ingest: cap max body size to ~128 MByte
Bryan Newbold
2021-04-27
1
-0
/
+6
*
|
persist: skip very long URLs
Bryan Newbold
2021-04-12
1
-0
/
+4
*
|
reingest scripts to run as sandcrawler
Bryan Newbold
2021-04-09
2
-12
/
+12
*
|
sql: notes on sql restore
Bryan Newbold
2021-04-09
1
-0
/
+9
*
|
sql: update paths to work with svc506 machine
Bryan Newbold
2021-04-09
12
-49
/
+49
*
|
sql: before/after pg13 table size stats
Bryan Newbold
2021-04-09
2
-1
/
+43
*
|
sql: update periodic retry/reingest scripts
Bryan Newbold
2021-04-09
4
-6
/
+14
*
|
update default postgrest ('db') API endpoint
Bryan Newbold
2021-04-09
1
-1
/
+1
[prev]
[next]