index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
|
persist grobid: add option to skip S3 upload
Bryan Newbold
2020-03-19
2
-7
/
+14
*
|
ingest: log every URL (from ia code side)
Bryan Newbold
2020-03-18
1
-0
/
+1
*
|
implement (unused) force_get flag for SPN2
Bryan Newbold
2020-03-18
2
-4
/
+19
*
|
unpaywall large ingest notes
Bryan Newbold
2020-03-17
1
-0
/
+10
*
|
make monitoring commands ingest_request local, not ingest_file_result
Bryan Newbold
2020-03-17
1
-2
/
+2
*
|
work around local redirect (resource.location)
Bryan Newbold
2020-03-17
1
-1
/
+6
*
|
Merge branch 'martin-abstract-class-process' into 'master'
bnewbold
2020-03-12
1
-0
/
+6
|
\
\
|
*
|
workers: add explicit process to base class
Martin Czygan
2020-03-12
1
-0
/
+6
|
|
/
*
|
pipenv: work around zipp issue
Bryan Newbold
2020-03-10
2
-4
/
+16
*
|
pipenv: add urlcanon; update pipefile.lock
Bryan Newbold
2020-03-10
2
-209
/
+221
*
|
DOI prefix example queries (SQL)
Bryan Newbold
2020-03-10
1
-3
/
+17
*
|
use local env in python scripts
Bryan Newbold
2020-03-10
3
-3
/
+3
*
|
url cleaning (canonicalization) for ingest base_url
Bryan Newbold
2020-03-10
4
-4
/
+21
*
|
ingest_file: --no-spn2 flag for single command
Bryan Newbold
2020-03-10
1
-1
/
+6
*
|
helpful daily/weekly monitoring SQL queries
Bryan Newbold
2020-03-10
1
-0
/
+94
*
|
sandcrawler schema: add MD5 index
Bryan Newbold
2020-03-05
1
-0
/
+1
|
/
*
more unpaywall ingest notes
Bryan Newbold
2020-03-05
1
-0
/
+416
*
ingestrequest_row2json: skip on unicode errors
Bryan Newbold
2020-03-05
1
-1
/
+4
*
fixes to ingest-request persist
Bryan Newbold
2020-03-05
2
-4
/
+2
*
persist: ingest_request tool (with no ingest_file_result)
Bryan Newbold
2020-03-05
3
-1
/
+48
*
ingest_tool: force-recrawl arg
Bryan Newbold
2020-03-05
1
-0
/
+5
*
ia: catch wayback ChunkedEncodingError
Bryan Newbold
2020-03-05
1
-0
/
+3
*
ingest: make content-decoding more robust
Bryan Newbold
2020-03-03
1
-1
/
+2
*
update (and move) ingest notes
Bryan Newbold
2020-03-03
6
-0
/
+480
*
make gzip content-encoding path more robust
Bryan Newbold
2020-03-03
1
-1
/
+10
*
ingest: crude content-encoding support
Bryan Newbold
2020-03-02
1
-1
/
+19
*
ingest: add force_recrawl flag to skip historical wayback lookup
Bryan Newbold
2020-03-02
2
-3
/
+6
*
more SQL queries
Bryan Newbold
2020-03-02
1
-0
/
+57
*
remove protocols.io octet-stream hack
Bryan Newbold
2020-03-02
1
-6
/
+2
*
more mime normalization
Bryan Newbold
2020-02-27
1
-1
/
+18
*
ingest: narrow xhtml filter
Bryan Newbold
2020-02-25
1
-1
/
+1
*
pdftrio: tweaks to avoid connection errors
Bryan Newbold
2020-02-24
1
-1
/
+9
*
fix warc_offset -> offset
Bryan Newbold
2020-02-24
1
-1
/
+1
*
ingest: handle broken revisit records
Bryan Newbold
2020-02-24
1
-1
/
+4
*
recent sandcrawler-db / ingest stats (interesting)
Bryan Newbold
2020-02-24
2
-0
/
+488
*
ingest: handle missing chemrxvi tag
Bryan Newbold
2020-02-24
1
-1
/
+1
*
ingest: treat CDX lookup error as a wayback-error
Bryan Newbold
2020-02-24
1
-1
/
+4
*
ingest: more direct americanarchivist PDF url guess
Bryan Newbold
2020-02-24
1
-0
/
+4
*
ingest backfill notes
Bryan Newbold
2020-02-24
3
-0
/
+150
*
ingest: make ehp.niehs.nih.gov rule more robust
Bryan Newbold
2020-02-24
1
-2
/
+3
*
small tweak to americanarchivist.org URL extraction
Bryan Newbold
2020-02-24
1
-1
/
+1
*
fetch_petabox_body: allow non-200 status code fetches
Bryan Newbold
2020-02-24
1
-2
/
+10
*
allow fuzzy revisit matches
Bryan Newbold
2020-02-24
1
-1
/
+26
*
ingest: more revisit fixes
Bryan Newbold
2020-02-22
1
-4
/
+4
*
html: more publisher-specific fulltext extraction tricks
Bryan Newbold
2020-02-22
1
-0
/
+47
*
ia: improve warc/revisit implementation
Bryan Newbold
2020-02-22
1
-26
/
+46
*
html: degruyter extraction; disabled journals.lww.com
Bryan Newbold
2020-02-22
1
-0
/
+19
*
ingest: include better terminal URL/status_code/dt
Bryan Newbold
2020-02-22
1
-0
/
+8
*
ingest: skip more non-pdf, non-paper domains
Bryan Newbold
2020-02-22
1
-0
/
+9
*
cdx: handle empty/null CDX response
Bryan Newbold
2020-02-22
1
-0
/
+2
[prev]
[next]