index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
pdftrio: fix error nesting in pdftrio key
Bryan Newbold
2020-02-18
1
-12
/
+20
*
include rel and oa_status in ingest request 'extra'
Bryan Newbold
2020-02-18
3
-2
/
+6
*
ingest: bulk workers don't hit SPNv2
Bryan Newbold
2020-02-13
1
-0
/
+2
*
pdftrio fixes from testing
Bryan Newbold
2020-02-13
1
-3
/
+9
*
move pdf_trio results back under key in JSON/Kafka
Bryan Newbold
2020-02-13
3
-22
/
+49
*
pdftrio JSON object as top-level in Kafka results
Bryan Newbold
2020-02-12
1
-16
/
+16
*
pdftrio: small fixes from testing
Bryan Newbold
2020-02-12
1
-2
/
+2
*
pdftrio basic python code
Bryan Newbold
2020-02-12
8
-3
/
+395
*
add minio.conf
Bryan Newbold
2020-02-12
1
-0
/
+14
*
dump_regrobid_pdf_petabox.sql script
Bryan Newbold
2020-02-12
1
-0
/
+15
*
sandcrawler-db extra stats
Bryan Newbold
2020-02-12
1
-0
/
+42
*
jan 2020 bulk ingest notes
Bryan Newbold
2020-02-12
1
-0
/
+26
*
pdftrio proposal and start on schema+kafka
Bryan Newbold
2020-02-12
3
-0
/
+122
*
add notes on recent ingest and backfill tasks
Bryan Newbold
2020-02-05
3
-0
/
+221
*
add ingestrequest_row2json.py
Bryan Newbold
2020-02-05
1
-0
/
+48
*
fix persist bug where ingest_request_source not saved
Bryan Newbold
2020-02-05
1
-0
/
+1
*
fix bug where ingest_request extra fields not persisted
Bryan Newbold
2020-02-05
1
-1
/
+2
*
handle alternative dt format in WARC headers
Bryan Newbold
2020-02-05
1
-2
/
+4
*
decrease SPNv2 polling timeout to 3 minutes
Bryan Newbold
2020-02-05
1
-2
/
+2
*
improvements to reliability from prod testing
Bryan Newbold
2020-02-03
2
-7
/
+20
*
hack-y backoff ingest attempt
Bryan Newbold
2020-02-03
2
-3
/
+26
*
more random sandcrawler-db queries
Bryan Newbold
2020-02-03
2
-32
/
+62
*
grobid petabox: fix fetch body/content
Bryan Newbold
2020-02-03
1
-1
/
+1
*
more SQL commands
Bryan Newbold
2020-02-02
1
-0
/
+15
*
wayback: try to resolve HTTPException due to many HTTP headers
Bryan Newbold
2020-02-02
1
-1
/
+9
*
sandcrawler_worker: ingest worker distinct consumer groups
Bryan Newbold
2020-01-29
1
-1
/
+3
*
2020q1 fulltext ingest plans
Bryan Newbold
2020-01-29
1
-0
/
+272
*
grobid worker: catch PetaboxError also
Bryan Newbold
2020-01-28
1
-2
/
+2
*
worker kafka setting tweaks
Bryan Newbold
2020-01-28
1
-2
/
+4
*
make grobid-extract worker batch size 1
Bryan Newbold
2020-01-28
1
-0
/
+1
*
sql stats: typo fix
Bryan Newbold
2020-01-28
1
-1
/
+1
*
sql howto: database dumps
Bryan Newbold
2020-01-28
1
-0
/
+7
*
workers: yes, poll is necessary
Bryan Newbold
2020-01-28
1
-1
/
+1
*
grobid worker: always set a key in response
Bryan Newbold
2020-01-28
1
-4
/
+25
*
fix kafka worker partition-specific error
Bryan Newbold
2020-01-28
1
-1
/
+1
*
fix WaybackError exception formating
Bryan Newbold
2020-01-28
1
-1
/
+1
*
fix elif syntax error
Bryan Newbold
2020-01-28
1
-1
/
+1
*
block springer page-one domain
Bryan Newbold
2020-01-28
1
-0
/
+3
*
clarify petabox fetch behavior
Bryan Newbold
2020-01-28
1
-3
/
+6
*
re-enable figshare and zenodo crawling
Bryan Newbold
2020-01-21
1
-8
/
+0
*
persist grobid: actually, status_code is required
Bryan Newbold
2020-01-21
2
-3
/
+10
*
ingest: check for null-body before file_meta
Bryan Newbold
2020-01-21
1
-0
/
+3
*
wayback: replay redirects have X-Archive-Redirect-Reason
Bryan Newbold
2020-01-21
1
-2
/
+4
*
persist: work around GROBID timeouts with no status_code
Bryan Newbold
2020-01-21
2
-3
/
+3
*
grobid: fix error_msg typo; set status_code for timeouts
Bryan Newbold
2020-01-21
1
-1
/
+2
*
add 200 second timeout to GROBID requests
Bryan Newbold
2020-01-17
1
-8
/
+15
*
add SKIP log line for skip-url-blocklist path
Bryan Newbold
2020-01-17
1
-0
/
+1
*
ingest: add URL blocklist feature
Bryan Newbold
2020-01-17
2
-4
/
+49
*
handle UnicodeDecodeError in the other GET instance
Bryan Newbold
2020-01-15
1
-0
/
+2
*
increase SPNv2 polling timeout to 4 minutes
Bryan Newbold
2020-01-15
1
-1
/
+3
[prev]
[next]