index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
location comes as a string, not list
Bryan Newbold
2020-01-09
2
-5
/
+5
*
fix http/https issue with GlobalWayback library
Bryan Newbold
2020-01-09
1
-1
/
+2
*
wayback fetch via replay; confirm hashes in crawl_resource()
Bryan Newbold
2020-01-09
1
-5
/
+40
*
wrap up basic (locally testable) ingest refactor
Bryan Newbold
2020-01-09
3
-182
/
+267
*
cli: allow multiple ingest single types
Bryan Newbold
2020-01-09
1
-3
/
+4
*
example.env: more required vars
Bryan Newbold
2020-01-09
1
-2
/
+7
*
basic elife+plos extraction tests
Bryan Newbold
2020-01-09
3
-0
/
+4842
*
fix grobid test (ISO-8859-1 encoding)
Bryan Newbold
2020-01-09
1
-6
/
+4
*
fix grobid tests for new wayback refactors
Bryan Newbold
2020-01-09
3
-15
/
+17
*
more wayback and SPN tests and fixes
Bryan Newbold
2020-01-09
4
-52
/
+220
*
pipenv: mock testing library
Bryan Newbold
2020-01-09
2
-309
/
+325
*
refactor CdxApiClient, add tests
Bryan Newbold
2020-01-08
2
-40
/
+240
*
refactor SavePaperNowClient and add test
Bryan Newbold
2020-01-07
2
-28
/
+314
*
remove SPNv1 code paths
Bryan Newbold
2020-01-07
2
-65
/
+25
*
sql: more cool random queries
Bryan Newbold
2020-01-02
1
-0
/
+5
*
TODO: check out python ratelimit library
Bryan Newbold
2020-01-02
1
-1
/
+1
*
minio version/compression notes
Bryan Newbold
2020-01-02
1
-0
/
+11
*
handle grobid2json errors in calling code instead
Bryan Newbold
2020-01-02
1
-1
/
+7
*
db: move duplicate row filtering into DB insert helpers
Bryan Newbold
2020-01-02
2
-15
/
+26
*
remove unused filter in grobid worker
Bryan Newbold
2020-01-02
1
-1
/
+0
*
fix dict typo
Bryan Newbold
2020-01-02
1
-1
/
+1
*
improvements to grobid persist worker
Bryan Newbold
2020-01-02
1
-13
/
+16
*
small (syntax?) changes to pig join script
Bryan Newbold
2020-01-02
1
-2
/
+2
*
update persist worker invocation to use batches
Bryan Newbold
2020-01-02
1
-15
/
+55
*
set mimetype when PUT to minio
Bryan Newbold
2020-01-02
1
-0
/
+4
*
fix DB import counting
Bryan Newbold
2020-01-02
1
-4
/
+5
*
include an example environment file
Bryan Newbold
2020-01-02
1
-0
/
+2
*
teixml2json test update for skipping null JSON keys
Bryan Newbold
2020-01-02
1
-10
/
+1
*
fix small errors found by pylint
Bryan Newbold
2020-01-02
2
-1
/
+2
*
fix sandcrawler persist workers
Bryan Newbold
2020-01-02
2
-8
/
+37
*
filter ingest results to not have key conflicts within batch
Bryan Newbold
2020-01-02
1
-1
/
+16
*
db: fancy insert/update separation using postgres xmax
Bryan Newbold
2020-01-02
2
-24
/
+45
*
add PersistGrobidDiskWorker
Bryan Newbold
2020-01-02
2
-0
/
+60
*
flush out minio helper, add to grobid persist
Bryan Newbold
2020-01-02
3
-24
/
+91
*
update minio README
Bryan Newbold
2020-01-02
1
-10
/
+42
*
implement counts properly for persist workers
Bryan Newbold
2020-01-02
1
-15
/
+19
*
improve DB helpers
Bryan Newbold
2020-01-02
1
-26
/
+81
*
be more parsimonious with GROBID metadata
Bryan Newbold
2020-01-02
2
-3
/
+20
*
start work on DB connector and minio client
Bryan Newbold
2020-01-02
2
-0
/
+200
*
have JsonLinePusher continue on JSON decode errors (but count)
Bryan Newbold
2020-01-02
1
-1
/
+5
*
start work on persist workers and tool
Bryan Newbold
2020-01-02
3
-5
/
+336
*
yet more tweaks to ingest proposal
Bryan Newbold
2020-01-02
1
-3
/
+2
*
SQL docs update for diesel change
Bryan Newbold
2020-01-02
2
-0
/
+48
*
move SQL schema to diesel migration pattern
Bryan Newbold
2020-01-02
5
-70
/
+157
*
hadoop job log rename and update
Bryan Newbold
2019-12-27
1
-0
/
+25
*
update job log with pig runs
Bryan Newbold
2019-12-26
1
-0
/
+10
*
update TODO
Bryan Newbold
2019-12-26
1
-1
/
+7
*
kafka topics: compress api-datacite
Bryan Newbold
2019-12-24
1
-1
/
+1
*
basic arabesque2ingestrequest script
Bryan Newbold
2019-12-24
1
-0
/
+69
*
commit grobid_tool transform mode
Bryan Newbold
2019-12-22
1
-0
/
+27
[prev]
[next]