index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
*
more live tests (for regressions)
Bryan Newbold
2020-01-10
2
-0
/
+42
*
disable CDX best lookup 'collapse'; leave comment
Bryan Newbold
2020-01-10
1
-1
/
+3
*
hack: reverse sort of CDX exact seems broken with SPNv2 results
Bryan Newbold
2020-01-10
1
-1
/
+1
*
many publisher-specific ingest improvements
Bryan Newbold
2020-01-10
1
-4
/
+96
*
improve ingest robustness (for legacy requests)
Bryan Newbold
2020-01-10
1
-6
/
+12
*
update Pipfile to be xenial-compatible
Bryan Newbold
2020-01-09
2
-16
/
+26
*
support forwarding url types other than pdf_url
Bryan Newbold
2020-01-09
1
-4
/
+5
*
wayback: datetime mismatch as an error
Bryan Newbold
2020-01-09
1
-1
/
+2
*
fill in more html extraction techniques
Bryan Newbold
2020-01-09
1
-7
/
+6
*
refactor ingest to a loop, allowing multiple hops
Bryan Newbold
2020-01-09
2
-27
/
+57
*
add (skipped) live tests for wayback services
Bryan Newbold
2020-01-09
1
-0
/
+73
*
add ingest test file
Bryan Newbold
2020-01-09
1
-0
/
+120
*
lots of progress on wayback refactoring
Bryan Newbold
2020-01-09
3
-51
/
+145
*
location comes as a string, not list
Bryan Newbold
2020-01-09
2
-5
/
+5
*
fix http/https issue with GlobalWayback library
Bryan Newbold
2020-01-09
1
-1
/
+2
*
wayback fetch via replay; confirm hashes in crawl_resource()
Bryan Newbold
2020-01-09
1
-5
/
+40
*
wrap up basic (locally testable) ingest refactor
Bryan Newbold
2020-01-09
3
-182
/
+267
*
cli: allow multiple ingest single types
Bryan Newbold
2020-01-09
1
-3
/
+4
*
example.env: more required vars
Bryan Newbold
2020-01-09
1
-2
/
+7
*
basic elife+plos extraction tests
Bryan Newbold
2020-01-09
3
-0
/
+4842
*
fix grobid test (ISO-8859-1 encoding)
Bryan Newbold
2020-01-09
1
-6
/
+4
*
fix grobid tests for new wayback refactors
Bryan Newbold
2020-01-09
3
-15
/
+17
*
more wayback and SPN tests and fixes
Bryan Newbold
2020-01-09
4
-52
/
+220
*
pipenv: mock testing library
Bryan Newbold
2020-01-09
2
-309
/
+325
*
refactor CdxApiClient, add tests
Bryan Newbold
2020-01-08
2
-40
/
+240
*
refactor SavePaperNowClient and add test
Bryan Newbold
2020-01-07
2
-28
/
+314
*
remove SPNv1 code paths
Bryan Newbold
2020-01-07
2
-65
/
+25
*
sql: more cool random queries
Bryan Newbold
2020-01-02
1
-0
/
+5
*
TODO: check out python ratelimit library
Bryan Newbold
2020-01-02
1
-1
/
+1
*
minio version/compression notes
Bryan Newbold
2020-01-02
1
-0
/
+11
*
handle grobid2json errors in calling code instead
Bryan Newbold
2020-01-02
1
-1
/
+7
*
db: move duplicate row filtering into DB insert helpers
Bryan Newbold
2020-01-02
2
-15
/
+26
*
remove unused filter in grobid worker
Bryan Newbold
2020-01-02
1
-1
/
+0
*
fix dict typo
Bryan Newbold
2020-01-02
1
-1
/
+1
*
improvements to grobid persist worker
Bryan Newbold
2020-01-02
1
-13
/
+16
*
small (syntax?) changes to pig join script
Bryan Newbold
2020-01-02
1
-2
/
+2
*
update persist worker invocation to use batches
Bryan Newbold
2020-01-02
1
-15
/
+55
*
set mimetype when PUT to minio
Bryan Newbold
2020-01-02
1
-0
/
+4
*
fix DB import counting
Bryan Newbold
2020-01-02
1
-4
/
+5
*
include an example environment file
Bryan Newbold
2020-01-02
1
-0
/
+2
*
teixml2json test update for skipping null JSON keys
Bryan Newbold
2020-01-02
1
-10
/
+1
*
fix small errors found by pylint
Bryan Newbold
2020-01-02
2
-1
/
+2
*
fix sandcrawler persist workers
Bryan Newbold
2020-01-02
2
-8
/
+37
*
filter ingest results to not have key conflicts within batch
Bryan Newbold
2020-01-02
1
-1
/
+16
*
db: fancy insert/update separation using postgres xmax
Bryan Newbold
2020-01-02
2
-24
/
+45
*
add PersistGrobidDiskWorker
Bryan Newbold
2020-01-02
2
-0
/
+60
*
flush out minio helper, add to grobid persist
Bryan Newbold
2020-01-02
3
-24
/
+91
*
update minio README
Bryan Newbold
2020-01-02
1
-10
/
+42
*
implement counts properly for persist workers
Bryan Newbold
2020-01-02
1
-15
/
+19
*
improve DB helpers
Bryan Newbold
2020-01-02
1
-26
/
+81
[next]