index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
*
SPNv2 doesn't support FTP; add a live test for non-revist FTP
Bryan Newbold
2020-01-14
2
-0
/
+26
*
more ftp status 226 support
Bryan Newbold
2020-01-14
5
-9
/
+23
*
add live tests for ftp, revisits
Bryan Newbold
2020-01-14
1
-1
/
+36
*
basic FTP ingest support; revist record resolution
Bryan Newbold
2020-01-14
2
-35
/
+78
*
arabesque2ingestrequest: ingest type flag
Bryan Newbold
2020-01-14
1
-1
/
+4
*
better print() output
Bryan Newbold
2020-01-10
2
-4
/
+4
*
fix trivial typo
Bryan Newbold
2020-01-10
1
-1
/
+1
*
hack/workaround for protocols.io octet PDFs
Bryan Newbold
2020-01-10
1
-2
/
+4
*
html extract: protocols.io, fix americanarchivist
Bryan Newbold
2020-01-10
1
-1
/
+7
*
fix redirect replay fetch method
Bryan Newbold
2020-01-10
1
-1
/
+4
*
limit length of error messages
Bryan Newbold
2020-01-10
1
-4
/
+4
*
setup notes and CI (for psycopg2 dependency)
Bryan Newbold
2020-01-10
2
-1
/
+6
*
handle SPNv2-then-CDX lookup failures
Bryan Newbold
2020-01-10
1
-6
/
+23
*
SPNv2 hack specifically for elsevier lookups
Bryan Newbold
2020-01-10
1
-0
/
+15
*
add support for redirect lookups from replay
Bryan Newbold
2020-01-10
1
-9
/
+69
*
more general ingest teaks and affordances
Bryan Newbold
2020-01-10
2
-15
/
+42
*
add sleep-and-retry workaround for CDX after SPNv2
Bryan Newbold
2020-01-10
1
-1
/
+9
*
more ingest HTML extraction hacks
Bryan Newbold
2020-01-10
1
-6
/
+46
*
more live tests (for regressions)
Bryan Newbold
2020-01-10
2
-0
/
+42
*
disable CDX best lookup 'collapse'; leave comment
Bryan Newbold
2020-01-10
1
-1
/
+3
*
hack: reverse sort of CDX exact seems broken with SPNv2 results
Bryan Newbold
2020-01-10
1
-1
/
+1
*
many publisher-specific ingest improvements
Bryan Newbold
2020-01-10
1
-4
/
+96
*
improve ingest robustness (for legacy requests)
Bryan Newbold
2020-01-10
1
-6
/
+12
*
update Pipfile to be xenial-compatible
Bryan Newbold
2020-01-09
2
-16
/
+26
*
support forwarding url types other than pdf_url
Bryan Newbold
2020-01-09
1
-4
/
+5
*
wayback: datetime mismatch as an error
Bryan Newbold
2020-01-09
1
-1
/
+2
*
fill in more html extraction techniques
Bryan Newbold
2020-01-09
1
-7
/
+6
*
refactor ingest to a loop, allowing multiple hops
Bryan Newbold
2020-01-09
2
-27
/
+57
*
add (skipped) live tests for wayback services
Bryan Newbold
2020-01-09
1
-0
/
+73
*
add ingest test file
Bryan Newbold
2020-01-09
1
-0
/
+120
*
lots of progress on wayback refactoring
Bryan Newbold
2020-01-09
3
-51
/
+145
*
location comes as a string, not list
Bryan Newbold
2020-01-09
2
-5
/
+5
*
fix http/https issue with GlobalWayback library
Bryan Newbold
2020-01-09
1
-1
/
+2
*
wayback fetch via replay; confirm hashes in crawl_resource()
Bryan Newbold
2020-01-09
1
-5
/
+40
*
wrap up basic (locally testable) ingest refactor
Bryan Newbold
2020-01-09
3
-182
/
+267
*
cli: allow multiple ingest single types
Bryan Newbold
2020-01-09
1
-3
/
+4
*
example.env: more required vars
Bryan Newbold
2020-01-09
1
-2
/
+7
*
basic elife+plos extraction tests
Bryan Newbold
2020-01-09
3
-0
/
+4842
*
fix grobid test (ISO-8859-1 encoding)
Bryan Newbold
2020-01-09
1
-6
/
+4
*
fix grobid tests for new wayback refactors
Bryan Newbold
2020-01-09
3
-15
/
+17
*
more wayback and SPN tests and fixes
Bryan Newbold
2020-01-09
4
-52
/
+220
*
pipenv: mock testing library
Bryan Newbold
2020-01-09
2
-309
/
+325
*
refactor CdxApiClient, add tests
Bryan Newbold
2020-01-08
2
-40
/
+240
*
refactor SavePaperNowClient and add test
Bryan Newbold
2020-01-07
2
-28
/
+314
*
remove SPNv1 code paths
Bryan Newbold
2020-01-07
2
-65
/
+25
*
sql: more cool random queries
Bryan Newbold
2020-01-02
1
-0
/
+5
*
TODO: check out python ratelimit library
Bryan Newbold
2020-01-02
1
-1
/
+1
*
minio version/compression notes
Bryan Newbold
2020-01-02
1
-0
/
+11
*
handle grobid2json errors in calling code instead
Bryan Newbold
2020-01-02
1
-1
/
+7
*
db: move duplicate row filtering into DB insert helpers
Bryan Newbold
2020-01-02
2
-15
/
+26
[next]