index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
tests
Commit message (
Expand
)
Author
Age
Files
Lines
*
make fmt
Bryan Newbold
2022-07-18
1
-1
/
+0
*
HTML: no longer extracting citation_pdf_url in main extract function
Bryan Newbold
2022-07-16
1
-24
/
+0
*
ingest spn2: fix tests
Bryan Newbold
2022-05-05
2
-5
/
+106
*
initial crossref-refs via GROBID helper routine
Bryan Newbold
2021-11-04
5
-1
/
+698
*
updates/corrections to old small.json GROBID metadata example file
Bryan Newbold
2021-10-27
1
-6
/
+1
*
remove grobid2json helper file, replace with grobid_tei_xml
Bryan Newbold
2021-10-27
1
-5
/
+9
*
make fmt (black 21.9b0)
Bryan Newbold
2021-10-27
13
-402
/
+595
*
more progress on type annotations and linting
Bryan Newbold
2021-10-26
2
-2
/
+2
*
live tests: FTP wayback replay now returns 200, not 226
Bryan Newbold
2021-10-26
1
-2
/
+2
*
flake8 clean (with current settings)
Bryan Newbold
2021-10-26
2
-1
/
+2
*
start handling trivial lint cleanups: unused imports, 'is None', etc
Bryan Newbold
2021-10-26
10
-42
/
+26
*
make fmt
Bryan Newbold
2021-10-26
13
-194
/
+294
*
python: isort all imports
Bryan Newbold
2021-10-26
12
-20
/
+30
*
local-file version of gen_file_metadata
Bryan Newbold
2021-10-15
1
-1
/
+13
*
wrap up previous renaming work
Bryan Newbold
2021-10-15
1
-1
/
+1
*
refactor and expand wall/block/cookie URL patterns
Bryan Newbold
2021-09-03
1
-0
/
+14
*
move some PDF URL extraction into declarative format
Bryan Newbold
2020-11-08
2
-9
/
+3
*
xml: re-encode XML docs into UTF-8 for persisting
Bryan Newbold
2020-11-03
2
-0
/
+354
*
html: some refactoring
Bryan Newbold
2020-11-03
1
-1
/
+1
*
html: syntax fixes; resolve relative URLs; extract more XML fulltext URLs
Bryan Newbold
2020-10-30
1
-7
/
+8
*
html: work around firstmonday DOCTYPE issue
Bryan Newbold
2020-10-30
2
-0
/
+455
*
tests: fix conditional on poppler version check
Bryan Newbold
2020-10-30
1
-1
/
+1
*
improve test running and config
Bryan Newbold
2020-10-29
1
-0
/
+2
*
html: more metadata tests
Bryan Newbold
2020-10-29
2
-0
/
+2453
*
HTML metadata: fix type warnings
Bryan Newbold
2020-10-27
1
-1
/
+2
*
start HTML metadata extraction code
Bryan Newbold
2020-10-27
5
-0
/
+2628
*
check for simple URL patterns that are usually paywalls or loginwalls
Bryan Newbold
2020-08-11
1
-0
/
+18
*
fix tests passing str as HTML
Bryan Newbold
2020-08-08
1
-3
/
+3
*
another bad/non PDF test; catch correct error
Bryan Newbold
2020-06-25
1
-0
/
+5
*
pdfextract support in ingest worker
Bryan Newbold
2020-06-25
1
-0
/
+7
*
fix tests for page0_height/width
Bryan Newbold
2020-06-25
1
-2
/
+2
*
lint fixes
Bryan Newbold
2020-06-17
1
-1
/
+1
*
rename pdf tools to pdfextract
Bryan Newbold
2020-06-17
1
-0
/
+0
*
partial test coverage of pdf extract worker
Bryan Newbold
2020-06-17
1
-0
/
+61
*
remove unused common.py
Bryan Newbold
2020-06-17
1
-40
/
+0
*
url cleaning (canonicalization) for ingest base_url
Bryan Newbold
2020-03-10
1
-1
/
+7
*
ingest: add URL blocklist feature
Bryan Newbold
2020-01-17
1
-0
/
+17
*
clarify ingest result schema and semantics
Bryan Newbold
2020-01-15
2
-3
/
+21
*
add postgrest checks to test mocks
Bryan Newbold
2020-01-14
1
-1
/
+9
*
tests: don't use localhost as a responses mock host
Bryan Newbold
2020-01-14
2
-6
/
+6
*
SPNv2 doesn't support FTP; add a live test for non-revist FTP
Bryan Newbold
2020-01-14
1
-0
/
+16
*
more ftp status 226 support
Bryan Newbold
2020-01-14
3
-3
/
+9
*
add live tests for ftp, revisits
Bryan Newbold
2020-01-14
1
-1
/
+36
*
more live tests (for regressions)
Bryan Newbold
2020-01-10
1
-0
/
+41
*
refactor ingest to a loop, allowing multiple hops
Bryan Newbold
2020-01-09
1
-2
/
+9
*
add (skipped) live tests for wayback services
Bryan Newbold
2020-01-09
1
-0
/
+73
*
add ingest test file
Bryan Newbold
2020-01-09
1
-0
/
+120
*
lots of progress on wayback refactoring
Bryan Newbold
2020-01-09
1
-1
/
+7
*
location comes as a string, not list
Bryan Newbold
2020-01-09
1
-4
/
+4
*
wrap up basic (locally testable) ingest refactor
Bryan Newbold
2020-01-09
1
-4
/
+48
[next]