index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
sandcrawler
/
misc.py
Commit message (
Expand
)
Author
Age
Files
Lines
*
make fmt (black 21.9b0)
Bryan Newbold
2021-10-27
1
-51
/
+64
*
lint collection membership (last lint for now)
Bryan Newbold
2021-10-26
1
-2
/
+2
*
more progress on type annotations and linting
Bryan Newbold
2021-10-26
1
-8
/
+8
*
start handling trivial lint cleanups: unused imports, 'is None', etc
Bryan Newbold
2021-10-26
1
-4
/
+4
*
make fmt
Bryan Newbold
2021-10-26
1
-19
/
+45
*
python: isort all imports
Bryan Newbold
2021-10-26
1
-5
/
+5
*
improve fileset ingest integration with file ingest
Bryan Newbold
2021-10-15
1
-0
/
+15
*
local-file version of gen_file_metadata
Bryan Newbold
2021-10-15
1
-1
/
+42
*
move fuzzy URL match method to misc
Bryan Newbold
2020-11-08
1
-0
/
+17
*
html: try to detect and mark XHTML (vs. HTML or XML)
Bryan Newbold
2020-11-08
1
-2
/
+4
*
gen_file_metadata: allow empty/null bodies (if flag set)
Bryan Newbold
2020-11-08
1
-2
/
+4
*
gen_file_metadata: detect JATS XML and use application/jats+xml
Bryan Newbold
2020-11-03
1
-0
/
+4
*
cdx datetime parsing improvements
Bryan Newbold
2020-10-30
1
-0
/
+11
*
misc: type annotations, fix parse_cdx_datetime
Bryan Newbold
2020-10-29
1
-14
/
+18
*
ingest: clean_url() in more places
Bryan Newbold
2020-03-23
1
-0
/
+1
*
url cleaning (canonicalization) for ingest base_url
Bryan Newbold
2020-03-10
1
-0
/
+7
*
more mime normalization
Bryan Newbold
2020-02-27
1
-1
/
+18
*
much progress on file ingest path
Bryan Newbold
2019-10-22
1
-0
/
+24
*
lots of grobid tool implementation (still WIP)
Bryan Newbold
2019-09-26
1
-5
/
+11
*
re-write parse_cdx_line for sandcrawler lib
Bryan Newbold
2019-09-25
1
-0
/
+84
*
start refactoring sandcrawler python common code
Bryan Newbold
2019-09-23
1
-0
/
+43