index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
html ingest: remove whitespace around relative URLs (eg, for d-lib)
Bryan Newbold
2021-05-21
1
-1
/
+1
*
add cdx_collection.py python script (from scratch repo)
Bryan Newbold
2021-05-04
1
-0
/
+80
*
ingest: cap max body size to ~128 MByte
Bryan Newbold
2021-04-27
1
-0
/
+6
*
persist: skip very long URLs
Bryan Newbold
2021-04-12
1
-0
/
+4
*
update default postgrest ('db') API endpoint
Bryan Newbold
2021-04-09
1
-1
/
+1
*
grobid: disable biblio-glutton consolidation
Bryan Newbold
2021-04-07
1
-3
/
+3
*
ingest: handle current degruyter PDF link pattern
Bryan Newbold
2021-03-26
1
-0
/
+8
*
add missing dotfiles (due to gitignore oops)
Bryan Newbold
2021-01-18
2
-0
/
+12
*
pipenv: lock minio S3 library to <7.0.0
Bryan Newbold
2021-01-14
2
-242
/
+196
*
more expansive python/.gitignore rules (all .gz)
Bryan Newbold
2021-01-05
1
-1
/
+1
*
doaj ingest request updates (from prod)
Bryan Newbold
2021-01-05
1
-1
/
+5
*
python makefile: don't duplicate 'lint' commands during 'test'
Bryan Newbold
2021-01-05
1
-2
/
+0
*
update to python3.8
Bryan Newbold
2021-01-05
2
-400
/
+413
*
pdf: yet more bad SHA1 (commiting lines from prod)
Bryan Newbold
2021-01-05
1
-0
/
+20
*
ia CDX: handle bad CDX rows
Bryan Newbold
2021-01-05
1
-2
/
+4
*
spn: more status codes
Bryan Newbold
2020-12-21
1
-1
/
+2
*
persist: html_meta is ON CONFLICT DO UPDATE
Bryan Newbold
2020-12-15
1
-1
/
+1
*
persist: don't expect HTML TEI-XML in result object
Bryan Newbold
2020-12-15
1
-1
/
+1
*
handle more wayback error conditions
Bryan Newbold
2020-11-20
1
-0
/
+6
*
html: more conservative parsing of element attr
Bryan Newbold
2020-11-20
1
-2
/
+4
*
xml: catch parse error
Bryan Newbold
2020-11-19
1
-3
/
+8
*
spn 'forbidden' status code
Bryan Newbold
2020-11-12
1
-1
/
+1
*
html biblio: handle 'content not in attrs' case
Bryan Newbold
2020-11-12
1
-2
/
+2
*
blacklist -> denylist
Bryan Newbold
2020-11-10
2
-9
/
+9
*
pipenv: updates (mostly for trafilatura 0.6.0)
Bryan Newbold
2020-11-10
1
-25
/
+32
*
DOAJ and HTML ingest tweaks from QA run
Bryan Newbold
2020-11-10
2
-3
/
+3
*
html: handle more traf error cases
Bryan Newbold
2020-11-08
1
-2
/
+2
*
html: more adblock
Bryan Newbold
2020-11-08
1
-1
/
+3
*
ingest: small html_bibli typo
Bryan Newbold
2020-11-08
1
-1
/
+1
*
html: most small platform tweaks
Bryan Newbold
2020-11-08
1
-5
/
+4
*
move fuzzy URL match method to misc
Bryan Newbold
2020-11-08
3
-19
/
+20
*
move some PDF URL extraction into declarative format
Bryan Newbold
2020-11-08
5
-143
/
+177
*
basic DOAJ ingest request conversion script
Bryan Newbold
2020-11-08
1
-0
/
+139
*
ingest: default to html_biblio for PDF URL extraction
Bryan Newbold
2020-11-08
1
-24
/
+17
*
ingest: shorted scope+platform keys; use html_biblio extraction for PDFs
Bryan Newbold
2020-11-08
1
-15
/
+35
*
html: more robust ingest; better platform and scope detection
Bryan Newbold
2020-11-08
1
-32
/
+96
*
html: more extraction patterns; bugfix; skip more crossmark
Bryan Newbold
2020-11-08
1
-1
/
+24
*
ingest html: return better status based on sniffed scope
Bryan Newbold
2020-11-08
1
-9
/
+31
*
ingest tool: more ingest control args
Bryan Newbold
2020-11-08
1
-1
/
+10
*
spn2-internal-server-error is a problem with remote server, not SPN2
Bryan Newbold
2020-11-08
1
-0
/
+2
*
ingest: better non-full URL fixup
Bryan Newbold
2020-11-08
1
-4
/
+3
*
html: small ingest improvements
Bryan Newbold
2020-11-08
2
-0
/
+19
*
html: start improving scope detection
Bryan Newbold
2020-11-08
2
-5
/
+49
*
ingest: retain html_biblio through hops; all ingest types
Bryan Newbold
2020-11-08
1
-1
/
+13
*
ingest tool: flag for HTML quick mode (CDX-only)
Bryan Newbold
2020-11-08
2
-1
/
+6
*
html: try to detect and mark XHTML (vs. HTML or XML)
Bryan Newbold
2020-11-08
2
-4
/
+6
*
gen_file_metadata: allow empty/null bodies (if flag set)
Bryan Newbold
2020-11-08
2
-3
/
+5
*
html: missing fetch is wayback-content-error, not wayback-error
Bryan Newbold
2020-11-08
1
-2
/
+2
*
direct some more warnings to sys.stderr, not stdout
Bryan Newbold
2020-11-08
1
-2
/
+2
*
html: handle no-capture for sub-resources
Bryan Newbold
2020-11-08
3
-9
/
+13
[prev]
[next]