index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
*
html: most small platform tweaks
Bryan Newbold
2020-11-08
1
-5
/
+4
*
move fuzzy URL match method to misc
Bryan Newbold
2020-11-08
3
-19
/
+20
*
move some PDF URL extraction into declarative format
Bryan Newbold
2020-11-08
5
-143
/
+177
*
basic DOAJ ingest request conversion script
Bryan Newbold
2020-11-08
1
-0
/
+139
*
ingest: default to html_biblio for PDF URL extraction
Bryan Newbold
2020-11-08
1
-24
/
+17
*
ingest: shorted scope+platform keys; use html_biblio extraction for PDFs
Bryan Newbold
2020-11-08
1
-15
/
+35
*
html: more robust ingest; better platform and scope detection
Bryan Newbold
2020-11-08
1
-32
/
+96
*
html: more extraction patterns; bugfix; skip more crossmark
Bryan Newbold
2020-11-08
1
-1
/
+24
*
ingest html: return better status based on sniffed scope
Bryan Newbold
2020-11-08
1
-9
/
+31
*
ingest tool: more ingest control args
Bryan Newbold
2020-11-08
1
-1
/
+10
*
spn2-internal-server-error is a problem with remote server, not SPN2
Bryan Newbold
2020-11-08
1
-0
/
+2
*
ingest: better non-full URL fixup
Bryan Newbold
2020-11-08
1
-4
/
+3
*
html: small ingest improvements
Bryan Newbold
2020-11-08
2
-0
/
+19
*
html: start improving scope detection
Bryan Newbold
2020-11-08
2
-5
/
+49
*
ingest: retain html_biblio through hops; all ingest types
Bryan Newbold
2020-11-08
1
-1
/
+13
*
ingest tool: flag for HTML quick mode (CDX-only)
Bryan Newbold
2020-11-08
2
-1
/
+6
*
html: try to detect and mark XHTML (vs. HTML or XML)
Bryan Newbold
2020-11-08
2
-4
/
+6
*
gen_file_metadata: allow empty/null bodies (if flag set)
Bryan Newbold
2020-11-08
2
-3
/
+5
*
html: missing fetch is wayback-content-error, not wayback-error
Bryan Newbold
2020-11-08
1
-2
/
+2
*
direct some more warnings to sys.stderr, not stdout
Bryan Newbold
2020-11-08
1
-2
/
+2
*
html: handle no-capture for sub-resources
Bryan Newbold
2020-11-08
3
-9
/
+13
*
ingest tool: consistency about ingest-type arg
Bryan Newbold
2020-11-08
1
-2
/
+2
*
ingest: fix null-body case
Bryan Newbold
2020-11-08
2
-0
/
+6
*
remove unused pytype tool
Bryan Newbold
2020-11-06
3
-76
/
+25
*
gitlab CI: upgrade pip (pip3) in environment
Bryan Newbold
2020-11-06
1
-2
/
+3
*
many bad PDF sha1 from prod
Bryan Newbold
2020-11-06
1
-0
/
+36
*
Merge branch 'bnewbold-html-ingest'
Bryan Newbold
2020-11-06
40
-549
/
+8227
|
\
|
*
html: update proposal (docs)
Bryan Newbold
2020-11-06
1
-19
/
+49
|
*
html: catch and report exceptions at process_hit() stage
Bryan Newbold
2020-11-06
1
-4
/
+27
|
*
html: pdf and html extract similar to XML
Bryan Newbold
2020-11-06
2
-22
/
+55
|
*
html: refactors/tweaks from testing
Bryan Newbold
2020-11-06
3
-17
/
+23
|
*
ia: use newer gwb (petabox) loading class
Bryan Newbold
2020-11-04
1
-5
/
+8
|
*
pipenv: fix lock file; add zstandard; update wayback+gwb deps
Bryan Newbold
2020-11-04
2
-27
/
+1069
|
*
persist: fix worker API/typing hacks (raw_key, key, key_str)
Bryan Newbold
2020-11-04
1
-9
/
+9
|
*
html: actually publish HTML TEI-XML to body; fix dataflow though ingest a bit
Bryan Newbold
2020-11-04
2
-5
/
+31
|
*
kafka: new XML+HTML topics
Bryan Newbold
2020-11-04
1
-0
/
+15
|
*
initial implementation of HTML ingest in existing worker
Bryan Newbold
2020-11-04
4
-15
/
+94
|
*
small fixes from local testing for XML ingest
Bryan Newbold
2020-11-03
3
-6
/
+9
|
*
xml: re-encode XML docs into UTF-8 for persisting
Bryan Newbold
2020-11-03
5
-2
/
+382
|
*
XML ingest proposal
Bryan Newbold
2020-11-03
1
-0
/
+64
|
*
persist: XML and HTML persist workers
Bryan Newbold
2020-11-03
2
-3
/
+121
|
*
ingest: handle publishing XML docs to kafka
Bryan Newbold
2020-11-03
2
-3
/
+27
|
*
refactor 'minio' to 'seaweedfs'; and BLOB env vars
Bryan Newbold
2020-11-03
5
-25
/
+27
|
*
gen_file_metadata: detect JATS XML and use application/jats+xml
Bryan Newbold
2020-11-03
1
-0
/
+4
|
*
basic support for XML ingest in worker
Bryan Newbold
2020-11-03
1
-23
/
+40
|
*
html: improve XML fulltext extraction for scielo
Bryan Newbold
2020-11-03
1
-4
/
+17
|
*
tweak html_meta SQL schema
Bryan Newbold
2020-11-03
2
-14
/
+21
|
*
html: some refactoring
Bryan Newbold
2020-11-03
3
-24
/
+57
|
*
move transfer encoding helper to sandcrawler/ia.py
Bryan Newbold
2020-11-03
2
-23
/
+42
|
*
ingest: cleanups, typing, start generalizing to xml and html
Bryan Newbold
2020-11-03
1
-122
/
+118
[next]