index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
python
/
sandcrawler
Commit message (
Expand
)
Author
Age
Files
Lines
*
small fixes from local testing for XML ingest
Bryan Newbold
2020-11-03
2
-4
/
+9
*
xml: re-encode XML docs into UTF-8 for persisting
Bryan Newbold
2020-11-03
2
-1
/
+10
*
persist: XML and HTML persist workers
Bryan Newbold
2020-11-03
1
-3
/
+74
*
ingest: handle publishing XML docs to kafka
Bryan Newbold
2020-11-03
1
-3
/
+21
*
refactor 'minio' to 'seaweedfs'; and BLOB env vars
Bryan Newbold
2020-11-03
2
-4
/
+6
*
gen_file_metadata: detect JATS XML and use application/jats+xml
Bryan Newbold
2020-11-03
1
-0
/
+4
*
basic support for XML ingest in worker
Bryan Newbold
2020-11-03
1
-23
/
+40
*
html: improve XML fulltext extraction for scielo
Bryan Newbold
2020-11-03
1
-4
/
+17
*
tweak html_meta SQL schema
Bryan Newbold
2020-11-03
1
-12
/
+19
*
html: some refactoring
Bryan Newbold
2020-11-03
2
-23
/
+56
*
move transfer encoding helper to sandcrawler/ia.py
Bryan Newbold
2020-11-03
2
-23
/
+42
*
ingest: cleanups, typing, start generalizing to xml and html
Bryan Newbold
2020-11-03
1
-122
/
+118
*
ingest: tweak debug printing alignment
Bryan Newbold
2020-11-03
2
-11
/
+10
*
ingest: add more IA domains
Bryan Newbold
2020-11-03
1
-0
/
+2
*
cdx: fix 'closest' support
Bryan Newbold
2020-11-03
1
-3
/
+2
*
html: start on SQL table
Bryan Newbold
2020-11-03
1
-0
/
+44
*
html: syntax fixes; resolve relative URLs; extract more XML fulltext URLs
Bryan Newbold
2020-10-30
2
-8
/
+15
*
html: work around firstmonday DOCTYPE issue
Bryan Newbold
2020-10-30
1
-0
/
+3
*
cdx datetime parsing improvements
Bryan Newbold
2020-10-30
1
-0
/
+11
*
cdx: add support for 'closest' time parameter
Bryan Newbold
2020-10-30
1
-3
/
+9
*
html: more ingest improvements
Bryan Newbold
2020-10-30
2
-18
/
+120
*
html ingest: improve data flow
Bryan Newbold
2020-10-29
1
-18
/
+41
*
better default CLI output (show usage)
Bryan Newbold
2020-10-29
1
-1
/
+1
*
misc: type annotations, fix parse_cdx_datetime
Bryan Newbold
2020-10-29
1
-14
/
+18
*
html: initial ingest implementation
Bryan Newbold
2020-10-29
1
-0
/
+193
*
html: more biblio selectors; resource extraction
Bryan Newbold
2020-10-29
1
-0
/
+102
*
HTML meta: more from online hunting/research
Bryan Newbold
2020-10-27
1
-3
/
+54
*
HTML metadata: fix type warnings
Bryan Newbold
2020-10-27
1
-1
/
+3
*
start HTML metadata extraction code
Bryan Newbold
2020-10-27
1
-0
/
+230
*
ingest: skip JSTOR DOI prefixes
Bryan Newbold
2020-10-23
1
-0
/
+3
*
Revert "reimplement worker timeout with multiprocessing"
Bryan Newbold
2020-10-22
1
-17
/
+23
*
ingest: decrease CDX timeout retries again
Bryan Newbold
2020-10-22
1
-1
/
+1
*
reimplement worker timeout with multiprocessing
Bryan Newbold
2020-10-22
1
-23
/
+17
*
ingest: fix WaybackContentError typo
Bryan Newbold
2020-10-21
1
-1
/
+1
*
ingest: add a check for blocked-cookie before trying PDF url extraction
Bryan Newbold
2020-10-21
1
-0
/
+11
*
differential wayback-error from wayback-content-error
Bryan Newbold
2020-10-21
7
-18
/
+22
*
ingest: add a cdx-error slowdown delay
Bryan Newbold
2020-10-19
1
-0
/
+3
*
SPN CDX delay now seems reasonable; increase to 40sec to catch most
Bryan Newbold
2020-10-19
1
-1
/
+1
*
ingest: fix old_failure datetime
Bryan Newbold
2020-10-19
1
-1
/
+1
*
CDX: when retrying, do so every 3 seconds up to limit
Bryan Newbold
2020-10-19
1
-5
/
+9
*
ingest: try SPNv2 for no-capture and old failures
Bryan Newbold
2020-10-19
1
-1
/
+5
*
SPN: more verbose status logging
Bryan Newbold
2020-10-19
1
-0
/
+4
*
ingest: disable soft404 and non-hit SPNv2 retries
Bryan Newbold
2020-10-19
1
-4
/
+5
*
CDX: revert post-SPN CDX lookup retry to 10 seconds
Bryan Newbold
2020-10-19
1
-1
/
+1
*
ingest: catch wayback-fail-after-SPN as separate status
Bryan Newbold
2020-10-19
1
-4
/
+17
*
SPN: better log line when starting a request
Bryan Newbold
2020-10-19
1
-0
/
+1
*
SPN: look for non-200 CDX responses
Bryan Newbold
2020-10-19
1
-1
/
+1
*
SPN: better check for partial URLs returned
Bryan Newbold
2020-10-19
1
-2
/
+2
*
CDX fetch: more permissive fuzzy/normalization check
Bryan Newbold
2020-10-19
1
-3
/
+9
*
ingest: experimentally reduce CDX API retry delay
Bryan Newbold
2020-10-17
1
-1
/
+1
[next]