aboutsummaryrefslogtreecommitdiffstats
path: root/notes/url_pattern_heuristic_backfill.txt
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@archive.org>2020-03-19 16:10:40 -0700
committerBryan Newbold <bnewbold@archive.org>2020-03-19 16:10:42 -0700
commit88f337f2cc40824ed3eaf32b1fec17c3b053bfdf (patch)
treeae7ae1a02906adf663098dc4e7762279d5ac2ac8 /notes/url_pattern_heuristic_backfill.txt
parente21fac21cc5a4267357a499f75f048ee5fd38ddb (diff)
downloadsandcrawler-88f337f2cc40824ed3eaf32b1fec17c3b053bfdf.tar.gz
sandcrawler-88f337f2cc40824ed3eaf32b1fec17c3b053bfdf.zip
persist grobid: add option to skip S3 upload
Motivation for this is that current S3 target (minio) is overloaded, with too many files on a single partition (80 million+). Going to look in to seaweedfs and other options, but for now stopping minio persist. Data is all stored in kafka anyways.
Diffstat (limited to 'notes/url_pattern_heuristic_backfill.txt')
0 files changed, 0 insertions, 0 deletions