aboutsummaryrefslogtreecommitdiffstats
path: root/kafka
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@archive.org>2020-11-04 17:07:46 -0800
committerBryan Newbold <bnewbold@archive.org>2020-11-04 17:07:46 -0800
commit2fdba24da0e0bf3d300cfb959514bf57a3cf6701 (patch)
tree7da06be7089d5a25e2aa85b7fe29a89a03435b7c /kafka
parenta87ca1de1d8b31c4fbf9fddead27cdc58b09565a (diff)
downloadsandcrawler-2fdba24da0e0bf3d300cfb959514bf57a3cf6701.tar.gz
sandcrawler-2fdba24da0e0bf3d300cfb959514bf57a3cf6701.zip
kafka: new XML+HTML topics
Diffstat (limited to 'kafka')
-rw-r--r--kafka/topics.md15
1 files changed, 15 insertions, 0 deletions
diff --git a/kafka/topics.md b/kafka/topics.md
index ebe7a61..fa1bd6d 100644
--- a/kafka/topics.md
+++ b/kafka/topics.md
@@ -59,6 +59,18 @@ retention (on both a size and time basis).
=> 12 partitions
=> key is sha1hex of PDF; enable key compaction; gzip compression
+ sandcrawler-ENV.xml-doc
+ => fulltext XML; mostly JATS XML
+ => schema is JSON, with 'jats_xml' field containing the XML as a string
+ => 6 partitions
+ => key is sha1hex of XML document; enable key compaction; gzip compression
+
+ sandcrawler-ENV.html-teixml
+ => extracted fulltext from HTML; mostly TEI-XML
+ => schema is JSON, with 'tei_xml' field containing the XML as a string
+ => 6 partitions
+ => key is sha1hex of source HTML document; enable key compaction; gzip compression
+
sandcrawler-ENV.pdf-thumbnail-SIZE-TYPE
=> thumbnail images (eg, png, jpg) from PDFs
=> raw bytes in message (no JSON or other wrapping). fields average 10 KByte
@@ -175,3 +187,6 @@ exists`; this seems safe, and the settings won't be over-ridden.
./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 12 --topic sandcrawler-qa.pdf-thumbnail-180px-jpg --config cleanup.policy=compact
./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 24 --topic sandcrawler-qa.unextracted
+ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 6 --topic sandcrawler-qa.xml-doc --config compression.type=gzip --config cleanup.policy=compact
+ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 6 --topic sandcrawler-qa.html-teixml --config compression.type=gzip --config cleanup.policy=compact
+