diff options
author | Bryan Newbold <bnewbold@archive.org> | 2019-12-01 15:42:05 -0800 |
---|---|---|
committer | Bryan Newbold <bnewbold@archive.org> | 2019-12-01 15:42:07 -0800 |
commit | aa49ab86e4a86067ba2346d8bccf389be940b8e2 (patch) | |
tree | 19272451a626fdf7cfc8cdbfe581cf7fe3ede05d /pig/tests/files/tarballs.cdx | |
parent | e28125db2735b53e28ab5148379cb8b804c184c6 (diff) | |
download | sandcrawler-aa49ab86e4a86067ba2346d8bccf389be940b8e2.tar.gz sandcrawler-aa49ab86e4a86067ba2346d8bccf389be940b8e2.zip |
filter out very large GROBID XML bodies
This is to prevent Kafka MSG_SIZE_TOO_LARGE publish errors. We should
probably bump this in the future.
Open problems: hand-coding this size number isn't good, need to update
in two places. Shouldn't filter out for non-Kafka sinks. Might still
exist a corner-case where JSON encoded XML is larger than XML character
string, due to encoding (eg, for unicode characters).
Diffstat (limited to 'pig/tests/files/tarballs.cdx')
0 files changed, 0 insertions, 0 deletions