diff options
| author | Bryan Newbold <bnewbold@archive.org> | 2019-12-01 15:42:05 -0800 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@archive.org> | 2019-12-01 15:42:07 -0800 |
| commit | aa49ab86e4a86067ba2346d8bccf389be940b8e2 (patch) | |
| tree | 19272451a626fdf7cfc8cdbfe581cf7fe3ede05d /python_hadoop | |
| parent | e28125db2735b53e28ab5148379cb8b804c184c6 (diff) | |
| download | sandcrawler-aa49ab86e4a86067ba2346d8bccf389be940b8e2.tar.gz sandcrawler-aa49ab86e4a86067ba2346d8bccf389be940b8e2.zip | |
filter out very large GROBID XML bodies
This is to prevent Kafka MSG_SIZE_TOO_LARGE publish errors. We should
probably bump this in the future.
Open problems: hand-coding this size number isn't good, need to update
in two places. Shouldn't filter out for non-Kafka sinks. Might still
exist a corner-case where JSON encoded XML is larger than XML character
string, due to encoding (eg, for unicode characters).
Diffstat (limited to 'python_hadoop')
0 files changed, 0 insertions, 0 deletions