improve text scrubbing

Was going to use textpipe, but dependency was too large and failed to
install with halfway modern GCC (due to CLD2 issue):

  https://github.com/GregBowyer/cld2-cffi/issues/12

So instead basically pulled out the clean_text function, which is quite
short.

1 files changed, 15 insertions, 0 deletions

diff --git a/tests/test_scrub.py b/tests/test_scrub.py
new file mode 100644
index 0000000..6c357ae
--- /dev/null
+++ b/tests/test_scrub.py
@@ -0,0 +1,15 @@
+
+import pytest
+
+from fatcat_scholar.schema import *
+
+
+def test_scrub():
+    vectors = [
+        ('“Please clean this piece… of text</b>„', '"Please clean this piece... of text"'),
+        ("<jats:p>blah", "blah"),
+    ]
+
+    for raw, fixed in vectors:
+        assert fixed == scrub_text(raw)
+