grobid_tei_xml: Python parser and transforms for GROBID-flavor TEI-XML

This is a simple python library for parsing the TEI-XML structured documents returned by GROBID, a machine learning tool for extracting text and bibliographic metadata from research article PDFs.

TEI-XML is a standard format, and there exist other libraries to parse entire documents and work with annotated text. This library is focused specifically on extracting "header" metadata from document (eg, title, authors, journal name, volume, issue), content in flattened text form (full abstract and body text as single strings, for things like search indexing), and structured citation metadata.

Quickstart

grobid_tei_xml works with Python 3, using only the standard library. It does not talk to the GROBID HTTP API or read files off disk on it's own, but see examples below. The library is packaged on pypi.org.

Install using pip, usually within a virtualenv:

pip install grobid_tei_xml

The main entry points are the functions process_document_xml(xml_text) and process_citation_xml(xml_text) (or process_citation_list_xml(xml_text) for multiple citations), which return python dataclass objects. The helper method .to_dict() can be useful for, eg, serializing these objects to JSON.

Usage Examples

Read an XML file from disk, parse it, and print to stdout as JSON:

import json
import grobid_tei_xml

xml_path = "./tests/files/small.xml"

with open(xml_path, 'r') as xml_file:
    doc = grobid_tei_xml.parse_document_xml(xml_file.read())

print(json.dumps(doc.to_dict(), indent=2))

Use requests to download a PDF from the web, submit to GROBID (via HTTP API), parse the TEI-XML response with grobid_tei_xml, and print some metadata fields:

import requests
import grobid_tei_xml

pdf_resp = requests.get("https://arxiv.org/pdf/1802.01168v3")
pdf_resp.raise_for_status()

grobid_resp = requests.post(
    "https://cloud.science-miner.com/grobid/api/processFulltextDocument",
    files={
        'input': pdf_resp.content,
        'consolidate_Citations': 0,
        'includeRawCitations': 1,
    },
    timeout=60.0,
)
grobid_resp.raise_for_status()

doc = grobid_tei_xml.parse_document_xml(grobid_resp.text)

print("title: " + doc.header.title)
print("authors: " + ", ".join([a.full_name for a in doc.header.authors]))
print("doi: " + str(doc.header.doi))
print("citation count: " + str(len(doc.citations)))
print("abstract: " + doc.abstract)

Use requests to submit a "raw" citation string to GROBID for extraction, parse the response with grobid_tei_xml, and print the structured output to stdout:

import requests
import grobid_tei_xml

raw_citation = "Kvenvolden K.A. and Field M.E. 1981. Thermogenic hydrocarbons in unconsolidated sediment of Eel River Basin, offshore northern California. AAPG Bulletin 65:1642-1646"

grobid_resp = requests.post(
    "https://cloud.science-miner.com/grobid/api/processCitation",
    data={
        'citations': raw_citation,
        'consolidateCitations': 0,
        'includeRawCitations': 1,
    },
    timeout=10.0,
)
grobid_resp.raise_for_status()

citation = grobid_tei_xml.parse_citation_xml(grobid_resp.text)
print(citation)

See Also

grobid_client_python: Python client and CLI tool for making requests to GROBID via HTTP API. Returns TEI-XML; could be used with this library (grobid_tei_xml) for parsing into python object or, eg, JSON.

GROBID Documentation

s2orc-doc2json: Python library from AI2 which includes a similar Python library for extracting both bibliographic metadata and (structured) full text from GROBID TEI-XML. Has nice features like resolving references to bibliography entry.

delb: more flexible/powerful interface to TEI-XML documents. would be a better tool for working with structured text (body, abstract, etc)

"Parsing TEI XML documents with Python" (2019): blog post about basic parsing of GROBID TEI-XML files into Pandas DataFrames

License

This library is available under the permissive MIT License. See LICENSE.txt for a copy.