diff options
Diffstat (limited to 'proposals/20201103_xml_ingest.md')
| -rw-r--r-- | proposals/20201103_xml_ingest.md | 21 |
1 files changed, 2 insertions, 19 deletions
diff --git a/proposals/20201103_xml_ingest.md b/proposals/20201103_xml_ingest.md index 181cc11..34e00b0 100644 --- a/proposals/20201103_xml_ingest.md +++ b/proposals/20201103_xml_ingest.md @@ -1,22 +1,5 @@ -status: wip - -TODO: -x XML fulltext URL extractor (based on HTML biblio metadata, not PDF url extractor) -x differential JATS XML and scielo XML from generic XML? - application/xml+jats is what fatcat is doing for abstracts - but it should be application/jats+xml? - application/tei+xml - if startswith "<article " and "<article-meta>" => JATS -x refactor ingest worker to be more general -x have ingest code publish body to kafka topic -x write a persist worker -/ create/configure kafka topic -- test everything locally -- fatcat: ingest tool to create requests -- fatcat: entity updates worker creates XML ingest requests for specific sources -- fatcat: ingest file import worker allows XML results -- ansible: deployment of persist worker +status: deployed XML Fulltext Ingest ==================== @@ -37,7 +20,7 @@ document. For recording in fatcat, the file metadata will be passed through. For storing in Kafka and blob store (for downstream analysis), we will parse the raw XML document (as "bytes") with an XML parser, then re-output with UTF-8 encoding. The hash of the *original* XML file will be used as the key for -refering to this document. This is unintuitive, but similar to what we are +referring to this document. This is unintuitive, but similar to what we are doing with PDF and HTML documents (extracting in a useful format, but keeping the original document's hash as a key). |