update grobid refs proposal

1 files changed, 72 insertions, 10 deletions

diff --git a/proposals/2021-10-28_grobid_refs.md b/proposals/2021-10-28_grobid_refs.md
index 3a17af3..ff835d4 100644
--- a/proposals/2021-10-28_grobid_refs.md
+++ b/proposals/2021-10-28_grobid_refs.md
@@ -13,18 +13,61 @@ in Crossref, and include them in refcat.
 
 ## Schema and Semantics
 
-Follows that of `grobid_tei_xml` version 0.1.
+The output JSON/dict schema for parsed references follows that of
+`grobid_tei_xml` version 0.1.x, for the `GrobidBiblio` field. The
+`unstructured` field that was parsed is included in the output, though it may
+not be byte-for-byte exact (see below). One notable change from the past (eg,
+older GROBID-parsed references) is that author `name` is now `full_name`. New
+fields include `editors` (same schema as `authors`), `book_title`, and
+`series_title`.
+
+The overall output schema matches that of the `grobid_refs` SQL table:
+
+    source: string, lower-case. eg 'crossref'
+    source_id: string, eg '10.1145/3366650.3366668'
+    source_ts: optional timestamp (full ISO datetime with timezone (eg, `Z`
+               suffix), which identifies version of upstream metadata
+    refs_json: JSONB, list of `GrobidBiblio` JSON objects
+
+References are re-processed on a per-article (or per-release) basis. All the
+references for an article are handled as a batch and output as a batch. If
+there are no upstream references, row with `ref_json` as empty list may be
+returned.
+
+Not all upstream references get re-parsed, even if an 'unstructured' field is
+available. If 'unstructured' is not available, no row is ever output. For
+example, if a reference includes `unstructured` (raw citation string), but also
+has structured metadata for authors, title, year, and journal name, we might
+not re-parse the `unstructured` string. Whether to re-parse is evaulated on a
+per-reference basis. This behavior may change over time.
+
+`unstructured` strings may be pre-processed before being submitted to GROBID.
+This is because many sources have systemic encoding issues. GROBID itself may
+also do some modification of the input citation string before returning it in
+the output. This means the `unstructured` string is not a reliable way to map
+between specific upstream references and parsed references. Instead, the `id`
+field (str) of `GrobidBiblio` gets set to any upstream "key" or "index"
+identifier used to track individual references. If there is only a numeric
+index, the `id` is that number as a string.
+
+The `key` or `id` may need to be woven back in to the ref objects manually,
+because GROBID `processCitationList` takes just a list of raw strings, with no
+attached reference-level key or id.
 
-Not all references are necessarily included for GROBID processing. They should
-identified and mapped using the entire unstructured string.
 
-When present, `key` or `id` is woven back in to the ref objects (GROBID
-`processCitationList` doesn't ever see the keys). `index`, returned by
-`grobid_tei_xml`, may not be accurate (because not all references were passed),
-and may be removed (TBD).
+## New SQL Table and View
 
+We may want to do re-parsing of references from sources other than `crossref`,
+so there is a generic `grobid_refs` table. But it is also common to fetch both
+the crossref metadata and any re-parsed references together, so as a convience
+there is a PostgreSQL view (virtual table) that includes both a crossref
+metadata record and parsed citations, if available. If downstream code cares a
+lot about having the refs and record be in sync, the `source_ts` field on
+`grobid_refs` can be matched againt the `indexed` column of `crossref` (or the
+`.indexed.date-time` JSON field in the record itself).
 
-## New SQL Table and View
+Remember that DOIs should always be lower-cased before querying, inserting,
+comparing, etc.
 
     CREATE TABLE IF NOT EXISTS grobid_refs (
         source              TEXT NOT NULL CHECK (octet_length(source) >= 1),
@@ -35,8 +78,7 @@ and may be removed (TBD).
         PRIMARY KEY(source, source_id)
     );
 
-    CREATE OR REPLACE VIEW crossref_with_refs
-        doi, indexed, record, source_ts, refs_json AS
+    CREATE OR REPLACE VIEW crossref_with_refs (doi, indexed, record, source_ts, refs_json) AS
         SELECT
             crossref.doi as doi,
             crossref.indexed as indexed,
@@ -61,3 +103,23 @@ locally on the machine with sandcrawler-db.
 Another tool will support taking large chunks of Crossref JSON (as lines),
 filter them, process with GROBID, and print JSON to stdout, in the
 `grobid_refs` JSON schema.
+
+
+## Task Examples
+
+Command to process crossref records with refs tool:
+
+    cat crossref_sample.json \
+        | parallel -j5 --linebuffer --round-robin --pipe ./grobid_tool.py parse-crossref-refs - \
+        | pv -l \
+        > crossref_sample.parsed.json
+
+    # => 10.0k 0:00:27 [ 368 /s]
+
+Load directly in to postgres (after tables have been created):
+
+    cat crossref_sample.parsed.json \
+        | jq -rc '[.source, .source_id, .source_ts, (.refs_json | tostring)] | @tsv' \
+        | psql sandcrawler -c "COPY grobid_refs (source, source_id, source_ts, refs_json) FROM STDIN (DELIMITER E'\t');"
+
+    # => COPY 9999