path: root/python/notes/version_4.md

# V4

Have not release v3, but many change to `skate` so we may continue with v4.

# Unstructured

```
{
  "biblio": {
    "unstructured": "J. Häger, W. Krieger, T. Rüegg, and H. Walther, J. Chem. Phys. 72, 4286 (1980).JCPSA60021-9606"
  },
  "index": 8,
  "key": "_r4",
  "ref_source": "crossref",
  "release_year": 1983,
  "release_ident": "tebzylkszzbyjggye5ssmebdcy",
  "work_ident": "aaaofyp6uzcdnbe7hvfahylyha"
}
```

We should be able to match: "J. Chem. Phys.", also maybe "72, 4286 (1980)", w/
id, title we should be able to match to:
https://fatcat.wiki/release/d2k7en7tzzdzzddwfo5xlqx4ce

If nothing else defined, and unstructured contains a URL, we may extract that.

```
{
  "biblio": {
    "unstructured": "Friedrich-Ebert-Stiftung (FES) 2008 FES in Nepal. FES http://www.fesnepal.org/about/fes_in_nepal.htm (accessed February 15, 2009)"
  },
  "index": 19,
  "key": "CIT0020",
  "ref_source": "crossref",
  "release_year": 2011,
  "release_ident": "xqgaanhpf5gotdxxxytgyxw2ty",
  "work_ident": "aaaq35j3angwzdpzcvzdil3v4y"
}
```

Also, these may say: "accessed at ..."

# URL

* url cleanup in place

# Partial Data Mapping

* how to map partial docs onto a key

# OL beyond ISBN

Example:

```
{
  "biblio": {
    "container_name": "The Debt: What America Owes to Blacks",
    "contrib_raw_names": [
      "R Robinson"
    ],
    "unstructured": "Randall Robinson, The Debt: What America Owes to Blacks. New York: Dutton Books, 2000, pp. 219–220.",
    "year": 2000
  },
  "index": 22,
  "key": "8_CR23",
  "ref_source": "crossref",
  "release_year": 2009,
  "release_ident": "2igycuiobvhxrcmmrzz6anufuq",
  "work_ident": "aaacj23jqbdxvajwj5kc6jpejq"
}
```

* https://openlibrary.org/works/OL488811W/The_debt?edition=debtwhatamerica000robi

However, there is no explicit "subtitle" fields, and in this case, the subtitle is buried in "text":

```
{
  "key": "/works/OL488811W",
  "text": [
    "/works/OL488811W",
    "The debt",
    "The Debt",
    "The Debt ",
    "what America owes to Blacks",
    "What America Owes to Blacks",
    "OL46591M",
    "OL7771042M",
    "OL7590904M",
    "OL3382710M",
    "Randall Robinson.",
    "2004556979",
    "99045728",
    "0452282101",
    "0525945245",
```

Subtitle in editions.

```
{
  "biblio": {
    "container_name": "BLACK AFRICA: The Economic and Cultural Basis for a Federated State",
    "unstructured": "For details on African Renaissance see Cheikh Anta Diop, BLACK AFRICA: The Economic and Cultural Basis for a Federated State, New Expanded Edition. Trenton, NJ: Africa World Press, 1987.",
    "year": 1987
  },
  "index": 28,
  "key": "8_CR29",
  "ref_source": "crossref",
  "release_year": 2009,
  "release_ident": "2igycuiobvhxrcmmrzz6anufuq",
  "work_ident": "aaacj23jqbdxvajwj5kc6jpejq"
}
```

## OL Loop

Some do not have an explicit "works" key, but still link to an edition.

* https://openlibrary.org/books/OL10000230M/Parliamentary_Debates_House_Of_Lords_2003-2004?edition=

> An edition of Parliamentary Debates, House Of Lords 2003-2004

Example edition:

```
{
  "publishers": [
    "Du Temps"
  ],
  "languages": [
    {
      "key": "/languages/fre"
    }
  ],
  "last_modified": {
    "type": "/type/datetime",
    "value": "2010-04-24T18:46:01.556464"
  },
  "weight": "5 ounces",
  "title": "Les Fleurs bleues de Raymond Queneau",
  "identifiers": {
    "goodreads": [
      "487215"
    ]
  },
  "isbn_13": [
    "9782842741013"
  ],
  "covers": [
    3140044
  ],
  "physical_format": "Paperback",
  "isbn_10": [
    "2842741013"
  ],
  "publish_date": "January 1, 2000",
  "key": "/books/OL12622734M",
  "authors": [
    {
      "key": "/authors/OL3964945A"
    }
  ],
  "latest_revision": 5,
  "works": [
    {
      "key": "/works/OL10000008W"
    }
  ],
  "type": {
    "key": "/type/edition"
  },
  "physical_dimensions": "8.4 x 5.7 x 0.3 inches",
  "revision": 5
}
```

Example Work:

```
{
  "title": "Les Fleurs bleues de Raymond Queneau",
  "created": {
    "type": "/type/datetime",
    "value": "2009-12-11T01:57:19.964652"
  },
  "covers": [
    3140044
  ],
  "last_modified": {
    "type": "/type/datetime",
    "value": "2010-04-28T06:54:19.472104"
  },
  "latest_revision": 3,
  "key": "/works/OL10000008W",
  "authors": [
    {
      "type": "/type/author_role",
      "author": {
        "key": "/authors/OL3964945A"
      }
    }
  ],
  "type": {
    "key": "/type/work"
  },
  "revision": 3
}
```

----

## Unmatched

If we exclude any id and title, we'll roughly have the following fields:

```
container_name|contrib_raw_names|year                                        64064559
unstructured                                                                 61711602
container_name|contrib_raw_names|volume|year                                 49701699
container_name|contrib_raw_names|unstructured|volume|year                    36401044
container_name|contrib_raw_names|unstructured|year                           26663422
contrib_raw_names|unstructured                                               16731608
container_name|contrib_raw_names|doi|unstructured|year                       14207167
container_name|contrib_raw_names|doi|year                                    13159340
```

Some examples:

```
{
  "biblio": {
    "container_name": "Intern. J. Comput. Math.",
    "contrib_raw_names": [
      "D. Levin"
    ],
    "volume": "B3",
    "year": 1973
  },
  "index": 19,
  "key": "PhysRevB.48.6913Cc15R1",
  "ref_source": "crossref",
  "release_year": 1993,
  "release_ident": "i6s6e64n55hh5oned32mdwrs2i",
  "work_ident": "aaaeuvgitzfafczctw3bseauri"
}
```

This refers to:

* https://www.tandfonline.com/doi/abs/10.1080/00207167308803075
* 1972, and not 1973, 1993
* https://fatcat.wiki/release/3cstmufhszalvpnppwxjohnnsa

It would help to go from "container name" to "issn", e.g. here: 0020-7160

* https://fatcat.wiki/release/search?q=levin+container_id%3A%22y4k3i2fvabgarkvywismzvy23a%22+year%3A1972

```
$ grep -i "Intern.*J.*Comput.*Math.*" jabbrev.json
{"name": "COMPEL-THE INTERNATIONAL JOURNAL FOR COMPUTATION AND MATHEMATICS IN ELECTRICAL AND ELECTRONIC ENGINEERING", "abbrev": "COMPEL"}
{"name": "INTERNATIONAL JOURNAL OF APPLIED MATHEMATICS AND COMPUTER SCIENCE", "abbrev": "INT J APPL MATH COMP"}
{"name": "INTERNATIONAL JOURNAL OF COMPUTER MATHEMATICS", "abbrev": "INT J COMPUT MATH"}
```

Lookup name in issn:


```
$ zstdcat tmp/data.ndj.zst | grep -i "INTERNATIONAL JOURNAL OF COMPUTER MATHEMATICS" | jq .

  "@graph": [
    {
      "@id": "http://id.loc.gov/vocabulary/countries/enk",
      "label": "England"
    },
    {
      "@id": "organization/ISSNCenter#_1",
      "@type": "http://schema.org/Organization"
    },
    {
      "@id": "resource/ISSN-L/0020-7160",
      "identifiedBy": "resource/ISSN/0020-7160#ISSN-L"
    },
    {
      "@id": "resource/ISSN/0020-7160",
      "@type": [
        "http://id.loc.gov/ontologies/bibframe/Instance",
        "http://id.loc.gov/ontologies/bibframe/Work",
        "http://schema.org/Periodical"
      ],
      "format": "vocabularies/medium#Print",
      "http://purl.org/ontology/bibo/issn": "0020-7160",
      "identifiedBy": [
        "resource/ISSN/0020-7160#ISSN-L",
        "resource/ISSN/0020-7160#ISSN",
        "resource/ISSN/0020-7160#KeyTitle"
      ],
```

We would need:

* rough abbrev name -> full name (jabbrev) -> issn (issnlister) -> container id (fatcat)

Example, title match with OL:

```
{
  "biblio": {
    "container_name": "Private schooling in less economically developed countries",
    "contrib_raw_names": [
      "Caddell M."
    ],
    "year": 2008
  },
  "index": 7,
  "key": "CIT0008",
  "ref_source": "crossref",
  "release_year": 2011,
  "release_ident": "xqgaanhpf5gotdxxxytgyxw2ty",
  "work_ident": "aaaq35j3angwzdpzcvzdil3v4y"
}
```

A matching OL edition record:

```
{
  "publishers": [
    "Symposium Books"
  ],
  "languages": [
    {
      "key": "/languages/eng"
    }
  ],
  "number_of_pages": 214,
  "subtitle": "Asian and African Perspectives (Oxford Studies in Comparative Education)",
  "weight": "12.6 ounces",
  "title": "Private Schooling in Less Economically Developed Countries",
  "isbn_10": [
    "1873927851"
  ],
  "type": {
    "key": "/type/edition"
  },
  "identifiers": {
    "goodreads": [
      "1078335"
    ]
  },
  "isbn_13": [
    "9781873927854"
  ],
  "covers": [
    3020365
  ],
  "physical_format": "Paperback",
  "key": "/books/OL12102259M",
  "publish_date": "April 1, 2007",
  "contributions": [
    "Prachi Srivastava (Editor)",
    "Geoffrey Walford (Editor)"
  ],
  "subjects": [
    "Organization & management of education",
    "ASIA",
    "Africa",
    "Reference / General"
  ],
  "physical_dimensions": "9.1 x 6.1 x 0.7 inches",
  "works": [
    {
      "key": "/works/OL24081822W"
    }
  ],
  "lccn": [
    "2007408632"
  ],
  "lc_classifications": [
    "LC57.5 .P75 2007"
  ],
```


----

# Completeness

```
{
  "biblio": {
    "container_name": "La vida y época de Prebisch",
    "year": 2010
  },
  "index": 5,
  "key": "key20191115064515_B6",
  "ref_source": "crossref",
  "release_year": 2019,
  "release_ident": "oc6nhkoah5gcnjfsjpct4ij3ea",
  "work_ident": "aaachbf2kbdnxekwdujbmnlw4a"
}
```

* https://fatcat.wiki/release/oc6nhkoah5gcnjfsjpct4ij3ea/references
* https://www.iberoamericana.se/articles/10.16993/iberoamericana.467/galley/445/download/

In the PDF, we find a DOI as well, but it seems to be not extracted. In fact;
the ref data comes from crossref.

Grobid gets the DOI:

```
<biblStruct xml:id="b5">
        <monogr>
                <title level="m" type="main">La vida y época de Prebisch. 1901-1986. Madrid: Marcial Pons</title>
                <author>
                        <persName xmlns="http://www.tei-c.org/ns/1.0"><forename type="first">E</forename><forename type="middle">J</forename><surname>Dosman</surname></persName>
                </author>
                <idno type="DOI">10.18356/40a5d411-es</idno>
                <ptr target="https://doi.org/10.18356/40a5d411-es" />
                <imprint>
                        <date type="published" when="2010" />
                </imprint>
        </monogr>
</biblStruct>
```

Other issues:

* year vs release_year

```
{
  "biblio": {
    "container_name": "The Methodology of Scientific Research Programmes",
    "year": 1980
  },
  "index": 13,
  "key": "key20191115064515_B14",
  "ref_source": "crossref",
  "release_year": 2019,
  "release_ident": "oc6nhkoah5gcnjfsjpct4ij3ea",
  "work_ident": "aaachbf2kbdnxekwdujbmnlw4a"
}
```

## Conservative Verification

* closeby, but different years, although it seems it would actually be a match

```
different       year    /works/OL13199655W      lvtfhk63kjbthacu2aam3jgudu      1000000delinquents      1000000 delinquents     1,000,000 Delinquents
different       year    /works/OL13199655W      wpp46slm6nca7b3nwdwtjbegla      1000000delinquents      1000000 delinquents     1,000,000 Delinquents
different       year    /works/OL13199655W      gvzsp7pz75d6roxrrarduprmie      1000000delinquents      1000000 delinquents     1,000,000 Delinquents
different       year    /works/OL13199655W      ilga2kj4nnaqdh4rmogbsgbgbe      1000000delinquents      1000000 delinquents     1,000,000 Delinquents
different       year    /works/OL13199655W      5ujpef3vjzhkvmse6ovey2q2zi      1000000delinquents      1000000 delinquents     1,000,000 Delinquents
```

## Journal name augmentation

In ~160M unmatched refs (release format) we could resolve 14M container names, via `skate-resolve-journal-name`.

```
$ zstdcat date-2021-05-06.tsv.zst | skate-resolve-journal-name -B -A /magna/data/jabbrev.json | cut -f 2 | pv -l | LC_ALL=C grep -cF resolved_container_name
2021/06/01 13:02:20 found 27178 abbreviation mappings
 160M 0:14:49 [ 180k/s] [                                          <=>                                                                                                                                                                        ]
14090677
```

## Discrepancy

* https://fatcat.wiki/release/cgmnjwrhlvccxnxyewd4buuhzm/references

UnmatchedRefs contains entry 11:

```
{
  "biblio": {
    "container_name": "Med J Aust",
    "contrib_raw_names": [
      "Dracup K"
    ],
    "volume": "166",
    "year": 1997
  },
  "index": 9,
  "key": "bibr11-010740830802800102",
  "ref_source": "crossref",
  "release_year": 2008,
  "release_ident": "cgmnjwrhlvccxnxyewd4buuhzm",
  "work_ident": "aabzzlohgza2pfaol7cgqlvpke"
}
```

In frontend, we only have a DOI; https://fatcat.wiki/release/lookup?doi=10.1016/s0147-9563(97)90082-0

## More OL matching

> ran open library and fatcat fuzzy matching (via container name) on all docs,
> that did not have an id-based match; found 139M link candidates, of which
> verification 11M strong or exact matches, of which around 3M had some IA
> identifier (about 200K unique; but looking at a few of them, it seems these were somewhat
> restricted items, e.g. "print-disabled"

Most referenced items were:

```
  13010 ia:discoverygrounde00glas
   9351 ia:selfefficacyexer0000band
   8341 ia:basicsofqualitat0000stra
   7562 ia:researchdesignqu00cres
   7027 ia:basicsqualitativ00stra
   6397 ia:qualitativedataa00mile
   5958 ia:briefhistoryneol00harv
   5779 ia:constructinggrou00char
   5291 ia:reassemblingsoci00lato
   4908 ia:econometricanaly0000gree_f5q0
   4762 ia:powerknowledges00fouc
   4733 ia:stressappraisalc00rich
   4673 ia:locationculture00bhab_220
   4405 ia:threeworldswelfa00espi
   4330 ia:bodiesthatmatter00butl_662
   4306 ia:gendertroublefem0000butl_d7d5
   4249 ia:contentanalysisi00krip
   4059 ia:practiceofeveryd01cert
   4013 ia:intermolecularsu00isra
   3925 ia:culturesorganiza00hofs
   3621 ia:fractalgeometryo00beno
   3564 ia:modernityatlarge00appa
   3483 ia:modelselectionmu00burn_141
   3454 ia:economicinstitut00will
   3300 ia:economictheoryof00down_0
   3289 ia:seeinglikestateh00scot_250
   3204 ia:densityfunctiona00parr
   3158 ia:experienceeducat00dewe_0
   3127 ia:infraredramanspe00naka
   3114 ia:qualitativeinqui00cres_711
   3088 ia:sciencehumanbeha00bfsk
   3050 ia:structuralequati0000byrn_g1v4
   3050 ia:numericalrecipes0000unse_j9c5
   3003 ia:conductionheatso00cars
   2955 ia:fifthdisciplineasen00seng
   2793 ia:homosacersoverei00agam_937
   2746 ia:naturalisticinqu00linc
   2737 ia:postmoderncondit00lyot_037
   2673 ia:mathematicsdiffu00cran
   2576 ia:principlespracti0000klin
   2530 ia:hydrodynamichydr00chan
   2499 ia:strategicmanagem00free
   2493 ia:viscoelasticprop00ferr
   2454 ia:wehaveneverbeenm00lato_404
   2428 ia:foucaultreader00fouc
   2386 ia:languagesymbolic0000bour_1991
   2381 ia:weaponsofweakeve0000scot
   2380 ia:greattransforma000pola
   2370 ia:crossingqualityc00amer_984
```

## Glitch in GS?

The fractal geometry of nature is "cited by 47428"
(https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22fractal+geometry+of+nature%22&btnG=);
on page one of references, there is "Gaussian processes in machine learning"
(2003), via:
https://www.researchgate.net/profile/Olivier_Bousquet/publication/238718428_Advanced_Lectures_on_Machine_Learning_ML_Summer_Schools_2003_Canberra_Australia_February_2-14_2003_Tubingen_Germany_August_4-16_2003_Revised_Lectures/links/02e7e52c5870850311000000/Advanced-Lectures-on-Machine-Learning-ML-Summer-Schools-2003-Canberra-Australia-February-2-14-2003-Tuebingen-Germany-August-4-16-2003-Revised-Lectures.pdf#page=70
- the paper itself does not contain a reference -- in the whole document.


## OL fuzzy different

Reasons, why pairs were marked as *different*:

```
$ zstdcat -T UnmatchedOpenLibraryMatchTable/date-2021-05-06.tsv.zst  | grep ^different | cut -f2 | LC_ALL=C sort -S50% | uniq -c | sort -nr
47324670 year
46016349 contribintersectionempty
 582618 pagecount
    460 titlefilename
     25 numdiff
```

The `year` may refer to different editions:

* https://fatcat.wiki/release/kngofkvoo5cinj4wqerrey4tpi/references
* https://openlibrary.org/works/OL16286792W/One_hundred_and_seventeen_days?edition=onehundredsevent00firs

> 117 Days: An Account of Confinement and Interrogation under the South African
> 90-Day Detention Law.2006 | vs This edition was published in 1965 by Penguin
> Books

## Data mismatch

* FE: https://fatcat.wiki/release/niivpohpabhajdsf35x7hr4efm/references, [8]: 2011

refs (2017 only)

```
{
  "container_name": "19 & 20: Notes for a New Social Protagonism",
  "container": {
    "container_type": "",
    "ident": "",
    "issnl": "",
    "name": "",
    "publisher": "",
    "revision": "",
    "state": "",
    "wikidata_qid": ""
  },
  "contribs": [
    {
      "raw_name": "Colective Situaciones"
    }
  ],
  "ext_ids": {},
  "ident": "niivpohpabhajdsf35x7hr4efm",
  "release_year": "2017",
  "work_id": "7eghl5lcivfmha6d4uavrrkpce",
  "extra": {
    "crossref": {},
    "datacite": {},
    "skate": {
      "status": "ref",
      "ref": {
        "index": 7,
        "key": "\nkey\n\t\t\t\t20171225032503_CIT0007"
      },
      "rg": {},
      "resolved_container_name": ""
    },
    "ol": {}
  }
}
{
  "container_name": "A Dictionary of Marxist Thought (2nd ed.)",
  "container": {
    "container_type": "",
    "ident": "",
    "issnl": "",
    "name": "",
    "publisher": "",
    "revision": "",
    "state": "",
    "wikidata_qid": ""
  },
  "ext_ids": {},
  "ident": "niivpohpabhajdsf35x7hr4efm",
  "release_year": "2017",
  "title": "Price of production and the transformation problem",
  "work_id": "7eghl5lcivfmha6d4uavrrkpce",
  "extra": {
    "crossref": {},
    "datacite": {},
    "skate": {
      "status": "ref",
      "ref": {
        "index": 12,
        "key": "\nkey\n\t\t\t\t20171225032503_CIT0012"
      },
      "rg": {},
      "resolved_container_name": ""
    },
    "ol": {}
  }
}
{
  "container_name": "A Grammar of the Multitude: For an Analysis of Contemporary Forms of Life",
  "container": {
    "container_type": "",
    "ident": "",
    "issnl": "",
    "name": "",
    "publisher": "",
    "revision": "",
    "state": "",
    "wikidata_qid": ""
  },
  "ext_ids": {},
  "ident": "niivpohpabhajdsf35x7hr4efm",
  "release_year": "2017",
  "work_id": "7eghl5lcivfmha6d4uavrrkpce",
  "extra": {
    "crossref": {},
    "datacite": {},
    "skate": {
      "status": "ref",
      "ref": {
        "index": 45,
        "key": "\nkey\n\t\t\t\t20171225032503_CIT0044"
      },
      "rg": {},
      "resolved_container_name": ""
    },
    "ol": {}
  }
}
{
  "container_name": "An Introduction to the Three Volumes of Karl Marx's Capital",
  "container": {
    "container_type": "",
    "ident": "",
    "issnl": "",
    "name": "",
    "publisher": "",
    "revision": "",
    "state": "",
    "wikidata_qid": ""
  },
  "ext_ids": {},
  "ident": "niivpohpabhajdsf35x7hr4efm",
  "release_year": "2017",
  "work_id": "7eghl5lcivfmha6d4uavrrkpce",
  "extra": {
    "crossref": {},
    "datacite": {},
    "skate": {
      "status": "ref",
      "ref": {
        "index": 21,
        "key": "\nkey\n\t\t\t\t20171225032503_CIT0020"
      },
      "rg": {},
      "resolved_container_name": ""
    },
    "ol": {}
  }
}
```