path: root/python/notes/version_4.md

# V4

Have not release v3, but many change to `skate` so we may continue with v4.

# Unstructured

```
{
  "biblio": {
    "unstructured": "J. Häger, W. Krieger, T. Rüegg, and H. Walther, J. Chem. Phys. 72, 4286 (1980).JCPSA60021-9606"
  },
  "index": 8,
  "key": "_r4",
  "ref_source": "crossref",
  "release_year": 1983,
  "release_ident": "tebzylkszzbyjggye5ssmebdcy",
  "work_ident": "aaaofyp6uzcdnbe7hvfahylyha"
}
```

We should be able to match: "J. Chem. Phys.", also maybe "72, 4286 (1980)", w/
id, title we should be able to match to:
https://fatcat.wiki/release/d2k7en7tzzdzzddwfo5xlqx4ce

If nothing else defined, and unstructured contains a URL, we may extract that.

```
{
  "biblio": {
    "unstructured": "Friedrich-Ebert-Stiftung (FES) 2008 FES in Nepal. FES http://www.fesnepal.org/about/fes_in_nepal.htm (accessed February 15, 2009)"
  },
  "index": 19,
  "key": "CIT0020",
  "ref_source": "crossref",
  "release_year": 2011,
  "release_ident": "xqgaanhpf5gotdxxxytgyxw2ty",
  "work_ident": "aaaq35j3angwzdpzcvzdil3v4y"
}
```

Also, these may say: "accessed at ..."

# URL

* url cleanup in place

# Partial Data Mapping

* how to map partial docs onto a key

# OL beyond ISBN

Example:

```
{
  "biblio": {
    "container_name": "The Debt: What America Owes to Blacks",
    "contrib_raw_names": [
      "R Robinson"
    ],
    "unstructured": "Randall Robinson, The Debt: What America Owes to Blacks. New York: Dutton Books, 2000, pp. 219–220.",
    "year": 2000
  },
  "index": 22,
  "key": "8_CR23",
  "ref_source": "crossref",
  "release_year": 2009,
  "release_ident": "2igycuiobvhxrcmmrzz6anufuq",
  "work_ident": "aaacj23jqbdxvajwj5kc6jpejq"
}
```

* https://openlibrary.org/works/OL488811W/The_debt?edition=debtwhatamerica000robi

However, there is no explicit "subtitle" fields, and in this case, the subtitle is buried in "text":

```
{
  "key": "/works/OL488811W",
  "text": [
    "/works/OL488811W",
    "The debt",
    "The Debt",
    "The Debt ",
    "what America owes to Blacks",
    "What America Owes to Blacks",
    "OL46591M",
    "OL7771042M",
    "OL7590904M",
    "OL3382710M",
    "Randall Robinson.",
    "2004556979",
    "99045728",
    "0452282101",
    "0525945245",
```

Subtitle in editions.

```
{
  "biblio": {
    "container_name": "BLACK AFRICA: The Economic and Cultural Basis for a Federated State",
    "unstructured": "For details on African Renaissance see Cheikh Anta Diop, BLACK AFRICA: The Economic and Cultural Basis for a Federated State, New Expanded Edition. Trenton, NJ: Africa World Press, 1987.",
    "year": 1987
  },
  "index": 28,
  "key": "8_CR29",
  "ref_source": "crossref",
  "release_year": 2009,
  "release_ident": "2igycuiobvhxrcmmrzz6anufuq",
  "work_ident": "aaacj23jqbdxvajwj5kc6jpejq"
}
```

## OL Loop

Some do not have an explicit "works" key, but still link to an edition.

* https://openlibrary.org/books/OL10000230M/Parliamentary_Debates_House_Of_Lords_2003-2004?edition=

> An edition of Parliamentary Debates, House Of Lords 2003-2004

Example edition:

```
{
  "publishers": [
    "Du Temps"
  ],
  "languages": [
    {
      "key": "/languages/fre"
    }
  ],
  "last_modified": {
    "type": "/type/datetime",
    "value": "2010-04-24T18:46:01.556464"
  },
  "weight": "5 ounces",
  "title": "Les Fleurs bleues de Raymond Queneau",
  "identifiers": {
    "goodreads": [
      "487215"
    ]
  },
  "isbn_13": [
    "9782842741013"
  ],
  "covers": [
    3140044
  ],
  "physical_format": "Paperback",
  "isbn_10": [
    "2842741013"
  ],
  "publish_date": "January 1, 2000",
  "key": "/books/OL12622734M",
  "authors": [
    {
      "key": "/authors/OL3964945A"
    }
  ],
  "latest_revision": 5,
  "works": [
    {
      "key": "/works/OL10000008W"
    }
  ],
  "type": {
    "key": "/type/edition"
  },
  "physical_dimensions": "8.4 x 5.7 x 0.3 inches",
  "revision": 5
}
```

Example Work:

```
{
  "title": "Les Fleurs bleues de Raymond Queneau",
  "created": {
    "type": "/type/datetime",
    "value": "2009-12-11T01:57:19.964652"
  },
  "covers": [
    3140044
  ],
  "last_modified": {
    "type": "/type/datetime",
    "value": "2010-04-28T06:54:19.472104"
  },
  "latest_revision": 3,
  "key": "/works/OL10000008W",
  "authors": [
    {
      "type": "/type/author_role",
      "author": {
        "key": "/authors/OL3964945A"
      }
    }
  ],
  "type": {
    "key": "/type/work"
  },
  "revision": 3
}
```

----

## Unmatched

If we exclude any id and title, we'll roughly have the following fields:

```
container_name|contrib_raw_names|year                                        64064559
unstructured                                                                 61711602
container_name|contrib_raw_names|volume|year                                 49701699
container_name|contrib_raw_names|unstructured|volume|year                    36401044
container_name|contrib_raw_names|unstructured|year                           26663422
contrib_raw_names|unstructured                                               16731608
container_name|contrib_raw_names|doi|unstructured|year                       14207167
container_name|contrib_raw_names|doi|year                                    13159340
```

Some examples:

```
{
  "biblio": {
    "container_name": "Intern. J. Comput. Math.",
    "contrib_raw_names": [
      "D. Levin"
    ],
    "volume": "B3",
    "year": 1973
  },
  "index": 19,
  "key": "PhysRevB.48.6913Cc15R1",
  "ref_source": "crossref",
  "release_year": 1993,
  "release_ident": "i6s6e64n55hh5oned32mdwrs2i",
  "work_ident": "aaaeuvgitzfafczctw3bseauri"
}
```

This refers to:

* https://www.tandfonline.com/doi/abs/10.1080/00207167308803075
* 1972, and not 1973, 1993
* https://fatcat.wiki/release/3cstmufhszalvpnppwxjohnnsa

It would help to go from "container name" to "issn", e.g. here: 0020-7160

* https://fatcat.wiki/release/search?q=levin+container_id%3A%22y4k3i2fvabgarkvywismzvy23a%22+year%3A1972

```
$ grep -i "Intern.*J.*Comput.*Math.*" jabbrev.json
{"name": "COMPEL-THE INTERNATIONAL JOURNAL FOR COMPUTATION AND MATHEMATICS IN ELECTRICAL AND ELECTRONIC ENGINEERING", "abbrev": "COMPEL"}
{"name": "INTERNATIONAL JOURNAL OF APPLIED MATHEMATICS AND COMPUTER SCIENCE", "abbrev": "INT J APPL MATH COMP"}
{"name": "INTERNATIONAL JOURNAL OF COMPUTER MATHEMATICS", "abbrev": "INT J COMPUT MATH"}
```

Lookup name in issn:


```
$ zstdcat tmp/data.ndj.zst | grep -i "INTERNATIONAL JOURNAL OF COMPUTER MATHEMATICS" | jq .

  "@graph": [
    {
      "@id": "http://id.loc.gov/vocabulary/countries/enk",
      "label": "England"
    },
    {
      "@id": "organization/ISSNCenter#_1",
      "@type": "http://schema.org/Organization"
    },
    {
      "@id": "resource/ISSN-L/0020-7160",
      "identifiedBy": "resource/ISSN/0020-7160#ISSN-L"
    },
    {
      "@id": "resource/ISSN/0020-7160",
      "@type": [
        "http://id.loc.gov/ontologies/bibframe/Instance",
        "http://id.loc.gov/ontologies/bibframe/Work",
        "http://schema.org/Periodical"
      ],
      "format": "vocabularies/medium#Print",
      "http://purl.org/ontology/bibo/issn": "0020-7160",
      "identifiedBy": [
        "resource/ISSN/0020-7160#ISSN-L",
        "resource/ISSN/0020-7160#ISSN",
        "resource/ISSN/0020-7160#KeyTitle"
      ],
```

We would need:

* rough abbrev name -> full name (jabbrev) -> issn (issnlister) -> container id (fatcat)

Example, title match with OL:

```
{
  "biblio": {
    "container_name": "Private schooling in less economically developed countries",
    "contrib_raw_names": [
      "Caddell M."
    ],
    "year": 2008
  },
  "index": 7,
  "key": "CIT0008",
  "ref_source": "crossref",
  "release_year": 2011,
  "release_ident": "xqgaanhpf5gotdxxxytgyxw2ty",
  "work_ident": "aaaq35j3angwzdpzcvzdil3v4y"
}
```

A matching OL edition record:

```
{
  "publishers": [
    "Symposium Books"
  ],
  "languages": [
    {
      "key": "/languages/eng"
    }
  ],
  "number_of_pages": 214,
  "subtitle": "Asian and African Perspectives (Oxford Studies in Comparative Education)",
  "weight": "12.6 ounces",
  "title": "Private Schooling in Less Economically Developed Countries",
  "isbn_10": [
    "1873927851"
  ],
  "type": {
    "key": "/type/edition"
  },
  "identifiers": {
    "goodreads": [
      "1078335"
    ]
  },
  "isbn_13": [
    "9781873927854"
  ],
  "covers": [
    3020365
  ],
  "physical_format": "Paperback",
  "key": "/books/OL12102259M",
  "publish_date": "April 1, 2007",
  "contributions": [
    "Prachi Srivastava (Editor)",
    "Geoffrey Walford (Editor)"
  ],
  "subjects": [
    "Organization & management of education",
    "ASIA",
    "Africa",
    "Reference / General"
  ],
  "physical_dimensions": "9.1 x 6.1 x 0.7 inches",
  "works": [
    {
      "key": "/works/OL24081822W"
    }
  ],
  "lccn": [
    "2007408632"
  ],
  "lc_classifications": [
    "LC57.5 .P75 2007"
  ],
```


----

# Completeness

```
{
  "biblio": {
    "container_name": "La vida y época de Prebisch",
    "year": 2010
  },
  "index": 5,
  "key": "key20191115064515_B6",
  "ref_source": "crossref",
  "release_year": 2019,
  "release_ident": "oc6nhkoah5gcnjfsjpct4ij3ea",
  "work_ident": "aaachbf2kbdnxekwdujbmnlw4a"
}
```

* https://fatcat.wiki/release/oc6nhkoah5gcnjfsjpct4ij3ea/references
* https://www.iberoamericana.se/articles/10.16993/iberoamericana.467/galley/445/download/

In the PDF, we find a DOI as well, but it seems to be not extracted. In fact;
the ref data comes from crossref.

Grobid gets the DOI:

```
<biblStruct xml:id="b5">
        <monogr>
                <title level="m" type="main">La vida y época de Prebisch. 1901-1986. Madrid: Marcial Pons</title>
                <author>
                        <persName xmlns="http://www.tei-c.org/ns/1.0"><forename type="first">E</forename><forename type="middle">J</forename><surname>Dosman</surname></persName>
                </author>
                <idno type="DOI">10.18356/40a5d411-es</idno>
                <ptr target="https://doi.org/10.18356/40a5d411-es" />
                <imprint>
                        <date type="published" when="2010" />
                </imprint>
        </monogr>
</biblStruct>
```

Other issues:

* year vs release_year

```
{
  "biblio": {
    "container_name": "The Methodology of Scientific Research Programmes",
    "year": 1980
  },
  "index": 13,
  "key": "key20191115064515_B14",
  "ref_source": "crossref",
  "release_year": 2019,
  "release_ident": "oc6nhkoah5gcnjfsjpct4ij3ea",
  "work_ident": "aaachbf2kbdnxekwdujbmnlw4a"
}
```

## Conservative Verification

* closeby, but different years, although it seems it would actually be a match

```
different       year    /works/OL13199655W      lvtfhk63kjbthacu2aam3jgudu      1000000delinquents      1000000 delinquents     1,000,000 Delinquents
different       year    /works/OL13199655W      wpp46slm6nca7b3nwdwtjbegla      1000000delinquents      1000000 delinquents     1,000,000 Delinquents
different       year    /works/OL13199655W      gvzsp7pz75d6roxrrarduprmie      1000000delinquents      1000000 delinquents     1,000,000 Delinquents
different       year    /works/OL13199655W      ilga2kj4nnaqdh4rmogbsgbgbe      1000000delinquents      1000000 delinquents     1,000,000 Delinquents
different       year    /works/OL13199655W      5ujpef3vjzhkvmse6ovey2q2zi      1000000delinquents      1000000 delinquents     1,000,000 Delinquents
```

## Journal name augmentation

In ~160M unmatched refs (release format) we could resolve 14M container names, via `skate-resolve-journal-name`.

```
$ zstdcat date-2021-05-06.tsv.zst | skate-resolve-journal-name -B -A /magna/data/jabbrev.json | cut -f 2 | pv -l | LC_ALL=C grep -cF resolved_container_name
2021/06/01 13:02:20 found 27178 abbreviation mappings
 160M 0:14:49 [ 180k/s] [                                          <=>                                                                                                                                                                        ]
14090677
```

## Discrepancy

* https://fatcat.wiki/release/cgmnjwrhlvccxnxyewd4buuhzm/references

UnmatchedRefs contains entry 11:

```
{
  "biblio": {
    "container_name": "Med J Aust",
    "contrib_raw_names": [
      "Dracup K"
    ],
    "volume": "166",
    "year": 1997
  },
  "index": 9,
  "key": "bibr11-010740830802800102",
  "ref_source": "crossref",
  "release_year": 2008,
  "release_ident": "cgmnjwrhlvccxnxyewd4buuhzm",
  "work_ident": "aabzzlohgza2pfaol7cgqlvpke"
}
```

In frontend, we only have a DOI; https://fatcat.wiki/release/lookup?doi=10.1016/s0147-9563(97)90082-0

## More OL matching

> ran open library and fatcat fuzzy matching (via container name) on all docs,
> that did not have an id-based match; found 139M link candidates, of which
> verification 11M strong or exact matches, of which around 3M had some IA
> identifier (about 200K unique; but looking at a few of them, it seems these were somewhat
> restricted items, e.g. "print-disabled"

Most referenced items were:

```
  13010 ia:discoverygrounde00glas
   9351 ia:selfefficacyexer0000band
   8341 ia:basicsofqualitat0000stra
   7562 ia:researchdesignqu00cres
   7027 ia:basicsqualitativ00stra
   6397 ia:qualitativedataa00mile
   5958 ia:briefhistoryneol00harv
   5779 ia:constructinggrou00char
   5291 ia:reassemblingsoci00lato
   4908 ia:econometricanaly0000gree_f5q0
   4762 ia:powerknowledges00fouc
   4733 ia:stressappraisalc00rich
   4673 ia:locationculture00bhab_220
   4405 ia:threeworldswelfa00espi
   4330 ia:bodiesthatmatter00butl_662
   4306 ia:gendertroublefem0000butl_d7d5
   4249 ia:contentanalysisi00krip
   4059 ia:practiceofeveryd01cert
   4013 ia:intermolecularsu00isra
   3925 ia:culturesorganiza00hofs
   3621 ia:fractalgeometryo00beno
   3564 ia:modernityatlarge00appa
   3483 ia:modelselectionmu00burn_141
   3454 ia:economicinstitut00will
   3300 ia:economictheoryof00down_0
   3289 ia:seeinglikestateh00scot_250
   3204 ia:densityfunctiona00parr
   3158 ia:experienceeducat00dewe_0
   3127 ia:infraredramanspe00naka
   3114 ia:qualitativeinqui00cres_711
   3088 ia:sciencehumanbeha00bfsk
   3050 ia:structuralequati0000byrn_g1v4
   3050 ia:numericalrecipes0000unse_j9c5
   3003 ia:conductionheatso00cars
   2955 ia:fifthdisciplineasen00seng
   2793 ia:homosacersoverei00agam_937
   2746 ia:naturalisticinqu00linc
   2737 ia:postmoderncondit00lyot_037
   2673 ia:mathematicsdiffu00cran
   2576 ia:principlespracti0000klin
   2530 ia:hydrodynamichydr00chan
   2499 ia:strategicmanagem00free
   2493 ia:viscoelasticprop00ferr
   2454 ia:wehaveneverbeenm00lato_404
   2428 ia:foucaultreader00fouc
   2386 ia:languagesymbolic0000bour_1991
   2381 ia:weaponsofweakeve0000scot
   2380 ia:greattransforma000pola
   2370 ia:crossingqualityc00amer_984
```

## Glitch in GS?

The fractal geometry of nature is "cited by 47428"
(https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22fractal+geometry+of+nature%22&btnG=);
on page one of references, there is "Gaussian processes in machine learning"
(2003), via:
https://www.researchgate.net/profile/Olivier_Bousquet/publication/238718428_Advanced_Lectures_on_Machine_Learning_ML_Summer_Schools_2003_Canberra_Australia_February_2-14_2003_Tubingen_Germany_August_4-16_2003_Revised_Lectures/links/02e7e52c5870850311000000/Advanced-Lectures-on-Machine-Learning-ML-Summer-Schools-2003-Canberra-Australia-February-2-14-2003-Tuebingen-Germany-August-4-16-2003-Revised-Lectures.pdf#page=70
- the paper itself does not contain a reference -- in the whole document.


## OL fuzzy different

Reasons, why pairs were marked as *different*:

```
$ zstdcat -T UnmatchedOpenLibraryMatchTable/date-2021-05-06.tsv.zst  | grep ^different | cut -f2 | LC_ALL=C sort -S50% | uniq -c | sort -nr
47324670 year
46016349 contribintersectionempty
 582618 pagecount
    460 titlefilename
     25 numdiff
```

The `year` may refer to different editions:

* https://fatcat.wiki/release/kngofkvoo5cinj4wqerrey4tpi/references
* https://openlibrary.org/works/OL16286792W/One_hundred_and_seventeen_days?edition=onehundredsevent00firs

> 117 Days: An Account of Confinement and Interrogation under the South African
> 90-Day Detention Law.2006 | vs This edition was published in 1965 by Penguin
> Books

## Data mismatch

* FE: https://fatcat.wiki/release/niivpohpabhajdsf35x7hr4efm/references, [8]: 2011

refs (2017 only)

```
{
  "container_name": "19 & 20: Notes for a New Social Protagonism",
  "container": {
    "container_type": "",
    "ident": "",
    "issnl": "",
    "name": "",
    "publisher": "",
    "revision": "",
    "state": "",
    "wikidata_qid": ""
  },
  "contribs": [
    {
      "raw_name": "Colective Situaciones"
    }
  ],
  "ext_ids": {},
  "ident": "niivpohpabhajdsf35x7hr4efm",
  "release_year": "2017",
  "work_id": "7eghl5lcivfmha6d4uavrrkpce",
  "extra": {
    "crossref": {},
    "datacite": {},
    "skate": {
      "status": "ref",
      "ref": {
        "index": 7,
        "key": "\nkey\n\t\t\t\t20171225032503_CIT0007"
      },
      "rg": {},
      "resolved_container_name": ""
    },
    "ol": {}
  }
}
{
  "container_name": "A Dictionary of Marxist Thought (2nd ed.)",
  "container": {
    "container_type": "",
    "ident": "",
    "issnl": "",
    "name": "",
    "publisher": "",
    "revision": "",
    "state": "",
    "wikidata_qid": ""
  },
  "ext_ids": {},
  "ident": "niivpohpabhajdsf35x7hr4efm",
  "release_year": "2017",
  "title": "Price of production and the transformation problem",
  "work_id": "7eghl5lcivfmha6d4uavrrkpce",
  "extra": {
    "crossref": {},
    "datacite": {},
    "skate": {
      "status": "ref",
      "ref": {
        "index": 12,
        "key": "\nkey\n\t\t\t\t20171225032503_CIT0012"
      },
      "rg": {},
      "resolved_container_name": ""
    },
    "ol": {}
  }
}
{
  "container_name": "A Grammar of the Multitude: For an Analysis of Contemporary Forms of Life",
  "container": {
    "container_type": "",
    "ident": "",
    "issnl": "",
    "name": "",
    "publisher": "",
    "revision": "",
    "state": "",
    "wikidata_qid": ""
  },
  "ext_ids": {},
  "ident": "niivpohpabhajdsf35x7hr4efm",
  "release_year": "2017",
  "work_id": "7eghl5lcivfmha6d4uavrrkpce",
  "extra": {
    "crossref": {},
    "datacite": {},
    "skate": {
      "status": "ref",
      "ref": {
        "index": 45,
        "key": "\nkey\n\t\t\t\t20171225032503_CIT0044"
      },
      "rg": {},
      "resolved_container_name": ""
    },
    "ol": {}
  }
}
{
  "container_name": "An Introduction to the Three Volumes of Karl Marx's Capital",
  "container": {
    "container_type": "",
    "ident": "",
    "issnl": "",
    "name": "",
    "publisher": "",
    "revision": "",
    "state": "",
    "wikidata_qid": ""
  },
  "ext_ids": {},
  "ident": "niivpohpabhajdsf35x7hr4efm",
  "release_year": "2017",
  "work_id": "7eghl5lcivfmha6d4uavrrkpce",
  "extra": {
    "crossref": {},
    "datacite": {},
    "skate": {
      "status": "ref",
      "ref": {
        "index": 21,
        "key": "\nkey\n\t\t\t\t20171225032503_CIT0020"
      },
      "rg": {},
      "resolved_container_name": ""
    },
    "ol": {}
  }
}
```

## Grobid misses (ISBN)

* PDF: https://web.archive.org/web/20031204233716/http://grace.wharton.upenn.edu:80/~sok/sokpapers/1999-0/indiana-transparency/flbc-transparency.pdf
* seems grobid does not recognize ISBN?

```
<biblStruct xml:id="b25">
        <analytic>
                <title level="a" type="main">Electronic data interchange in logistics</title>
                <author>
                        <persName xmlns="http://www.tei-c.org/ns/1.0"><forename type="first">Margaret</forename><forename type="middle">A</forename><surname>Emmelhainz</surname></persName>
                </author>
        </analytic>
        <monogr>
                <title level="m">The Logistics Handbook</title>
                <editor>James F. Robeson and William C. Copacino</editor>
                <meeting><address><addrLine>New York, NY</addrLine></address></meeting>
                <imprint>
                        <publisher>The Free Press</publisher>
                        <date type="published" when="1994" />
                        <biblScope unit="page" from="0" to="02" />
                </imprint>
        </monogr>
</biblStruct>

<biblStruct xml:id="b26">
        <monogr>
                <title level="m" type="main">WordNet: An Electronic Lexical Database</title>
                <editor>Christiane Fellbaum</editor>
                <imprint>
                        <date type="published" when="1998" />
                        <publisher>The MIT Press</publisher>
                        <biblScope unit="page" from="0" to="262" />
                        <pubPlace>Cambridge, MA</pubPlace>
                </imprint>
        </monogr>
</biblStruct>
```

## Duplicates in combined dataset

When we merge matches with refs, we find duplicates, e.g.:

```
{
  "_id": "4kg2dejsgzaf3cszs2lt5hz4by_9",
  "indexed_ts": "2021-06-15T15:30:42Z",
  "source_release_ident": "4kg2dejsgzaf3cszs2lt5hz4by",
  "source_work_ident": "2222jduvonfg3p2no5gvvf2sj4",
  "source_year": "2011",
  "ref_index": 9,
  "ref_key": "ref9",
  "target_release_ident": "itntzdjbczfmhhaynqvqcwp6wm",
  "target_work_ident": "csff4o7yjzbz3mfszl4zvfkcua",
  "match_provenance": "crossref",
  "match_status": "exact",
  "match_reason": "doi"
}
{
  "_id": "4kg2dejsgzaf3cszs2lt5hz4by_9",
  "indexed_ts": "2021-06-15T15:30:42Z",
  "source_release_ident": "4kg2dejsgzaf3cszs2lt5hz4by",
  "source_work_ident": "2222jduvonfg3p2no5gvvf2sj4",
  "source_year": "2011",
  "ref_index": 9,
  "ref_key": "b9",
  "match_status": "unmatched",
  "match_reason": "unknown",
  "target_unstructured": "Danaceau JP, Deering CE, Day JE, Smeal SJ, Johnson-Davis KL, et al. (2007) Persistence of tolerance to methamphetamine-induced monoamine deficits. Eur J Pharmacol 559: 46-54."
}
{
  "_id": "4kg2dejsgzaf3cszs2lt5hz4by_9",
  "indexed_ts": "2021-06-15T15:30:42Z",
  "source_release_ident": "4kg2dejsgzaf3cszs2lt5hz4by",
  "source_work_ident": "2222jduvonfg3p2no5gvvf2sj4",
  "source_year": "2011",
  "ref_index": 9,
  "ref_key": "ref10",
  "match_status": "unmatched",
  "match_reason": "unknown"
}
```

Here, the ref index is 9, but ref keys are different, which might come from a
different grobid run. I feel, we should not depend on a value that we have
little control over.

As a mititgation, we'll run a final deduplication step; but that won't catch
all duplicates, e.g. when the indices are different, but the reference is
actually the same.

Would need to "uniq" tool for the whole ref blob or something like that.

----

## QA: duplicates

There seem to be many self-links in the dataset:

* sample: 25668733, duplicate rows: 1913155; about 8% (although only 145030 uniq; many repetitions)

```
$ LC_ALL=C awk '$1 == $2' bref_tabs.tsv # ....
56fbxcue6rdxlmxqto7vibg2xi      56fbxcue6rdxlmxqto7vibg2xi      exact   doi     crossref
o2juqzskxzdtpbait5gxg3yf4q      o2juqzskxzdtpbait5gxg3yf4q      exact   doi     crossref
6mwdlhvbljgtdntz5qifywhsn4      6mwdlhvbljgtdntz5qifywhsn4      exact   doi     crossref
t7vluqxmgbe4pipf4nkfcayedq      t7vluqxmgbe4pipf4nkfcayedq      exact   doi     crossref
iofm6brptvczlnrys5vxw34x3i      iofm6brptvczlnrys5vxw34x3i      exact   doi     crossref
soa44abzivcnfnsx4ymxvbyg44      soa44abzivcnfnsx4ymxvbyg44      exact   doi     crossref
7fs4c3u2ofcmxie344o5e4wuxi      7fs4c3u2ofcmxie344o5e4wuxi      exact   doi     crossref
igyewr6er5epfozhk7dyfqa5tu      igyewr6er5epfozhk7dyfqa5tu      exact   doi     crossref
```

----

# Numbers

* old crossref dump; no partial matches yet
* 740,248,530 unique edges


----

# Stats

```
553414112  exact   doi
75738037   strong  jaccardauthors
66257136   exact   pmid
19646986   strong  slugtitleauthormatch
17202451   strong  tokenizedauthors
3730080    exact   arxiv
2798816    exact   titleauthormatch
482811     strong  versioneddoi
303336     strong  pmiddoipair
279212     exact   isbn
240405     exact   workid
52678      strong  customieeearxiv
43797      strong  dataciterelatedid
29027      strong  arxivversion
27150      exact   pmcid
1652       strong  figshareversion
832        strong  titleartifact
10         strong  custombsiundated
2          strong  custombsisubdoc
```

* total unique edges: 740248530
* matches by id: 623707690
* matches though title/author (fuzzy) matching: 116540840
* scholarly resources: 727853720
* linked open library titles: 12394810
* URLs extracted from corpus: 25405592
* sample ratio IA/URL from corpus (N=100000):

----

# Timing

Completed 24/25 jobs in 42h with a few failures from disk space issues.

* 2562m34.844s

----

# Stats

* BrefCombined

```
1,175,653,287 exact      doi
  540,957,187 unmatched  unknown
   62,621,153 strong     jaccardauthors
   56,950,207 exact      pmid
   11,942,348 strong     slugtitleauthormatch
   10,278,122 strong     tokenizedauthors
    3,419,958 exact      arxiv
    2,479,491 exact      titleauthormatch
      592,522 exact      isbn
      462,252 strong     versioneddoi
       91,076 strong     pmiddoipair
       77,030 strong     customieeearxiv
       57,290 strong     dataciterelatedid
       33,837 strong     arxivversion
       18,448 exact      pmcid
        1,749 exact      workid
        1,241 strong     figshareversion
          557 strong     titleartifact
           10 strong     custombsiundated
            2 strong     custombsisubdoc
```

Of the 540M unmatched, we have:

```
          Unstr CSL

404285013 true  false
136672174 false true
```

----

## Web links

A single copy in archive, and no-live web copy:

* [http://web.archive.org/web/20180509015159/http://0-hera.ugr.es.adrastea.ugr.es/tesisugr/21007196.pdf](http://web.archive.org/web/20180509015159/http://0-hera.ugr.es.adrastea.ugr.es/tesisugr/21007196.pdf)
* [http://web.archive.org/web/20130108031412/http://www.birdlife.org/datazone/userfiles/file/IBAs/AfricaCntryPDFs/Djibouti.pdf](http://web.archive.org/web/20130108031412/http://www.birdlife.org/datazone/userfiles/file/IBAs/AfricaCntryPDFs/Djibouti.pdf)

----

# Rerun 2021-07-28

Only a single thread; ssd tmp;

* 1340min, then low disk space
* another 428min for the remaining two tasks
* total:1768min, about 30h

Performance headroom: run all mappers in one go; run tasks in parallel