path: root/python/notes/version_0.md

# Version Zero Graph

* ~1.8B refs
* ~250M w/ titles
* ~40M URLs
* ~25M titles exact match w/ fatcat

```
127589929 fatcat_lower.tsv
251934865 refs_lower.tsv

 34931967 common_lower.tsv
 26840211 common.tsv
```

----

# Input

Heavy intermediate schema from
[fatcat-scholar](https://git.archive.org/webgroup/fatcat-scholar/), raw names
of journal titles, partial metadata.

Example line:

```json
{
  "biblio": {
    "contrib_raw_names": [
      "Maria Azevedo E Castro",
      "Gabriela"
    ],
    "title": "Imaginação em Paul Ricoeur",
    "unstructured": "AZEVEDO E CASTRO, Maria Gabriela. Imaginação em Paul Ricoeur."
  },
  "index": 0,
  "key": "b0",
  "ref_source": "grobid",
  "release_ident": "ruhcoyvxxnbc5ljsgtwhnolx3i",
  "release_year": 2018,
  "work_ident": "aaaes3j4argnjbkzdvud5r4zdi"
}
```

280M docs per file:

```
$ unpigz -c fatcat_scholar_work_fulltext.split_00.refs.json.gz | wc -l
285004451
```

Around 1,733,267,886 total reference entries. Sample completeness (10M, 1M
docs, ...):

```
{
  "biblio": 10000000,
  "biblio.arxiv_id": 23227,
  "biblio.container_name": 5760760,
  "biblio.contrib_raw_names": 7156385,
  "biblio.doi": 3584451,
  "biblio.issue": 763784,
  "biblio.pages": 3331911,
  "biblio.pmcid": 776,
  "biblio.pmid": 471338,
  "biblio.publisher": 398305,
  "biblio.title": 5164864,
  "biblio.unstructured": 6304402,
  "biblio.url": 256771,
  "biblio.volume": 5202508,
  "biblio.year": 7055442,
  "index": 10000000,
  "key": 8986307,
  "locator": 2390436,
  "ref_source": 10000000,
  "release_ident": 10000000,
  "release_year": 9629380,
  "target_release_id": 419033,
  "work_ident": 10000000
}
```

A smaller sample:

```
{
  "biblio": 1000000,
  "biblio.arxiv_id": 1804,
  "biblio.container_name": 580018,
  "biblio.contrib_raw_names": 722526,
  "biblio.doi": 355664,
  "biblio.issue": 79145,
  "biblio.pages": 337716,
  "biblio.pmcid": 241,
  "biblio.pmid": 47500,
  "biblio.publisher": 39840,
  "biblio.title": 518449,
  "biblio.unstructured": 643743,
  "biblio.url": 27535,
  "biblio.volume": 526148,
  "biblio.year": 713331,
  "index": 1000000,
  "key": 904205,
  "locator": 241850,
  "ref_source": 1000000,
  "release_ident": 1000000,
  "release_year": 966723,
  "target_release_id": 42333,
  "work_ident": 1000000
}
```


----

## DOI graph (v2020-01-22)

* 615514019 lines joined

We have "inbound" links for 41950903 records.

However, the top ranked docs seem invalid, datasets from a few prefixes, mostly
from datacite.

A few paper examples:

* https://fatcat.wiki/release/6ykebula5vgtbinbqhftun7jcy

> Effect of intensive blood-glucose control with metformin on complications in overweight patients with type 2 diabetes (UKPDS 34)

* Google Scholar: [7981](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Effect+of+intensive+blood-glucose+control+with+metformin+on+complications+in+overweight+&btnG=)
* CG: 2105

----

* https://fatcat.wiki/release/3katpfxlafezdb2rmgoesgbhkq

> The pyramid of corporate social responsibility

* GS: [12690](https://scholar.google.com/scholar?cluster=13669080523806449819&hl=en&as_sdt=0,5&sciodt=0,5)
* CG: 2100

----

> Policy Paradigms, Social Learning, and the State

* https://fatcat.wiki/release/m3kxbmcsxnfhzajp5zrdmvirlm
* GS: [8675](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Policy+Paradigms%2C+Social+Learning%2C+and+the+State&btnG=)
* CG: 2077
* OCI: [2457](https://opencitations.net/index/coci/api/v1/citations/10.2307/422246)

----

> Efficacy and safety of sorafenib in patients in the Asia-Pacific

* GS: [4780](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22Efficacy+and+safety+of+sorafenib+in+patients+in+the+Asia-Pacific%22&btnG=)
* CG: 1829
* OCI: [2326](https://opencitations.net/index/coci/api/v1/citations/10.1016/s1470-2045(08)70285-7)

----

* https://fatcat.wiki/release/qbdkrwg2dzbpfcdcsczy7ekjk4
* GS: [4172](https://scholar.google.com/scholar?cluster=10863619180312621990&hl=en&as_sdt=0,5&sciodt=0,5)
* CG: 1303
* OCI: [1776](https://opencitations.net/index/coci/api/v1/citations/10.1038/18884), 48, https://opencitations.net/browser/br/107462

----

>  Oxidative stress and some antioxidant systems in acid rain-treated bean plants

* https://fatcat.wiki/release/qrsuwvixbvgvpn4qzlum7btysi
* GS: [2613](https://scholar.google.com/scholar?cluster=13765746500938441584&hl=en&as_sdt=2005&sciodt=0,5)
* CG: 940
* OCI: [1042](https://opencitations.net/index/coci/api/v1/citations/10.1016/s0168-9452(99)00197-1)

Not sure, what this interface does, but it says "20" (?):
[https://opencitations.net/search?text=10.1016%2Fs0168-9452%2899%2900197-1&rule=doi](https://opencitations.net/search?text=10.1016%2Fs0168-9452%2899%2900197-1&rule=doi), https://opencitations.net/browser/br/808900

----

> Weak pairwise correlations imply strongly correlated network states in a neural population

* https://fatcat.wiki/release/zga73tc25nabdpt42m5ehm2kmm
* GS: [1597](https://scholar.google.com/scholar?cluster=3499093978246315979&hl=en&as_sdt=0,5&sciodt=0,5)
* CG: 706
* OCI: [794](https://opencitations.net/index/coci/api/v1/citations/10.1038/nature04701)

----

* https://fatcat.wiki/release/fm3ni5ppyzbknj2ke3rcn3qwva
* GS: [3118](https://scholar.google.com/scholar?cluster=495710774871014506&hl=en&as_sdt=0,5&sciodt=0,5)
* CG: 679
* OCI: [1092](https://opencitations.net/index/coci/api/v1/citations/10.1056/nejm199505183322008)

----

* https://fatcat.wiki/release/5i5j4qejbjey5pt56n25qmxque
* GS: [1043](https://scholar.google.de/scholar?cluster=2106266772718631212&hl=en&as_sdt=2005&sciodt=0,5)
* CG: 421
* OCI: [406](https://opencitations.net/index/coci/api/v1/citations/10.1029/2003wr002086)

----

* https://fatcat.wiki/release/agrh573u7nc4hd5pmyptoasvua
* GS: [1391](https://scholar.google.de/scholar?cluster=17826621684475874878&hl=en&as_sdt=2005&sciodt=0,5)
* CG: 388
* OCI: [444](https://opencitations.net/index/coci/api/v1/citations/10.1136/bjsm.2006.033548)

----

* https://fatcat.wiki/release/ellwakbl7rdlvgqcu7d3ss7wwa
* GS: [593](https://scholar.google.com/scholar?cluster=537065614856795966&hl=en&as_sdt=2005&sciodt=0,5)
* CG: 315
* OCI: [348](https://opencitations.net/index/coci/api/v1/citations/10.1038/nature05136)

----

* https://fatcat.wiki/release/dj6pufwjnnhfvpwj7tuyfl43mq
* GS: [1948](https://scholar.google.com/scholar?q=A%20Simple%20Approximate%20Long-Memory%20Model%20of%20Realized%20Volatility)
* CG: 275
* OCI: [336](https://opencitations.net/index/coci/api/v1/citations/10.1093/jjfinec/nbp001)

## Refs completeness

Use `ref_counter.py` and `ref_key_counter.py` to assess completeness of refs.

Example (1.8B):

```json
{
  "has_any_extid": 700483339,
  "has_arxiv_id": 4229730,
  "has_container_name": 998375407,
  "has_container_volume_issue_pages": 99419112,
  "has_contrib_container_year": 951540575,
  "has_contrib_raw_names": 1243059610,
  "has_doi": 620638102,
  "has_index": 1682302484,
  "has_issue": 132575565,
  "has_key": 1559104305,
  "has_locator": 415631451,
  "has_pages": 578644324,
  "has_pmcid": 129593,
  "has_pmid": 80598430,
  "has_publisher": 68809489,
  "has_release_ident": 1733948346,
  "has_release_year": 1671156907,
  "has_target_release_id": 71771437,
  "has_title": 895913346,
  "has_title_container_year": 608802846,
  "has_title_contrib_year": 800324815,
  "has_unstructured": 1095871772,
  "has_url": 44860180,
  "has_volume": 901926534,
  "has_work_ident": 1733948346,
  "has_year": 1224602612,
  "source_crossref": 716691444,
  "source_datacite": 97088644,
  "source_fatcat": 31681853,
  "source_grobid": 778474113,
  "source_pubmed": 110012292,
  "total": 1733948346
}
```

Overview of field cooccurences (running):

```shell
$ zstdcat -T0 /bigger/scholar/fatcat_scholar_work_fulltext.refs.json.zst | pv -l | \
    parallel -j 16 --pipe --roundrobin python extra/ref_key_counter.py > ref_key_counter.json
$ python extra/ref_key_counter_merge.py < ref_key_counter.json
```

Top 3 key combinations:

```
$ jq -r '.c | to_entries[] | [.key, .value] | @tsv' ref_key_counter_merge.json | sort -nrk2,2 2> /dev/null | head -30 | column -t
container_name|contrib_raw_names|pages|title|unstructured|volume|year        259790699
doi                                                                          257006166
container_name|contrib_raw_names|issue|pages|title|unstructured|volume|year  82516807
container_name|contrib_raw_names|doi|unstructured|volume|year                76858089
pmid|unstructured                                                            74462682
container_name|contrib_raw_names|doi|title|volume|year                       70821103
container_name|contrib_raw_names|year                                        64064559
container_name|contrib_raw_names|doi|volume|year                             62559951
unstructured                                                                 61711602
contrib_raw_names|pages|title|unstructured|volume|year                       61557246
container_name|contrib_raw_names|volume|year                                 49701699
container_name|contrib_raw_names|unstructured|volume|year                    36401044
contrib_raw_names|title|unstructured|year                                    35976833
container_name|contrib_raw_names|doi|pages|title|unstructured|volume|year    32506970
contrib_raw_names|publisher|title|unstructured|year                          29668363
container_name|contrib_raw_names|title|volume|year                           27447296
container_name|contrib_raw_names|unstructured|year                           26663422
container_name|contrib_raw_names|pages|title|unstructured|year               18216147
contrib_raw_names|unstructured                                               16731608
container_name|contrib_raw_names|title|unstructured|year                     15103791
doi|unstructured                                                             14464285
container_name|contrib_raw_names|doi|unstructured|year                       14207167
contrib_raw_names|pages|title|unstructured|year                              14198485
container_name|contrib_raw_names|doi|year                                    13159340
contrib_raw_names|title|year                                                 12980769
contrib_raw_names|title|unstructured                                         12595768
title|unstructured                                                           12391968
contrib_raw_names|title|volume|year                                          10636816
container_name|contrib_raw_names|doi|title|year                              10438502
container_name|contrib_raw_names|pages|publisher|title|unstructured|year     8343539
```

Loaded key sets and counts into sqlite3.

```
sqlite> select sum(count) from card;
1733948346

sqlite> select sum(count) from card where key like '%doi%' or key like '%pmid%' or key like '%pmcid%' or key like '%arxiv%';
700483341

sqlite> select sum(count) from card where key like '%title%';
895913346

sqlite> select sum(count) from card where key like '%doi%';
620638104

sqlite3> select sum(count) from card where key like '%doi%' and key like '%title%';
166109956

sqlite> select sum(count) from card where key like '%doi%' and key not like '%title%';
454528148

sqlite> select sum(count) from card where key like 'unstructured';
61711602

sqlite> select sum(count) from card where key not like '%pmid%' and key not like '%doi%' and key not like '%title%';
309033306

sqlite> select * from card order by count desc limit 20;
container_name|contrib_raw_names|pages|title|unstructured|volume|year  259790699
doi                                                                    257006166
container_name|contrib_raw_names|issue|pages|title|unstructured|volum  82516807
container_name|contrib_raw_names|doi|unstructured|volume|year          76858089
pmid|unstructured                                                      74462682
container_name|contrib_raw_names|doi|title|volume|year                 70821103
container_name|contrib_raw_names|year                                  64064559
container_name|contrib_raw_names|doi|volume|year                       62559951
unstructured                                                           61711602
contrib_raw_names|pages|title|unstructured|volume|year                 61557246
container_name|contrib_raw_names|volume|year                           49701699
container_name|contrib_raw_names|unstructured|volume|year              36401044
contrib_raw_names|title|unstructured|year                              35976833
container_name|contrib_raw_names|doi|pages|title|unstructured|volume|  32506970
contrib_raw_names|publisher|title|unstructured|year                    29668363
container_name|contrib_raw_names|title|volume|year                     27447296
container_name|contrib_raw_names|unstructured|year                     26663422
container_name|contrib_raw_names|pages|title|unstructured|year         18216147
contrib_raw_names|unstructured                                         16731608
container_name|contrib_raw_names|title|unstructured|year               15103791

sqlite> select * from card where key not like '%pmid%' and key not like '%doi%' and key not like '%title%' order by count desc limit 30;
container_name|contrib_raw_names|year|64064559
unstructured|61711602
container_name|contrib_raw_names|volume|year|49701699
container_name|contrib_raw_names|unstructured|volume|year|36401044
container_name|contrib_raw_names|unstructured|year|26663422
contrib_raw_names|unstructured|16731608
contrib_raw_names|unstructured|year|7668998
container_name|year|4946373
contrib_raw_names|pages|unstructured|4574398
|3931030
container_name|volume|year|3879804
contrib_raw_names|year|3634698
container_name|contrib_raw_names|2530058
contrib_raw_names|pages|unstructured|volume|1963485
container_name|contrib_raw_names|issue|unstructured|volume|year|1895141
contrib_raw_names|pages|unstructured|volume|year|1825755
contrib_raw_names|pages|unstructured|year|1759744
contrib_raw_names|publisher|unstructured|year|1652931
contrib_raw_names|1533772
container_name|contrib_raw_names|volume|1183234
year|1057837
container_name|contrib_raw_names|issue|volume|year|857175
contrib_raw_names|unstructured|volume|850630
container_name|unstructured|year|842953
contrib_raw_names|unstructured|url|year|764268
contrib_raw_names|unstructured|url|644762
contrib_raw_names|unstructured|volume|year|591568
contrib_raw_names|publisher|unstructured|579832
container_name|471755
contrib_raw_names|pages|publisher|unstructured|year|402852
```

## Refs as releases

We convert refs to release like docs in order to run clustering on it.

About 530M refs have to title, yet they have some kind of identifier (e.g. doi,
pmid, arxiv, ...).

```
$ zstdcat -T0 sha1-ef1756a5856085807742966f48d95b4cb00299a0.tsv.zst | LC_ALL=C grep -c -v -F "title"
529498074
```

We have 895M refs with title.

Rough metadata, extracted, partial or wrong, e.g "LANCET" as title.

How many items have both title and `ext_ids` - via:

```
$ zstdcat -T0 with_title.json.zst | LC_ALL=C rg -c -F "ext_ids"
```

170985267 have both title and some identifier.

Another performance data point: grepping through 300G on disk, uncompressed
takes about twice the time to uncompress (zstd) a 60G file on the fly.

Sampling with `awk` - `cat 1B.txt | awk 'NR % 1000000 == 0'`

## Ref only DOIs

There are 2649552 unique strings identified as DOI in refs, which do not match
any DOI in fatcat (normalized to lowercase). However, a link check of a sample
of 2K reveals that only 20% of those DOI actually resolve.

There may be around 500K DOI in references that do not appear in fatcat.

## Fuzzy matches

* [x] convert refs docs into partial release entities
* [x] concatenate with fatcat releases
* [-] cluster; note: huge file, stopped

Ran a sample on 11M (1.4G compressed) docs (10M refs, 1M fatcat), clustered in
18min.

Found 15214 clusters, but all with fatcat, not refs/fatcat. Need to extend the dataset size.

Running sample with 110M docs (100M refs, 10M fatcat; 14G compressed; clustered
96min); found 231998 clusters, but again none between the refs and fatcat
groups.

## Quality observations

A case, where we seem to have GROBID output in the refs dataset, from a PDF, which does not fit the PDF linked:

* https://fatcat.wiki/release/5al5q6ksx5dxhhtmt4cjajehge (should be an article, 157-187), but:
* [https://smartech.gatech.edu/bitstream/handle/1853/26610/zhang_lei_200812_phd.pdf](https://smartech.gatech.edu/bitstream/handle/1853/26610/zhang_lei_200812_phd.pdf) is a thesis (150+ pages)

## Approximate string match

* in refs: "Turkish Cypriots fear implications of Cyprus EU presidency" - but
  we have: https://fatcat.wiki/release/vp4h6jp65bfuro2wrkh3qg3ody, "Turkey's
boycott of the Cyprus EU presidency" plus duplication. We would need an approx. string match, or substring match.


# Lookup storage

* Can we convert our JSON into parquet for faster queries? Should we index by
  release? By title?

# Funnel Setup

* [ ] refs with a DOI (620,638,104); lookup the DOI; keep (refrid, rid)
* [ ] refs with other extids; lookup id in extracted lists from fatcat
* [ ] refs w/o extids, but with a title