Version Zero Graph
- ~1.8B refs
- ~250M w/ titles
- ~40M URLs
- ~25M titles exact match w/ fatcat
127589929 fatcat_lower.tsv
251934865 refs_lower.tsv
34931967 common_lower.tsv
26840211 common.tsv
Input
Heavy intermediate schema from fatcat-scholar, raw names of journal titles, partial metadata.
Example line:
{
"biblio": {
"contrib_raw_names": [
"Maria Azevedo E Castro",
"Gabriela"
],
"title": "Imaginação em Paul Ricoeur",
"unstructured": "AZEVEDO E CASTRO, Maria Gabriela. Imaginação em Paul Ricoeur."
},
"index": 0,
"key": "b0",
"ref_source": "grobid",
"release_ident": "ruhcoyvxxnbc5ljsgtwhnolx3i",
"release_year": 2018,
"work_ident": "aaaes3j4argnjbkzdvud5r4zdi"
}
280M docs per file:
$ unpigz -c fatcat_scholar_work_fulltext.split_00.refs.json.gz | wc -l
285004451
Around 1,733,267,886 total reference entries. Sample completeness (10M, 1M docs, ...):
{
"biblio": 10000000,
"biblio.arxiv_id": 23227,
"biblio.container_name": 5760760,
"biblio.contrib_raw_names": 7156385,
"biblio.doi": 3584451,
"biblio.issue": 763784,
"biblio.pages": 3331911,
"biblio.pmcid": 776,
"biblio.pmid": 471338,
"biblio.publisher": 398305,
"biblio.title": 5164864,
"biblio.unstructured": 6304402,
"biblio.url": 256771,
"biblio.volume": 5202508,
"biblio.year": 7055442,
"index": 10000000,
"key": 8986307,
"locator": 2390436,
"ref_source": 10000000,
"release_ident": 10000000,
"release_year": 9629380,
"target_release_id": 419033,
"work_ident": 10000000
}
A smaller sample:
{
"biblio": 1000000,
"biblio.arxiv_id": 1804,
"biblio.container_name": 580018,
"biblio.contrib_raw_names": 722526,
"biblio.doi": 355664,
"biblio.issue": 79145,
"biblio.pages": 337716,
"biblio.pmcid": 241,
"biblio.pmid": 47500,
"biblio.publisher": 39840,
"biblio.title": 518449,
"biblio.unstructured": 643743,
"biblio.url": 27535,
"biblio.volume": 526148,
"biblio.year": 713331,
"index": 1000000,
"key": 904205,
"locator": 241850,
"ref_source": 1000000,
"release_ident": 1000000,
"release_year": 966723,
"target_release_id": 42333,
"work_ident": 1000000
}
DOI graph (v2020-01-22)
- 615514019 lines joined
We have "inbound" links for 41950903 records.
However, the top ranked docs seem invalid, datasets from a few prefixes, mostly from datacite.
A few paper examples:
- https://fatcat.wiki/release/6ykebula5vgtbinbqhftun7jcy
Effect of intensive blood-glucose control with metformin on complications in overweight patients with type 2 diabetes (UKPDS 34)
- Google Scholar: 7981
- CG: 2105
- https://fatcat.wiki/release/3katpfxlafezdb2rmgoesgbhkq
The pyramid of corporate social responsibility
- GS: 12690
- CG: 2100
Policy Paradigms, Social Learning, and the State
Efficacy and safety of sorafenib in patients in the Asia-Pacific
- https://fatcat.wiki/release/qbdkrwg2dzbpfcdcsczy7ekjk4
- GS: 4172
- CG: 1303
- OCI: 1776, 48, https://opencitations.net/browser/br/107462
Oxidative stress and some antioxidant systems in acid rain-treated bean plants
Not sure, what this interface does, but it says "20" (?): https://opencitations.net/search?text=10.1016%2Fs0168-9452%2899%2900197-1&rule=doi, https://opencitations.net/browser/br/808900
Weak pairwise correlations imply strongly correlated network states in a neural population
Refs completeness
Use ref_counter.py and ref_key_counter.py to assess completeness of refs.
Example (1.8B):
{
"has_any_extid": 700483339,
"has_arxiv_id": 4229730,
"has_container_name": 998375407,
"has_container_volume_issue_pages": 99419112,
"has_contrib_container_year": 951540575,
"has_contrib_raw_names": 1243059610,
"has_doi": 620638102,
"has_index": 1682302484,
"has_issue": 132575565,
"has_key": 1559104305,
"has_locator": 415631451,
"has_pages": 578644324,
"has_pmcid": 129593,
"has_pmid": 80598430,
"has_publisher": 68809489,
"has_release_ident": 1733948346,
"has_release_year": 1671156907,
"has_target_release_id": 71771437,
"has_title": 895913346,
"has_title_container_year": 608802846,
"has_title_contrib_year": 800324815,
"has_unstructured": 1095871772,
"has_url": 44860180,
"has_volume": 901926534,
"has_work_ident": 1733948346,
"has_year": 1224602612,
"source_crossref": 716691444,
"source_datacite": 97088644,
"source_fatcat": 31681853,
"source_grobid": 778474113,
"source_pubmed": 110012292,
"total": 1733948346
}
Overview of field cooccurences (running):
$ zstdcat -T0 /bigger/scholar/fatcat_scholar_work_fulltext.refs.json.zst | pv -l | \
parallel -j 16 --pipe --roundrobin python extra/ref_key_counter.py > ref_key_counter.json
$ python extra/ref_key_counter_merge.py < ref_key_counter.json
Top 3 key combinations:
$ jq -r '.c | to_entries[] | [.key, .value] | @tsv' ref_key_counter_merge.json | sort -nrk2,2 2> /dev/null | head -30 | column -t
container_name|contrib_raw_names|pages|title|unstructured|volume|year 259790699
doi 257006166
container_name|contrib_raw_names|issue|pages|title|unstructured|volume|year 82516807
container_name|contrib_raw_names|doi|unstructured|volume|year 76858089
pmid|unstructured 74462682
container_name|contrib_raw_names|doi|title|volume|year 70821103
container_name|contrib_raw_names|year 64064559
container_name|contrib_raw_names|doi|volume|year 62559951
unstructured 61711602
contrib_raw_names|pages|title|unstructured|volume|year 61557246
container_name|contrib_raw_names|volume|year 49701699
container_name|contrib_raw_names|unstructured|volume|year 36401044
contrib_raw_names|title|unstructured|year 35976833
container_name|contrib_raw_names|doi|pages|title|unstructured|volume|year 32506970
contrib_raw_names|publisher|title|unstructured|year 29668363
container_name|contrib_raw_names|title|volume|year 27447296
container_name|contrib_raw_names|unstructured|year 26663422
container_name|contrib_raw_names|pages|title|unstructured|year 18216147
contrib_raw_names|unstructured 16731608
container_name|contrib_raw_names|title|unstructured|year 15103791
doi|unstructured 14464285
container_name|contrib_raw_names|doi|unstructured|year 14207167
contrib_raw_names|pages|title|unstructured|year 14198485
container_name|contrib_raw_names|doi|year 13159340
contrib_raw_names|title|year 12980769
contrib_raw_names|title|unstructured 12595768
title|unstructured 12391968
contrib_raw_names|title|volume|year 10636816
container_name|contrib_raw_names|doi|title|year 10438502
container_name|contrib_raw_names|pages|publisher|title|unstructured|year 8343539
Loaded key sets and counts into sqlite3.
sqlite> select sum(count) from card;
1733948346
sqlite> select sum(count) from card where key like '%doi%' or key like '%pmid%' or key like '%pmcid%' or key like '%arxiv%';
700483341
sqlite> select sum(count) from card where key like '%title%';
895913346
sqlite> select sum(count) from card where key like '%doi%';
620638104
sqlite3> select sum(count) from card where key like '%doi%' and key like '%title%';
166109956
sqlite> select sum(count) from card where key like '%doi%' and key not like '%title%';
454528148
sqlite> select sum(count) from card where key like 'unstructured';
61711602
sqlite> select sum(count) from card where key not like '%pmid%' and key not like '%doi%' and key not like '%title%';
309033306
sqlite> select * from card order by count desc limit 20;
container_name|contrib_raw_names|pages|title|unstructured|volume|year 259790699
doi 257006166
container_name|contrib_raw_names|issue|pages|title|unstructured|volum 82516807
container_name|contrib_raw_names|doi|unstructured|volume|year 76858089
pmid|unstructured 74462682
container_name|contrib_raw_names|doi|title|volume|year 70821103
container_name|contrib_raw_names|year 64064559
container_name|contrib_raw_names|doi|volume|year 62559951
unstructured 61711602
contrib_raw_names|pages|title|unstructured|volume|year 61557246
container_name|contrib_raw_names|volume|year 49701699
container_name|contrib_raw_names|unstructured|volume|year 36401044
contrib_raw_names|title|unstructured|year 35976833
container_name|contrib_raw_names|doi|pages|title|unstructured|volume| 32506970
contrib_raw_names|publisher|title|unstructured|year 29668363
container_name|contrib_raw_names|title|volume|year 27447296
container_name|contrib_raw_names|unstructured|year 26663422
container_name|contrib_raw_names|pages|title|unstructured|year 18216147
contrib_raw_names|unstructured 16731608
container_name|contrib_raw_names|title|unstructured|year 15103791
sqlite> select * from card where key not like '%pmid%' and key not like '%doi%' and key not like '%title%' order by count desc limit 30;
container_name|contrib_raw_names|year|64064559
unstructured|61711602
container_name|contrib_raw_names|volume|year|49701699
container_name|contrib_raw_names|unstructured|volume|year|36401044
container_name|contrib_raw_names|unstructured|year|26663422
contrib_raw_names|unstructured|16731608
contrib_raw_names|unstructured|year|7668998
container_name|year|4946373
contrib_raw_names|pages|unstructured|4574398
|3931030
container_name|volume|year|3879804
contrib_raw_names|year|3634698
container_name|contrib_raw_names|2530058
contrib_raw_names|pages|unstructured|volume|1963485
container_name|contrib_raw_names|issue|unstructured|volume|year|1895141
contrib_raw_names|pages|unstructured|volume|year|1825755
contrib_raw_names|pages|unstructured|year|1759744
contrib_raw_names|publisher|unstructured|year|1652931
contrib_raw_names|1533772
container_name|contrib_raw_names|volume|1183234
year|1057837
container_name|contrib_raw_names|issue|volume|year|857175
contrib_raw_names|unstructured|volume|850630
container_name|unstructured|year|842953
contrib_raw_names|unstructured|url|year|764268
contrib_raw_names|unstructured|url|644762
contrib_raw_names|unstructured|volume|year|591568
contrib_raw_names|publisher|unstructured|579832
container_name|471755
contrib_raw_names|pages|publisher|unstructured|year|402852
Refs as releases
We convert refs to release like docs in order to run clustering on it.
About 530M refs have to title, yet they have some kind of identifier (e.g. doi, pmid, arxiv, ...).
$ zstdcat -T0 sha1-ef1756a5856085807742966f48d95b4cb00299a0.tsv.zst | LC_ALL=C grep -c -v -F "title"
529498074
We have 895M refs with title.
Rough metadata, extracted, partial or wrong, e.g "LANCET" as title.
How many items have both title and ext_ids - via:
$ zstdcat -T0 with_title.json.zst | LC_ALL=C rg -c -F "ext_ids"
170985267 have both title and some identifier.
Another performance data point: grepping through 300G on disk, uncompressed takes about twice the time to uncompress (zstd) a 60G file on the fly.
Sampling with awk - cat 1B.txt | awk 'NR % 1000000 == 0'
Ref only DOIs
There are 2649552 unique strings identified as DOI in refs, which do not match any DOI in fatcat (normalized to lowercase). However, a link check of a sample of 2K reveals that only 20% of those DOI actually resolve.
There may be around 500K DOI in references that do not appear in fatcat.
Fuzzy matches
- [x] convert refs docs into partial release entities
- [x] concatenate with fatcat releases
- [-] cluster; note: huge file, stopped
Ran a sample on 11M (1.4G compressed) docs (10M refs, 1M fatcat), clustered in 18min.
Found 15214 clusters, but all with fatcat, not refs/fatcat. Need to extend the dataset size.
Running sample with 110M docs (100M refs, 10M fatcat; 14G compressed; clustered 96min); found 231998 clusters, but again none between the refs and fatcat groups.
Quality observations
A case, where we seem to have GROBID output in the refs dataset, from a PDF, which does not fit the PDF linked:
- https://fatcat.wiki/release/5al5q6ksx5dxhhtmt4cjajehge (should be an article, 157-187), but:
- https://smartech.gatech.edu/bitstream/handle/1853/26610/zhang_lei_200812_phd.pdf is a thesis (150+ pages)
Approximate string match
- in refs: "Turkish Cypriots fear implications of Cyprus EU presidency" - but we have: https://fatcat.wiki/release/vp4h6jp65bfuro2wrkh3qg3ody, "Turkey's boycott of the Cyprus EU presidency" plus duplication. We would need an approx. string match, or substring match.
Lookup storage
- Can we convert our JSON into parquet for faster queries? Should we index by release? By title?
Funnel Setup
- [ ] refs with a DOI (620,638,104); lookup the DOI; keep (refrid, rid)
- [ ] refs with other extids; lookup id in extracted lists from fatcat
- [ ] refs w/o extids, but with a title