# Version Zero Graph * ~1.8B refs * ~250M w/ titles * ~40M URLs * ~25M titles exact match w/ fatcat ``` 127589929 fatcat_lower.tsv 251934865 refs_lower.tsv 34931967 common_lower.tsv 26840211 common.tsv ``` ---- # Input Heavy intermediate schema from [fatcat-scholar](https://git.archive.org/webgroup/fatcat-scholar/), raw names of journal titles, partial metadata. Example line: ```json { "biblio": { "contrib_raw_names": [ "Maria Azevedo E Castro", "Gabriela" ], "title": "Imaginação em Paul Ricoeur", "unstructured": "AZEVEDO E CASTRO, Maria Gabriela. Imaginação em Paul Ricoeur." }, "index": 0, "key": "b0", "ref_source": "grobid", "release_ident": "ruhcoyvxxnbc5ljsgtwhnolx3i", "release_year": 2018, "work_ident": "aaaes3j4argnjbkzdvud5r4zdi" } ``` 280M docs per file: ``` $ unpigz -c fatcat_scholar_work_fulltext.split_00.refs.json.gz | wc -l 285004451 ``` Around 1,733,267,886 total reference entries. Sample completeness (10M, 1M docs, ...): ``` { "biblio": 10000000, "biblio.arxiv_id": 23227, "biblio.container_name": 5760760, "biblio.contrib_raw_names": 7156385, "biblio.doi": 3584451, "biblio.issue": 763784, "biblio.pages": 3331911, "biblio.pmcid": 776, "biblio.pmid": 471338, "biblio.publisher": 398305, "biblio.title": 5164864, "biblio.unstructured": 6304402, "biblio.url": 256771, "biblio.volume": 5202508, "biblio.year": 7055442, "index": 10000000, "key": 8986307, "locator": 2390436, "ref_source": 10000000, "release_ident": 10000000, "release_year": 9629380, "target_release_id": 419033, "work_ident": 10000000 } ``` A smaller sample: ``` { "biblio": 1000000, "biblio.arxiv_id": 1804, "biblio.container_name": 580018, "biblio.contrib_raw_names": 722526, "biblio.doi": 355664, "biblio.issue": 79145, "biblio.pages": 337716, "biblio.pmcid": 241, "biblio.pmid": 47500, "biblio.publisher": 39840, "biblio.title": 518449, "biblio.unstructured": 643743, "biblio.url": 27535, "biblio.volume": 526148, "biblio.year": 713331, "index": 1000000, "key": 904205, "locator": 241850, "ref_source": 1000000, "release_ident": 1000000, "release_year": 966723, "target_release_id": 42333, "work_ident": 1000000 } ``` ---- ## DOI graph (v2020-01-22) * 615514019 lines joined We have "inbound" links for 41950903 records. However, the top ranked docs seem invalid, datasets from a few prefixes, mostly from datacite. A few paper examples: * https://fatcat.wiki/release/6ykebula5vgtbinbqhftun7jcy > Effect of intensive blood-glucose control with metformin on complications in overweight patients with type 2 diabetes (UKPDS 34) * Google Scholar: [7981](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Effect+of+intensive+blood-glucose+control+with+metformin+on+complications+in+overweight+&btnG=) * CG: 2105 ---- * https://fatcat.wiki/release/3katpfxlafezdb2rmgoesgbhkq > The pyramid of corporate social responsibility * GS: [12690](https://scholar.google.com/scholar?cluster=13669080523806449819&hl=en&as_sdt=0,5&sciodt=0,5) * CG: 2100 ---- > Policy Paradigms, Social Learning, and the State * https://fatcat.wiki/release/m3kxbmcsxnfhzajp5zrdmvirlm * GS: [8675](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Policy+Paradigms%2C+Social+Learning%2C+and+the+State&btnG=) * CG: 2077 * OCI: [2457](https://opencitations.net/index/coci/api/v1/citations/10.2307/422246) ---- > Efficacy and safety of sorafenib in patients in the Asia-Pacific * GS: [4780](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22Efficacy+and+safety+of+sorafenib+in+patients+in+the+Asia-Pacific%22&btnG=) * CG: 1829 * OCI: [2326](https://opencitations.net/index/coci/api/v1/citations/10.1016/s1470-2045(08)70285-7) ---- * https://fatcat.wiki/release/qbdkrwg2dzbpfcdcsczy7ekjk4 * GS: [4172](https://scholar.google.com/scholar?cluster=10863619180312621990&hl=en&as_sdt=0,5&sciodt=0,5) * CG: 1303 * OCI: [1776](https://opencitations.net/index/coci/api/v1/citations/10.1038/18884), 48, https://opencitations.net/browser/br/107462 ---- > Oxidative stress and some antioxidant systems in acid rain-treated bean plants * https://fatcat.wiki/release/qrsuwvixbvgvpn4qzlum7btysi * GS: [2613](https://scholar.google.com/scholar?cluster=13765746500938441584&hl=en&as_sdt=2005&sciodt=0,5) * CG: 940 * OCI: [1042](https://opencitations.net/index/coci/api/v1/citations/10.1016/s0168-9452(99)00197-1) Not sure, what this interface does, but it says "20" (?): [https://opencitations.net/search?text=10.1016%2Fs0168-9452%2899%2900197-1&rule=doi](https://opencitations.net/search?text=10.1016%2Fs0168-9452%2899%2900197-1&rule=doi), https://opencitations.net/browser/br/808900 ---- > Weak pairwise correlations imply strongly correlated network states in a neural population * https://fatcat.wiki/release/zga73tc25nabdpt42m5ehm2kmm * GS: [1597](https://scholar.google.com/scholar?cluster=3499093978246315979&hl=en&as_sdt=0,5&sciodt=0,5) * CG: 706 * OCI: [794](https://opencitations.net/index/coci/api/v1/citations/10.1038/nature04701) ---- * https://fatcat.wiki/release/fm3ni5ppyzbknj2ke3rcn3qwva * GS: [3118](https://scholar.google.com/scholar?cluster=495710774871014506&hl=en&as_sdt=0,5&sciodt=0,5) * CG: 679 * OCI: [1092](https://opencitations.net/index/coci/api/v1/citations/10.1056/nejm199505183322008) ---- * https://fatcat.wiki/release/5i5j4qejbjey5pt56n25qmxque * GS: [1043](https://scholar.google.de/scholar?cluster=2106266772718631212&hl=en&as_sdt=2005&sciodt=0,5) * CG: 421 * OCI: [406](https://opencitations.net/index/coci/api/v1/citations/10.1029/2003wr002086) ---- * https://fatcat.wiki/release/agrh573u7nc4hd5pmyptoasvua * GS: [1391](https://scholar.google.de/scholar?cluster=17826621684475874878&hl=en&as_sdt=2005&sciodt=0,5) * CG: 388 * OCI: [444](https://opencitations.net/index/coci/api/v1/citations/10.1136/bjsm.2006.033548) ---- * https://fatcat.wiki/release/ellwakbl7rdlvgqcu7d3ss7wwa * GS: [593](https://scholar.google.com/scholar?cluster=537065614856795966&hl=en&as_sdt=2005&sciodt=0,5) * CG: 315 * OCI: [348](https://opencitations.net/index/coci/api/v1/citations/10.1038/nature05136) ---- * https://fatcat.wiki/release/dj6pufwjnnhfvpwj7tuyfl43mq * GS: [1948](https://scholar.google.com/scholar?q=A%20Simple%20Approximate%20Long-Memory%20Model%20of%20Realized%20Volatility) * CG: 275 * OCI: [336](https://opencitations.net/index/coci/api/v1/citations/10.1093/jjfinec/nbp001) ## Refs completeness Use `ref_counter.py` and `ref_key_counter.py` to assess completeness of refs. Example (1.8B): ```json { "has_any_extid": 700483339, "has_arxiv_id": 4229730, "has_container_name": 998375407, "has_container_volume_issue_pages": 99419112, "has_contrib_container_year": 951540575, "has_contrib_raw_names": 1243059610, "has_doi": 620638102, "has_index": 1682302484, "has_issue": 132575565, "has_key": 1559104305, "has_locator": 415631451, "has_pages": 578644324, "has_pmcid": 129593, "has_pmid": 80598430, "has_publisher": 68809489, "has_release_ident": 1733948346, "has_release_year": 1671156907, "has_target_release_id": 71771437, "has_title": 895913346, "has_title_container_year": 608802846, "has_title_contrib_year": 800324815, "has_unstructured": 1095871772, "has_url": 44860180, "has_volume": 901926534, "has_work_ident": 1733948346, "has_year": 1224602612, "source_crossref": 716691444, "source_datacite": 97088644, "source_fatcat": 31681853, "source_grobid": 778474113, "source_pubmed": 110012292, "total": 1733948346 } ``` Overview of field cooccurences (running): ```shell $ zstdcat -T0 /bigger/scholar/fatcat_scholar_work_fulltext.refs.json.zst | pv -l | \ parallel -j 16 --pipe --roundrobin python extra/ref_key_counter.py > ref_key_counter.json $ python extra/ref_key_counter_merge.py < ref_key_counter.json ``` Top 3 key combinations: ``` $ jq -r '.c | to_entries[] | [.key, .value] | @tsv' ref_key_counter_merge.json | sort -nrk2,2 2> /dev/null | head -30 | column -t container_name|contrib_raw_names|pages|title|unstructured|volume|year 259790699 doi 257006166 container_name|contrib_raw_names|issue|pages|title|unstructured|volume|year 82516807 container_name|contrib_raw_names|doi|unstructured|volume|year 76858089 pmid|unstructured 74462682 container_name|contrib_raw_names|doi|title|volume|year 70821103 container_name|contrib_raw_names|year 64064559 container_name|contrib_raw_names|doi|volume|year 62559951 unstructured 61711602 contrib_raw_names|pages|title|unstructured|volume|year 61557246 container_name|contrib_raw_names|volume|year 49701699 container_name|contrib_raw_names|unstructured|volume|year 36401044 contrib_raw_names|title|unstructured|year 35976833 container_name|contrib_raw_names|doi|pages|title|unstructured|volume|year 32506970 contrib_raw_names|publisher|title|unstructured|year 29668363 container_name|contrib_raw_names|title|volume|year 27447296 container_name|contrib_raw_names|unstructured|year 26663422 container_name|contrib_raw_names|pages|title|unstructured|year 18216147 contrib_raw_names|unstructured 16731608 container_name|contrib_raw_names|title|unstructured|year 15103791 doi|unstructured 14464285 container_name|contrib_raw_names|doi|unstructured|year 14207167 contrib_raw_names|pages|title|unstructured|year 14198485 container_name|contrib_raw_names|doi|year 13159340 contrib_raw_names|title|year 12980769 contrib_raw_names|title|unstructured 12595768 title|unstructured 12391968 contrib_raw_names|title|volume|year 10636816 container_name|contrib_raw_names|doi|title|year 10438502 container_name|contrib_raw_names|pages|publisher|title|unstructured|year 8343539 ``` Loaded key sets and counts into sqlite3. ``` sqlite> select sum(count) from card; 1733948346 sqlite> select sum(count) from card where key like '%doi%' or key like '%pmid%' or key like '%pmcid%' or key like '%arxiv%'; 700483341 sqlite> select sum(count) from card where key like '%title%'; 895913346 sqlite> select sum(count) from card where key like '%doi%'; 620638104 sqlite3> select sum(count) from card where key like '%doi%' and key like '%title%'; 166109956 sqlite> select sum(count) from card where key like '%doi%' and key not like '%title%'; 454528148 sqlite> select sum(count) from card where key like 'unstructured'; 61711602 sqlite> select sum(count) from card where key not like '%pmid%' and key not like '%doi%' and key not like '%title%'; 309033306 sqlite> select * from card order by count desc limit 20; container_name|contrib_raw_names|pages|title|unstructured|volume|year 259790699 doi 257006166 container_name|contrib_raw_names|issue|pages|title|unstructured|volum 82516807 container_name|contrib_raw_names|doi|unstructured|volume|year 76858089 pmid|unstructured 74462682 container_name|contrib_raw_names|doi|title|volume|year 70821103 container_name|contrib_raw_names|year 64064559 container_name|contrib_raw_names|doi|volume|year 62559951 unstructured 61711602 contrib_raw_names|pages|title|unstructured|volume|year 61557246 container_name|contrib_raw_names|volume|year 49701699 container_name|contrib_raw_names|unstructured|volume|year 36401044 contrib_raw_names|title|unstructured|year 35976833 container_name|contrib_raw_names|doi|pages|title|unstructured|volume| 32506970 contrib_raw_names|publisher|title|unstructured|year 29668363 container_name|contrib_raw_names|title|volume|year 27447296 container_name|contrib_raw_names|unstructured|year 26663422 container_name|contrib_raw_names|pages|title|unstructured|year 18216147 contrib_raw_names|unstructured 16731608 container_name|contrib_raw_names|title|unstructured|year 15103791 sqlite> select * from card where key not like '%pmid%' and key not like '%doi%' and key not like '%title%' order by count desc limit 30; container_name|contrib_raw_names|year|64064559 unstructured|61711602 container_name|contrib_raw_names|volume|year|49701699 container_name|contrib_raw_names|unstructured|volume|year|36401044 container_name|contrib_raw_names|unstructured|year|26663422 contrib_raw_names|unstructured|16731608 contrib_raw_names|unstructured|year|7668998 container_name|year|4946373 contrib_raw_names|pages|unstructured|4574398 |3931030 container_name|volume|year|3879804 contrib_raw_names|year|3634698 container_name|contrib_raw_names|2530058 contrib_raw_names|pages|unstructured|volume|1963485 container_name|contrib_raw_names|issue|unstructured|volume|year|1895141 contrib_raw_names|pages|unstructured|volume|year|1825755 contrib_raw_names|pages|unstructured|year|1759744 contrib_raw_names|publisher|unstructured|year|1652931 contrib_raw_names|1533772 container_name|contrib_raw_names|volume|1183234 year|1057837 container_name|contrib_raw_names|issue|volume|year|857175 contrib_raw_names|unstructured|volume|850630 container_name|unstructured|year|842953 contrib_raw_names|unstructured|url|year|764268 contrib_raw_names|unstructured|url|644762 contrib_raw_names|unstructured|volume|year|591568 contrib_raw_names|publisher|unstructured|579832 container_name|471755 contrib_raw_names|pages|publisher|unstructured|year|402852 ``` ## Refs as releases We convert refs to release like docs in order to run clustering on it. About 530M refs have to title, yet they have some kind of identifier (e.g. doi, pmid, arxiv, ...). ``` $ zstdcat -T0 sha1-ef1756a5856085807742966f48d95b4cb00299a0.tsv.zst | LC_ALL=C grep -c -v -F "title" 529498074 ``` We have 895M refs with title. Rough metadata, extracted, partial or wrong, e.g "LANCET" as title. How many items have both title and `ext_ids` - via: ``` $ zstdcat -T0 with_title.json.zst | LC_ALL=C rg -c -F "ext_ids" ``` 170985267 have both title and some identifier. Another performance data point: grepping through 300G on disk, uncompressed takes about twice the time to uncompress (zstd) a 60G file on the fly. Sampling with `awk` - `cat 1B.txt | awk 'NR % 1000000 == 0'` ## Ref only DOIs There are 2649552 unique strings identified as DOI in refs, which do not match any DOI in fatcat (normalized to lowercase). However, a link check of a sample of 2K reveals that only 20% of those DOI actually resolve. There may be around 500K DOI in references that do not appear in fatcat. ## Fuzzy matches * [x] convert refs docs into partial release entities * [x] concatenate with fatcat releases * [-] cluster; note: huge file, stopped Ran a sample on 11M (1.4G compressed) docs (10M refs, 1M fatcat), clustered in 18min. Found 15214 clusters, but all with fatcat, not refs/fatcat. Need to extend the dataset size. Running sample with 110M docs (100M refs, 10M fatcat; 14G compressed; clustered 96min); found 231998 clusters, but again none between the refs and fatcat groups. ## Quality observations A case, where we seem to have GROBID output in the refs dataset, from a PDF, which does not fit the PDF linked: * https://fatcat.wiki/release/5al5q6ksx5dxhhtmt4cjajehge (should be an article, 157-187), but: * [https://smartech.gatech.edu/bitstream/handle/1853/26610/zhang_lei_200812_phd.pdf](https://smartech.gatech.edu/bitstream/handle/1853/26610/zhang_lei_200812_phd.pdf) is a thesis (150+ pages) ## Approximate string match * in refs: "Turkish Cypriots fear implications of Cyprus EU presidency" - but we have: https://fatcat.wiki/release/vp4h6jp65bfuro2wrkh3qg3ody, "Turkey's boycott of the Cyprus EU presidency" plus duplication. We would need an approx. string match, or substring match. # Lookup storage * Can we convert our JSON into parquet for faster queries? Should we index by release? By title? # Funnel Setup * [ ] refs with a DOI (620,638,104); lookup the DOI; keep (refrid, rid) * [ ] refs with other extids; lookup id in extracted lists from fatcat * [ ] refs w/o extids, but with a title