# Notes on MAG

Using: https://archive.org/details/mag-2021-06-07

* 260M entities
* 96.7M with DOI

In order to generate a doi-to-doi version, we need to:

* create a mapping from id-to-doi (db is slow, in-memory needs 16G+ RAM)
* apply the mapping to the PaperReferences file

```
# https://docs.microsoft.com/en-us/academic-services/graph/reference-data-schema
# MAG has a Papers.txt.gz file containing the ID and DOI
#
# (1) create a mapping from id to doi (e.g. in sqlite3)
# (2) turn id-to-id references into doi-to-doi with lookup table
#
# $ unpigz -c /magna/data/mag-2020-06-25/Papers.txt.gz | cut -f1,3 | pv -l
# 238M 0:11:16 [ 353k/s]
#
# $ unpigz -c /magna/data/mag-2021-06-07/Papers.txt.gz | cut -f1,3 | pv -l > /dev/null
# 260M 0:10:32 [ 412k/s]
#
# $ unpigz -c /magna/data/mag-2021-06-07/Papers.txt.gz | cut -f1,3 | awk '$2 != ""' | pv -l > /dev/null
# 96.7M 0:11:05 [ 145k/s]
#
# $ time unpigz -c /magna/data/mag-2021-06-07/Papers.txt.gz | cut -f1,3 | awk '$2 != ""' | mkocidb -o mag_id_doi.db
#
# 2021/09/27 17:08:45 [ok] initialized database -- /sandcrawler-db/tmp-refcat/mag_id_doi.db
# written 2.9G -- 3.4M/s
# 2021/09/27 17:23:11 import done
# 2021/09/27 17:23:11 creating index
# 2021/09/27 17:26:44 [ok] 1/2 created index -- /sandcrawler-db/tmp-refcat/mag_id_doi.db
# 2021/09/27 17:31:53 [ok] 2/2 created index -- /sandcrawler-db/tmp-refcat/mag_id_doi.db
#
# real    23m7.744s
# user    30m55.642s
# sys     4m17.959s
#
# Can use a in memory map, too - sqlite3 lookups for 2B+ items takes a while at
# 30 Kqps; takes 30% of RAM on 48GB, sigh; just map[int]string from paper id to doi.
#
# Prepare the 2-TSV list.
#
# $ time unpigz -c /magna/data/mag-2021-06-07/Papers.txt.gz | cut -f1,3 | awk '$2 != ""' | pv -l > /magna/data/mag-2021-06-07/mapping.tsv
#
# real    15m27.348s
# user    21m4.297s
# sys     3m34.441s
#
# $ time zstdcat -T0 PaperReferences.txt.zst |
#     magrefs-mem -f /magna/data/mag-2021-06-07/mapping.tsv |
#     pv -l | zstd -c -T0 > doi_refs.tsv.zst
#
# real    41m12.911s
# user    223m14.467s
# sys     30m43.260s
#
# $ zstdcat -T0 doi_refs.tsv.zst| pv -l | wc -l
# 1.32G 0:06:33 [3.34M/s] [                             <=>                                                                                                                                                                                     ]1315040677
# 1315040677
```

Finding 1,315,040,677 DOI-to-DOI mappings.

Total edges: 1,832,226,781

```
$ zstdcat -T0 PaperReferences.txt.zst | pv -l | wc -l
1832226781
```

Non-DOI edges: 517,186,104

Creating lowercase, unique sorted version:

```
$ time zstdcat -T0 doi_refs.tsv.zst| tr '[[:upper:]]' '[[:lower:]]' | LC_ALL=C sort -u -T /sandcrawler-db/tmp-refcat/ -S50% > doi_refs_lower_sorted.tsv.zst
```

## Synopsis

* OCI
* MAG
* refcat


refcat:

```
$ zstdcat -T0 /magna/refcat/2021-07-28/BrefDOIOnly/date-2021-07-28.tsv.zst| pv -l | wc -l
1.52G 0:09:30 [2.66M/s] [                                                                                                                         <=>                                                                                         ]
1516746047
```

slight filtering:

```
zstdcat -T0 /magna/refcat/2021-07-28/BrefDOIOnly/date-2021-07-28.tsv.zst| pv -l | LC_ALL=C  grep -c ^1
1482827332
```


oci:

```
$ zstdcat -T0 /magna/refcat/2021-07-28/COCIDOIOnly/date-2021-07-28.tsv.zst| pv -l | wc -l
1.09G 0:07:12 [2.53M/s]
1094394799
```