aboutsummaryrefslogtreecommitdiffstats
path: root/extra/mag
diff options
context:
space:
mode:
authorMartin Czygan <martin.czygan@gmail.com>2021-09-28 17:16:06 +0200
committerMartin Czygan <martin.czygan@gmail.com>2021-09-28 17:16:06 +0200
commite8484d7a78cbbe2905f45002a7f15adbbfcb86ef (patch)
treea3dce5fc340dc193e637adcb4d8829dc82f2fd01 /extra/mag
parent67d163e6e57275a3969e0450b3ea2cb6c3e285b6 (diff)
downloadrefcat-e8484d7a78cbbe2905f45002a7f15adbbfcb86ef.tar.gz
refcat-e8484d7a78cbbe2905f45002a7f15adbbfcb86ef.zip
mag: update notes
Diffstat (limited to 'extra/mag')
-rw-r--r--extra/mag/README.md31
1 files changed, 31 insertions, 0 deletions
diff --git a/extra/mag/README.md b/extra/mag/README.md
index cd4ec70..1c9b063 100644
--- a/extra/mag/README.md
+++ b/extra/mag/README.md
@@ -79,3 +79,34 @@ Creating lowercase, unique sorted version:
```
$ time zstdcat -T0 doi_refs.tsv.zst| tr '[[:upper:]]' '[[:lower:]]' | LC_ALL=C sort -u -T /sandcrawler-db/tmp-refcat/ -S50% > doi_refs_lower_sorted.tsv.zst
```
+
+## Synopsis
+
+* OCI
+* MAG
+* refcat
+
+
+refcat:
+
+```
+$ zstdcat -T0 /magna/refcat/2021-07-28/BrefDOIOnly/date-2021-07-28.tsv.zst| pv -l | wc -l
+1.52G 0:09:30 [2.66M/s] [ <=> ]
+1516746047
+```
+
+slight filtering:
+
+```
+zstdcat -T0 /magna/refcat/2021-07-28/BrefDOIOnly/date-2021-07-28.tsv.zst| pv -l | LC_ALL=C grep -c ^1
+1482827332
+```
+
+
+oci:
+
+```
+$ zstdcat -T0 /magna/refcat/2021-07-28/COCIDOIOnly/date-2021-07-28.tsv.zst| pv -l | wc -l
+1.09G 0:07:12 [2.53M/s]
+1094394799
+```