initial ES 7.10 indexing notes

1 files changed, 103 insertions, 0 deletions

diff --git a/notes/es-indexing.md b/notes/es-indexing.md
new file mode 100644
index 0000000..04fca2e
--- /dev/null
+++ b/notes/es-indexing.md
@@ -0,0 +1,103 @@
+
+Going to do some initial indexing of refs ("BiblioRefs" schema) into
+elasticsearch 7 cluster.
+
+## 2021-04-12
+
+Reduced `number_of_shards` from 12 to 6.
+
+Create index from schema:
+
+    # note, *not* include_type_name=true in the URL
+    http put :9200/fatcat_ref_v01 < ref_schema.json
+
+Force all shards to a single machine (wbgrp-svc500):
+
+    http put ":9200/fatcat_ref_v01/_settings" index.routing.allocation.include._name=wbgrp-svc500
+
+    # would run this later, after adding more nodes to list, for production deployment
+    #http put ":9200/fatcat_ref_v01/_settings" index.number_of_replicas=1
+
+Confirm shard locations:
+
+    http get :9200/_cat/shards/fatcat_ref_v01
+
+        fatcat_ref_v01 3 p STARTED 0 208b 207.241.225.228 wbgrp-svc500
+        fatcat_ref_v01 1 p STARTED 0 208b 207.241.225.228 wbgrp-svc500
+        fatcat_ref_v01 2 p STARTED 0 208b 207.241.225.228 wbgrp-svc500
+        fatcat_ref_v01 4 p STARTED 0 208b 207.241.225.228 wbgrp-svc500
+        fatcat_ref_v01 5 p STARTED 0 208b 207.241.225.228 wbgrp-svc500
+        fatcat_ref_v01 0 p STARTED 0 208b 207.241.225.228 wbgrp-svc500
+
+Copied over `aitio:/magna/refcat/BiblioRefV2/date-2021-02-20.json.zst`, which
+seems to have been output on 2021-03-24 (the date in the filename refers to the
+date of the source raw reference dump, I believe).
+
+    du -sh /srv/fatcat/datasets/date-2021-02-20.json.zst
+    39G     /srv/fatcat/datasets/date-2021-02-20.json.zst
+
+Check that esbulk is updated:
+
+    esbulk -v
+    0.7.3
+
+Expecting on the order of:
+
+    785,569,011 edges (~103% of 12/2020 OCI/crossref release), ~39G compressed, ~288G uncompressed
+
+Start with 1 million documents to index:
+
+    zstdcat /srv/fatcat/datasets/date-2021-02-20.json.zst | head -n1000000 | esbulk -verbose -size 2000 -id _id -w 8 -index fatcat_ref_v01
+
+    => 2021/04/13 02:55:17 1000000 docs in 19.89s at 50265.935 docs/s with 8 workers
+
+    => green open fatcat_ref_v01 tpHidEK_RSSrY0YDYgTH2Q 6 0 983111 16889 305.5mb 305.5mb
+
+Very fast!
+
+Not 1 million documents; presumably some duplication due to duplicate checks?
+If we thought we could extrapolate from 1mil ~= 0.3 GB, then 1bil ~= 300 GByte,
+which seems pretty reasonable.
+
+Bump to 20 million:
+
+    zstdcat /srv/fatcat/datasets/date-2021-02-20.json.zst | head -n20000000 | esbulk -verbose -size 2000 -id _id -w 8 -index fatcat_ref_v01
+
+    => 2021/04/13 03:05:17 20000000 docs in 396.53s at 50437.434 docs/s with 8 workers
+
+    => green open fatcat_ref_v01 tpHidEK_RSSrY0YDYgTH2Q 6 0 19731816 171926 5.4gb 5.4gb
+
+Configured es-public-proxy to enable public access to `fatcat_ref`, and added an alias:
+
+    http put :9200/fatcat_ref_v01/_alias/fatcat_ref
+
+Crude queries are like:
+
+    http https://search.fatcat.wiki/fatcat_ref/_search q==target_work_ident:o547luzdqragbd66ejlm4pughi
+
+Seems like these were all join/exact matches. Would like some others, so do an additional `shuf` instead of head:
+
+    zstdcat /srv/fatcat/datasets/date-2021-02-20.json.zst | shuf -n20000000 | esbulk -verbose -size 2000 -id _id -w 8 -index fatcat_ref_v01
+
+Interestingly, `shuf` takes more CPU than `zstdcat`, presumably because it is
+doing a lot of random number generation. Note that `shuf` uses reservoir
+sampling, so is usually pretty efficient.
+
+Back over on `aitio`, trying:
+
+    zstdcat /magna/tmp/date-2021-02-20.json.zst | jq ._id -c | sort -u | wc -l
+
+Ok, just run the whole thing:
+
+    zstdcat /srv/fatcat/datasets/date-2021-02-20.json.zst | esbulk -verbose -size 2000 -id _id -w 8 -index fatcat_ref_v01
+
+    => 2021/04/13 10:10:12 785569011 docs in 22309.29s at 35212.641 docs/s with 8 workers
+    => green open fatcat_ref_v01 tpHidEK_RSSrY0YDYgTH2Q 6 0 734856532 38611510 184.2gb 184.2gb
+
+Took a bit over 6 hours, and ~180 GByte total index size. Nice!
+
+## Notes and Thoughts
+
+- many of the `ref_key` are pretty large, like `bibr4-0036933016674866`. Maybe these can be parsed down.
+- might want aggregations on `source_release_stage`, `source_year`, etc?
+- `update_ts` may not have come through correctly? it is an integer, not a datetime