aboutsummaryrefslogtreecommitdiffstats
path: root/extra/container_count_update
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@robocracy.org>2022-01-21 16:53:19 -0800
committerBryan Newbold <bnewbold@robocracy.org>2022-01-21 16:53:19 -0800
commit58b38322d190655bbd4df35285daf0aa2ed4036c (patch)
tree39f33da21b232e762dd91dd0c85000bbe1507051 /extra/container_count_update
parent04b5d02a3dad37d66a73afb7ea6bf9c7a78427dd (diff)
downloadfatcat-58b38322d190655bbd4df35285daf0aa2ed4036c.tar.gz
fatcat-58b38322d190655bbd4df35285daf0aa2ed4036c.zip
container counts update process README
Diffstat (limited to 'extra/container_count_update')
-rw-r--r--extra/container_count_update/README.md41
1 files changed, 41 insertions, 0 deletions
diff --git a/extra/container_count_update/README.md b/extra/container_count_update/README.md
new file mode 100644
index 00000000..3e9a4315
--- /dev/null
+++ b/extra/container_count_update/README.md
@@ -0,0 +1,41 @@
+
+Here are the fields we want to populate:
+
+ "releases_total": { "type": "integer" },
+ "preservation_bright": { "type": "integer" },
+ "preservation_dark": { "type": "integer" },
+ "preservation_shadows_only":{ "type": "integer" },
+ "preservation_none": { "type": "integer" },
+
+Populate local index for testing:
+
+ fatcat-cli search container --index-json --limit 100 state:active \
+ | pv -l \
+ > container_es_docs.json
+
+ cat container_es_docs.json \
+ | esbulk -verbose -index fatcat_container_v03c -id ident
+
+ cat container_es_docs.json \
+ | jq .ident -r \
+ > container_idents.tsv
+
+Quick way to dump all idents in the current index:
+
+ fatcat-cli search container --index-json --limit 0 state:active \
+ | jq .ident -r \
+ | pv -l \
+ > container_idents.tsv
+
+ cat container_idents.tsv \
+ | parallel -j10 curl --fail -s 'https://fatcat.wiki/container/{}/stats.json' \
+ | jq -c . \
+ | pv -l \
+ > container_stats.json
+
+ cat container_stats.json \
+ | jq '{ ident: .ident, releases_total: .total, preservation_bright: .preservation.bright, preservation_dark: .preservation.dark, preservation_shadows_only: .preservation.shadows_only, preservation_none: .preservation.none }' -c \
+ | esbulk -verbose -index fatcat_container_v03c -optype update -id ident
+
+This requires a recent version of esbulk (v0.7.5+)
+