diff options
| author | Bryan Newbold <bnewbold@robocracy.org> | 2022-01-21 16:53:19 -0800 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@robocracy.org> | 2022-01-21 16:53:19 -0800 |
| commit | 58b38322d190655bbd4df35285daf0aa2ed4036c (patch) | |
| tree | 39f33da21b232e762dd91dd0c85000bbe1507051 | |
| parent | 04b5d02a3dad37d66a73afb7ea6bf9c7a78427dd (diff) | |
| download | fatcat-58b38322d190655bbd4df35285daf0aa2ed4036c.tar.gz fatcat-58b38322d190655bbd4df35285daf0aa2ed4036c.zip | |
container counts update process README
| -rw-r--r-- | extra/container_count_update/README.md | 41 |
1 files changed, 41 insertions, 0 deletions
diff --git a/extra/container_count_update/README.md b/extra/container_count_update/README.md new file mode 100644 index 00000000..3e9a4315 --- /dev/null +++ b/extra/container_count_update/README.md @@ -0,0 +1,41 @@ + +Here are the fields we want to populate: + + "releases_total": { "type": "integer" }, + "preservation_bright": { "type": "integer" }, + "preservation_dark": { "type": "integer" }, + "preservation_shadows_only":{ "type": "integer" }, + "preservation_none": { "type": "integer" }, + +Populate local index for testing: + + fatcat-cli search container --index-json --limit 100 state:active \ + | pv -l \ + > container_es_docs.json + + cat container_es_docs.json \ + | esbulk -verbose -index fatcat_container_v03c -id ident + + cat container_es_docs.json \ + | jq .ident -r \ + > container_idents.tsv + +Quick way to dump all idents in the current index: + + fatcat-cli search container --index-json --limit 0 state:active \ + | jq .ident -r \ + | pv -l \ + > container_idents.tsv + + cat container_idents.tsv \ + | parallel -j10 curl --fail -s 'https://fatcat.wiki/container/{}/stats.json' \ + | jq -c . \ + | pv -l \ + > container_stats.json + + cat container_stats.json \ + | jq '{ ident: .ident, releases_total: .total, preservation_bright: .preservation.bright, preservation_dark: .preservation.dark, preservation_shadows_only: .preservation.shadows_only, preservation_none: .preservation.none }' -c \ + | esbulk -verbose -index fatcat_container_v03c -optype update -id ident + +This requires a recent version of esbulk (v0.7.5+) + |