aboutsummaryrefslogtreecommitdiffstats
path: root/fatcat_scholar/sim_pipeline.py
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@archive.org>2020-07-27 15:54:55 -0700
committerBryan Newbold <bnewbold@archive.org>2020-07-27 15:54:57 -0700
commit6443fd5756779e2cca986e73e5008a073ef96452 (patch)
treedeaac6ff17a9dbafcd2142c1b2a4679691403634 /fatcat_scholar/sim_pipeline.py
parent0234f6188b6f10dad96e2ea8dc67d2674c26c02d (diff)
downloadfatcat-scholar-6443fd5756779e2cca986e73e5008a073ef96452.tar.gz
fatcat-scholar-6443fd5756779e2cca986e73e5008a073ef96452.zip
json: exclude None in output, and sort keys
These are both size/performance enhancements. Not including 'None' values will reduce document sizes on-disk and over network, particularly for intermediate objects. Sorting by key should improve compression ratios across multiple documents, both on-disk (gzip) and in elasticsearch itself: https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html#_put_fields_in_the_same_order_in_documents
Diffstat (limited to 'fatcat_scholar/sim_pipeline.py')
-rw-r--r--fatcat_scholar/sim_pipeline.py2
1 files changed, 1 insertions, 1 deletions
diff --git a/fatcat_scholar/sim_pipeline.py b/fatcat_scholar/sim_pipeline.py
index f99471f..95e5cad 100644
--- a/fatcat_scholar/sim_pipeline.py
+++ b/fatcat_scholar/sim_pipeline.py
@@ -149,7 +149,7 @@ class SimPipeline:
issue_item_metadata=full_issue["issue_item_metadata"],
),
)
- print(bundle.json())
+ print(bundle.json(exclude_none=True, sort_keys=True))
count += 1
if limit is not None and count >= limit:
break