aboutsummaryrefslogtreecommitdiffstats
path: root/notes/import_timing_20180920.txt
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@robocracy.org>2018-09-24 18:04:54 -0700
committerBryan Newbold <bnewbold@robocracy.org>2018-09-24 18:04:54 -0700
commited78736b5d96a294c89b201c1d992bd30d809434 (patch)
treeb860457b9b1d09b8ecf52e512dddfcfd3e3e60ed /notes/import_timing_20180920.txt
parent94831f130e87286ad7394ec10f8e61d7ecd72adf (diff)
downloadfatcat-ed78736b5d96a294c89b201c1d992bd30d809434.tar.gz
fatcat-ed78736b5d96a294c89b201c1d992bd30d809434.zip
updated import timing
Diffstat (limited to 'notes/import_timing_20180920.txt')
-rw-r--r--notes/import_timing_20180920.txt30
1 files changed, 29 insertions, 1 deletions
diff --git a/notes/import_timing_20180920.txt b/notes/import_timing_20180920.txt
index 3bcf2a57..a57ffd77 100644
--- a/notes/import_timing_20180920.txt
+++ b/notes/import_timing_20180920.txt
@@ -61,8 +61,36 @@ Table sizes at this point:
"public"."container_edit" | 32 MB | 30 MB | 62 MB
"public"."container_ident" | 24 MB | 28 MB | 52 MB
+
+ relname | too_much_seq | case | rel_size | seq_scan | idx_scan
+----------------------+--------------+------+--------------+----------+----------
+ release_edit | 0 | OK | 6993084416 | 0 | 0
+ container_rev | 0 | OK | 54124544 | 0 | 0
+ creator_ident | 0 | OK | 266919936 | 0 | 0
+ creator_edit | 0 | OK | 363921408 | 0 | 0
+ work_rev | 0 | OK | 3327262720 | 0 | 0
+ creator_rev | 0 | OK | 388726784 | 0 | 0
+ work_ident | 0 | OK | 5128560640 | 0 | 0
+ work_edit | 0 | OK | 6993108992 | 0 | 0
+ container_ident | 0 | OK | 25092096 | 0 | 0
+ file_edit | 0 | OK | 598278144 | 0 | 0
+ container_edit | 0 | OK | 33857536 | 0 | 0
+ changelog | 0 | OK | 127549440 | 0 | 0
+ abstracts | -4714 | OK | 1706713088 | 0 | 4714
+ file_release | -13583 | OK | 401752064 | 0 | 13583
+ file_rev_url | -13583 | OK | 3832389632 | 0 | 13583
+ editgroup | -74109 | OK | 144277504 | 0 | 74109
+ release_contrib | -76699 | OK | 20357849088 | 0 | 76699
+ release_ref | -76700 | OK | 183009157120 | 3 | 76703
+ release_rev_abstract | -76939 | OK | 192102400 | 0 | 76939
+ release_rev | -77965 | OK | 47602647040 | 0 | 77965
+ file_ident | -100089 | OK | 438255616 | 3 | 100092
+ release_ident | -152809 | OK | 5128617984 | 0 | 152809
+ file_rev | -440780 | OK | 837705728 | 0 | 440780
+(23 rows)
+
Continuing imports:
zcat /srv/fatcat/datasets/2018-08-27-2352.17-matchcrossref.insertable.json.gz | pv -l | time parallel -j12 --round-robin --pipe ./fatcat_import.py import-matched -
-
=> HTTP response body: {"message":"duplicate key value violates unique constraint \"file_edit_editgroup_id_ident_id_key\""}
+