aboutsummaryrefslogtreecommitdiffstats
path: root/notes/import_timing_20180923.txt
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@robocracy.org>2018-09-25 15:44:39 -0700
committerBryan Newbold <bnewbold@robocracy.org>2018-09-25 15:44:39 -0700
commit186c03c2b9e1b08e4298383fc6593d1b2c3a5dd8 (patch)
treeb116678710d9a69df023f7dc90525d7563506891 /notes/import_timing_20180923.txt
parentc1391060cc4575365f81fc674e35d8c182ccde83 (diff)
downloadfatcat-186c03c2b9e1b08e4298383fc6593d1b2c3a5dd8.tar.gz
fatcat-186c03c2b9e1b08e4298383fc6593d1b2c3a5dd8.zip
start organizing notes into subdirectories
Diffstat (limited to 'notes/import_timing_20180923.txt')
-rw-r--r--notes/import_timing_20180923.txt39
1 files changed, 0 insertions, 39 deletions
diff --git a/notes/import_timing_20180923.txt b/notes/import_timing_20180923.txt
deleted file mode 100644
index f8814f3d..00000000
--- a/notes/import_timing_20180923.txt
+++ /dev/null
@@ -1,39 +0,0 @@
-
- 105595.18user 3903.65system 15:59:39elapsed 190%CPU (0avgtext+0avgdata 458836maxresident)k
- 71022792inputs+327828472outputs (176major+31149593minor)pagefaults 0swaps
-
- real 959m39.521s
- user 1845m10.392s
- sys 70m33.780s
-
-Did I get the same error again? I'm confused:
-
- HTTP response body: {"message":"number of parameters must be between 0 and 65535\n"}
- (but not in all threads)
-
-Yes, ugh, because 50*2500 can be over (it's not just individual large releases,
-they come in big batches).
-
-But:
-
- select count(id) from release_ident; => 70006121
-
-A lot, though not 72 million like last time, hrm. I'm... going to move ahead I
-guess.
-
-"Processed 4440850 lines, inserted 3509600, updated 0."
- => implies 79029915 records
-
- time zcat /srv/fatcat/datasets/ia_papers_manifest_2018-01-25.matched.json.gz | pv -l | time parallel -j12 --round-robin --pipe ./fatcat_import.py import-matched --no-file-update -
- Processed 530750 lines, inserted 435239, updated 0. (etc)
- Command exited with non-zero status 1
- 15121.47user 676.49system 2:23:52elapsed 183%CPU (0avgtext+0avgdata 70076maxresident)k
- 127760inputs+3477184outputs (116major+475489minor)pagefaults 0swaps
-
- real 143m52.681s
- user 252m31.620s
- sys 11m21.608s
-
- zcat /srv/fatcat/datasets/2018-08-27-2352.17-matchcrossref.insertable.json.gz | pv -l | time parallel -j12 --round-robin --pipe ./fatcat_import.py import-matched -
-
- (running...)