blob: f885aea75bca1a121cc7b4c6f8e3a0d9f7ffd9bb (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
|
## Early Prototyping
### 2018-04-23
- fatcat as marshmallow+sqlalchemy+flask, with API client
- no refs, contibs, files, release contribs, containers, etc
- no extra_json
- sqlite
- laptop
- editgroup every 250 edits
/data/crossref/crossref-works.2018-01-21.badsample_5k.json
real 3m42.912s
user 0m20.448s
sys 0m2.852s
~22 lines per second
12.5 hours per million
~52 days for crossref (100 million)
target:
crossref (100 million) loaded in 48 hours
579 lines per second
this test in under 10 seconds
... but could be in parallel
same except postgres, via:
docker run -p 5432:5432 postgres:latest
./run.py --init-db --database-uri postgres://postgres@localhost:5432
./run.py --database-uri postgres://postgres@localhost:5432
API processing using 60-100% of a core. postgres 12% of a core;
docker-proxy similar (!). overall 70 of system CPU idle.
real 2m27.771s
user 0m22.860s
sys 0m2.852s
no profiling yet; need to look at database ops. probably don't even have any
indices!
## Rust Updates (2018-05-23)
Re-running with tweaked python code, 5k sample file, postgres 9.6 running locally (not in docker):
real 2m27.598s
user 0m24.892s
sys 0m2.836s
Using postgres and fatcat rust:
real 0m44.443s
user 0m25.288s
sys 0m0.880s
api_client about half a core; fatcatd 3x processes, about 10% each; postgres
very small.
a bit faster, basically maxing out CPU:
time cat /data/crossref/crossref-works.2018-01-21.badsample_5k.json | parallel -j4 --pipe ./fatcat_client.py --host-url http://localhost:9411 ic -
real 0m28.998s
user 1m5.304s
sys 0m3.420s
200 lines per second; within a factor of 3; can perhaps hit target with
non-python client?
python processes (clients) seem to be CPU limit in this case; all 4 cores
effectively maxed out.
running python again in parallel mode:
real 2m29.532s
user 0m47.692s
sys 0m4.840s
|
