aboutsummaryrefslogtreecommitdiffstats
path: root/notes/tasks/2020-02-14_pdftrio.md
blob: e6f8d8e04c8e54161ccfe4b7497b731f0e6b6d12 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162

First end-to-end `pdf_trio` results!

## Source

Will use AIT partner #1830 (U Alberta) CDX as input. These are unique by
digest, about 100k.

    ArchiveIt-Collection-1830.download.cdx

## Testing/Prep

Versions/setup:

    sandcrawler: f613f69a40fcc9a445f21cadd35d7c36c8061db8
    => patched to 'auto' mode

    pdf_trio: 03bd3fdc15418462b2b1582e4f967f26ddcb43e2

    pdftrio: 'auto' mode

    uwsgi: 16x processes

    sudo docker run --rm -p 8501:8501 -e TF_XLA_FLAGS=--tf_xla_cpu_global_jit -e KMP_AFFINITY=granularity=fine,compact,1,0 -e KMP_BLOCKTIME=0 -e OMP_NUM_THREADS=24 -e TENSORFLOW_INTER_OP_PARALLELISM=1 -e TENSORFLOW_INTRA_OP_PARALLELISM=24 -v /srv/pdftrio//models/bert_models:/models/bert_model -v /srv/pdftrio//models/pdf_image_classifier_model:/models/image_model -v /srv/pdftrio//config/tfserving_models_docker.config:/models/tfserving_models.config -v /srv/pdftrio/config/tfserving_batch.config:/models/tfserving_batch.config --name pdftrio-tfserving tensorflow/serving --model_config_file=/models/tfserving_models.config --enable_batching=true --batching_parameters_file=/models/tfserving_batch.config

Basic testing::

    head -n100 /srv/sandcrawler/tasks/ArchiveIt-Collection-1830.download.cdx | parallel -j20 --pipe --linebuffer ./pdftrio_tool.py --pdftrio-host http://localhost:3939 -j0 classify-pdf-cdx - | jq .

    head -n100 /srv/sandcrawler/tasks/ArchiveIt-Collection-1830.download.cdx | parallel -j20 --pipe --linebuffer ./pdftrio_tool.py --kafka-mode --kafka-env qa --kafka-hosts wbgrp-svc263.us.archive.org,wbgrp-svc284.us.archive.org,wbgrp-svc285.us.archive.org --pdftrio-host http://localhost:3939 -j0 classify-pdf-cdx -
    => Running in kafka output mode, publishing to sandcrawler-qa.pdftrio-output


On the persist side:

    kafkacat -C -b wbgrp-svc263.us.archive.org -t sandcrawler-qa.pdftrio-output | head | jq .
    => looks fine

    ./sandcrawler_worker.py --kafka-hosts wbgrp-svc263.us.archive.org --env qa persist-pdftrio
    => Consuming from kafka topic sandcrawler-qa.pdftrio-output, group persist-pdftrio

Ah, don't forget, start persist before writing to topic! Or would need to reset
offsets to start.

Seems to be only a single pdftext instance running? Very low CPU

    head -n500 /srv/sandcrawler/tasks/ArchiveIt-Collection-1830.download.cdx | parallel -j40 -N1 --pipe --round-robin --linebuffer ./pdftrio_tool.py --kafka-mode --kafka-env qa --kafka-hosts wbgrp-svc263.us.archive.org,wbgrp-svc284.us.archive.org,wbgrp-svc285.us.archive.org --pdftrio-host http://localhost:3939 -j0 classify-pdf-cdx -

That is much better! CPU still not pegged, so maybe could do 50x processes? Lots of I/O wait. Blech.

Zero ("0") not getting persisted for any columns (fixed in sandcrawler/db.py)

`models_date` not getting set. Added `PDFTRIO_MODELS_DATE="2020-01-01"` to env. (TODO: ansible)

## Prod Run

    ./sandcrawler_worker.py --kafka-hosts wbgrp-svc263.us.archive.org --env prod persist-pdftrio

    time cat /srv/sandcrawler/tasks/ArchiveIt-Collection-1830.download.cdx | parallel -j40 -N1 --pipe --round-robin --linebuffer ./pdftrio_tool.py --kafka-mode --kafka-env prod --kafka-hosts wbgrp-svc263.us.archive.org,wbgrp-svc284.us.archive.org,wbgrp-svc285.us.archive.org --pdftrio-host http://localhost:3939 -j0 classify-pdf-cdx -

Worker CPU basically blocked on pdftotext, multiple 100% CPU. Presumably I/O
wait? Though not totally sure.

htop:

      PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command                                                
    17951 pdftrio    20   0 51756 12868  5856 R 90.1  0.0  0:06.61 pdftotext -nopgbrk -eol unix -enc UTF-8 /tmp/research-p
    17870 pdftrio    20   0 52004 12964  5684 R 87.4  0.0  0:08.61 pdftotext -nopgbrk -eol unix -enc UTF-8 /tmp/research-p
    13735 root       20   0 10.4G 3815M  4144 S 79.6  7.6 48h02:37 tensorflow_model_server --port=8500 --rest_api_port=850
    14522 pdftrio    20   0 2817M 1331M 16896 R 43.1  2.6  0:57.75 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    18027 pdftrio    20   0 49192 10692  6116 R 39.8  0.0  0:00.61 pdftotext -nopgbrk -eol unix -enc UTF-8 /tmp/research-p
    14518 pdftrio    20   0 2818M 1336M 16836 S 33.3  2.7  0:47.46 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    14504 pdftrio    20   0 2731M 1310M 13164 D 32.6  2.6  0:34.81 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    14526 pdftrio    20   0 2816M 1333M 16832 R 28.7  2.7  0:57.22 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    14500 pdftrio    20   0 2729M 1306M 13160 R 20.9  2.6  0:22.57 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    14492 pdftrio    20   0 2729M 1307M 13156 S 17.6  2.6  0:17.91 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    14508 pdftrio    20   0 2734M 1312M 14380 D 14.4  2.6  0:38.75 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    14496 pdftrio    20   0 2728M 1300M 13160 S 13.7  2.6  0:18.00 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    17314 sandcrawl  20   0 56668 18228  4304 D 13.7  0.0  0:02.31 perl /usr/bin/parallel -j40 -N1 --pipe --round-robin --
    14472 pdftrio    20   0 2725M 1283M 13136 S 12.4  2.6  0:05.69 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    14513 pdftrio    20   0 2730M 1309M 14300 S 11.1  2.6  0:40.32 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    14480 pdftrio    20   0 2725M 1291M 13144 S 10.4  2.6  0:08.77 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    14488 pdftrio    20   0 2725M 1294M 13152 S  9.8  2.6  0:08.18 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    14468 pdftrio    20   0 2717M 1271M 13088 S  6.5  2.5  0:02.42 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
    17411 sandcrawl  20   0  556M 53840 14936 S  6.5  0.1  0:01.57 python3 ./pdftrio_tool.py --kafka-mode --kafka-env qa -
    14530 pdftrio    20   0 2524M 1252M  3492 S  4.6  2.5  0:12.72 /usr/bin/uwsgi --ini /srv/pdftrio/src/uwsgi.ini
     7311 bnewbold   20   0 27716  5520  3128 R  3.9  0.0  0:41.59 htop
    17444 sandcrawl  20   0  552M 50456 14892 S  3.9  0.1  0:01.54 python3 ./pdftrio_tool.py --kafka-mode --kafka-env qa -
    18042 pdftrio    20   0 46068  6588  5328 R  3.3  0.0  0:00.05 pdftotext -nopgbrk -eol unix -enc UTF-8 /tmp/research-p
    18043 pdftrio    20   0     4     4     0 R  2.6  0.0  0:00.04
     2203 grobid     20   0 6334M  126M  4188 S  0.7  0.3  3h27:32 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -XX:MaxMetas
    17419 sandcrawl  20   0  619M  116M 15248 S  0.7  0.2  0:02.68 python3 ./pdftrio_tool.py --kafka-mode --kafka-env qa -
    17440 sandcrawl  20   0  578M 76948 15160 S  0.7  0.1  0:01.54 python3 ./pdftrio_tool.py --kafka-mode --kafka-env qa -
    13848 root       20   0     0     0     0 D  0.7  0.0  0:00.69 kworker/u60:1
    17443 sandcrawl  20   0  578M 76500 14912 S  0.7  0.1  0:01.74 python3 ./pdftrio_tool.py --kafka-mode --kafka-env qa -
    17414 sandcrawl  20   0  580M 77720 15036 S  0.0  0.2  0:01.77 python3 ./pdftrio_tool.py --kafka-mode --kafka-env qa -
    17432 sandcrawl  20   0  563M 61460 14976 S  0.0  0.1  0:01.59 python3 ./pdftrio_tool.py --kafka-mode --kafka-env qa -
    17442 sandcrawl  20   0  561M 53096 15240 S  0.0  0.1  0:01.47 python3 ./pdftrio_tool.py --kafka-mode --kafka-env qa -
    17433 sandcrawl  20   0  559M 57160 15176 S  0.0  0.1  0:01.52 python3 ./pdftrio_tool.py --kafka-mode --kafka-env qa -
    17431 sandcrawl  20   0  554M 50960 14892 S  0.0  0.1  0:01.37 python3 ./pdftrio_tool.py --kafka-mode --kafka-env qa -
    17413 sandcrawl  20   0  554M 52376 14920 S  0.0  0.1  0:01.57 python3 ./pdftrio_tool.py --kafka-mode --kafka-env qa -

dstat:

    ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
    usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
     32   9  22  37   0   0|   0    37M|  20M   12M|   0     0 |  35k   64k
     20   6  24  50   0   0|   0    20M|  30M 5662k|   0     0 |  27k   48k
     27   7  24  43   0   0|   0    26M|8712k 6289k|   0     0 |  21k  114k
     30   8  23  38   0   0|4096B   61M|  17M   20M|   0     0 |  31k   54k
     33   6  17  44   0   0|   0    32M|  14M 6384k|   0     0 |  27k   46k
     25   6  24  44   0   0|   0    19M|  18M   13M|   0     0 |  27k  179k
     40   6  19  35   0   0|8192B   25M|7855k 6661k|   0     0 |  31k   85k
     59   8  12  20   0   0|   0    39M|4177k   33M|   0     0 |  34k   64k
     34   4  17  44   0   0|   0    16M|7527k   11M|   0     0 |  22k   45k
     44   7  17  32   0   0|   0    30M|  20M  291k|   0     0 |  36k   62k

Create tmpfs:

    sudo mkdir -p /pdftrio-ramdisk
    #sudo mount -t tmpfs -o size=2g tmpfs /pdftrio-ramdisk
    sudo mount -t tmpfs -o size=6g tmpfs /pdftrio-ramdisk

add to pdftrio config env and restart:

    TEMP=/run/pdf_trio

Seems to have worked. Pretty much maxed CPU, may need to back-off parallelism. Doing more than  31/sec.

Errors were not getting encoded correctly:

    File "/fast/sandcrawler/python/sandcrawler/persist.py", line 331, in push_batch
      r['pdf_trio']['key'] = r['key']
    KeyError: 'pdf_trio'

Fixed in sandcrawler worker, and patched persist to work around this.

    time cat /srv/sandcrawler/tasks/ArchiveIt-Collection-1830.download.cdx | parallel -j30 -N1 --pipe --round-robin --linebuffer ./pdftrio_tool.py --kafka-mode --kafka-env prod --kafka-hosts wbgrp-svc263.us.archive.org,wbgrp-svc284.us.archive.org,wbgrp-svc285.us.archive.org --pdftrio-host http://localhost:3939 -j0 classify-pdf-cdx -

Wow, 30x parallelism waaaay less?

    time cat /srv/sandcrawler/tasks/ArchiveIt-Collection-1830.download.cdx | parallel -j30 -N1 --pipe --round-robin --linebuffer ./pdftrio_tool.py --kafka-mode --kafka-env prod --kafka-hosts wbgrp-svc263.us.archive.org,wbgrp-svc284.us.archive.org,wbgrp-svc285.us.archive.org --pdftrio-host http://localhost:3939 -j0 classify-pdf-cdx -

What changed? Confused. Load average was like 40.

Via kafka, as much as 69.71/sec! Errors?

Hrm, this whole `auto` thing. I am very skeptical. Should also do a run as `all`, -j20.

    Worker: Counter({'total': 1916, 'pushed': 1916})
    CDX lines pushed: Counter({'total': 1934, 'pushed': 1916, 'skip-parse': 18})

Hit some bugs, causing failure, but still seem to have processed a good chunk.

Switched to `all`, running a different batch:

    time cat /srv/sandcrawler/tasks/ArchiveIt-Collection-1914.download.cdx | parallel -j20 -N1 --pipe --round-robin --linebuffer ./pdftrio_tool.py --kafka-mode --kafka-env prod --kafka-hosts wbgrp-svc263.us.archive.org,wbgrp-svc284.us.archive.org,wbgrp-svc285.us.archive.org --pdftrio-host http://localhost:3939 -j0 classify-pdf-cdx -

After flag change, another batch in `all`:

    time cat /srv/sandcrawler/tasks/ArchiveIt-Collection-2566.download.cdx | parallel -j20 -N1 --pipe --round-robin --linebuffer ./pdftrio_tool.py --kafka-mode --kafka-env prod --kafka-hosts wbgrp-svc263.us.archive.org,wbgrp-svc284.us.archive.org,wbgrp-svc285.us.archive.org --pdftrio-host http://localhost:3939 -j0 classify-pdf-cdx -