Nginx disk io problem

Question

I run a small video streaming website and i've been noticing some problems with videos on my site.

TOP

top - 15:43:48 up 34 days,  1:22,  1 user,  load average: 37.80, 38.08, 38.38
Tasks: 184 total,   1 running, 183 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.5 us,  0.5 sy,  0.0 ni,  6.0 id, 85.1 wa,  0.0 hi,  7.8 si,  0.0 st
KiB Mem : 32730324 total,  1575916 free,  1470200 used, 29684208 buff/cache
KiB Swap: 33313788 total, 33308768 free,     5020 used. 30448240 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    3 root      20   0       0      0      0 S  16.9  0.0   5554:20 ksoftirqd/0
 2827 www-data  20   0  398696  84756   5548 S   6.0  0.3 213:14.78 nginx
 2828 www-data  20   0  324716  10900   5548 S   1.7  0.0  14:46.99 nginx
   40 root      20   0       0      0      0 S   0.3  0.0 148:03.47 ksoftirqd/5
  198 root       0 -20       0      0      0 S   0.3  0.0  11:16.66 kworker/0:+
 1086 root      20   0   44044   3668   2988 R   0.3  0.0   0:00.21 top
 2829 www-data  20   0  319180   6660   5396 S   0.3  0.0   1:51.37 nginx
27962 root      20   0  109956   9704   7836 S   0.3  0.0   0:00.53 exim4
    1 root      20   0  204648   5072   3644 S   0.0  0.0   1:47.36 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.16 kthreadd
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+
    7 root      20   0       0      0      0 S   0.0  0.0  31:06.23 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:01.11 migration/0
   10 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-dr+
   11 root      rt   0       0      0      0 S   0.0  0.0   1:16.83 watchdog/0
   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/0

IOTOP

Total DISK READ :      59.96 M/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:      59.96 M/s | Actual DISK WRITE:       8.84 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 2902 be/4 www-data  377.17 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2845 be/4 www-data 1508.66 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2864 be/4 www-data  754.33 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2832 be/4 www-data 1005.78 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2888 be/4 www-data  754.33 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2883 be/4 www-data  848.62 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2836 be/4 www-data 1382.94 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2849 be/4 www-data  628.61 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2841 be/4 www-data  660.04 K/s    0.00 B/s  0.00 %  99.58 % nginx: wo~er process
 2853 be/4 www-data  880.05 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2831 be/4 www-data  754.33 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2859 be/4 www-data 1382.94 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2835 be/4 www-data 1005.78 K/s    0.00 B/s  0.00 %  99.87 % nginx: wo~er process
 2851 be/4 www-data  628.61 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2834 be/4 www-data  754.33 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2839 be/4 www-data  880.05 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2860 be/4 www-data  251.44 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2892 be/4 www-data  628.61 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2843 be/4 www-data  880.05 K/s    0.00 B/s  0.00 %  99.99 % nginx: wo~er process
 2833 be/4 www-data  377.17 K/s    0.00 B/s  0.00 %  99.97 % nginx: wo~er process
 2856 be/4 www-data  754.33 K/s    0.00 B/s  0.00 %  99.97 % nginx: wo~er process

When i stop my website and i try to run disk speed this is what i have.

# dd if=/dev/zero of=tempfile bs=10M count=1024 conv=fdatasync,notrunc status=progress
10622074880 bytes (11 GB, 9.9 GiB) copied, 46.0385 s, 231 MB/s
1024+0 records in
1024+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 92.1228 s, 117 MB/s

So, current i can push up to 120 MB/s and iotop report i do 60 MB/s, why i'm high load IO issue if only i'm using half speed?

How i can tweak or fix this problem? i'm using nginx to serve mp4 videos.

Answer 1

Back when SSDs first came out, there was a common misconception that SSDs were always faster than HDDs. This wasn't true at all — you could easily do 125+ MB/s even in those times with an HDD, the only requirement being that you do sequential reading or writing. By contrast, sequential writes to the original SSDs were often slower than that, and the reading speeds often weren't all that impressive, either.

Long story short: your dd tests sequential throughput, whereas your nginx uses random access. HDDs have very limited capacity for random seek operations, on the order of well under 100 operations per second for a 7.2k RPM drive of any generation (compared to 10k+ IOPS in modern SSDs), which greatly limits total throughput if you depend on random filesystem access for your operation. Nowadays, 100 IOPS is so slow, if HDDs were "invented" today, they'd probably be dismissed as being out of touch with reality.

Because you're serving video, where sequential big-file access is more common than in other web applications, your numbers for real-life throughput are already not that bad — having 59.96 MB/s in real-life when a sequential benchmark shows 117 MB/s is already quite impressive. The solution may be to upgrade to more HDDs, have HDDs with faster RPM for more IOPS, or move to an SSD.