Linux systems, at their core, are about managing processes and their access to resources. When a process needs data, it asks the kernel. The kernel then asks the storage subsystem. If that storage subsystem is slow, everything grinds to a halt. Understanding how to monitor disk I/O is critical for diagnosing performance bottlenecks, and iostat, iotop, and blktrace are your primary tools for this.
Let’s see iostat in action. Imagine you’re seeing some sluggishness in your application. You log into the server and run iostat -xz 5.
Linux 5.15.0-76-generic (my-server) 08/24/2023 _x86_64_ (4 CPU core(s))
avg-cpu: %user %nice %system %iowait %steal %guest %gnice %idle
10.50 0.00 5.20 25.80 0.00 0.00 0.00 58.50
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz %util
sda 0.50 5.20 8.00 20.80 0.00 0.10 0.00 1.90 30.50 45.20 0.20 15.50
sdb 2.50 10.50 40.00 105.00 0.00 0.20 0.00 1.80 25.80 38.50 0.30 22.10
avg-cpu: %user %nice %system %iowait %steal %guest %gnice %idle
12.10 0.00 6.10 24.90 0.00 0.00 0.00 56.90
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz %util
sda 0.60 5.50 9.60 22.00 0.00 0.15 0.00 2.60 32.10 48.90 0.25 16.20
sdb 2.30 11.20 36.80 115.50 0.00 0.25 0.00 2.10 27.00 40.10 0.35 24.50
Here, iostat -xz 5 gives us extended statistics (-x) for all devices (-z) every 5 seconds. The key metrics to watch are:
%iowait: This is the percentage of time the CPU spent waiting for I/O operations to complete. A high%iowait(like the 25.80% in the first block) is a dead giveaway that the CPU is bottlenecked by disk speed.r/s,w/s: Reads and writes per second. This tells you the rate of I/O operations.rkB/s,wkB/s: Kilobytes read and written per second. This tells you the throughput.r_await,w_await: The average time (in milliseconds) for I/O requests to be served. High values here mean individual operations are taking a long time.%util: The percentage of time the device was busy processing I/O requests. Consistently high%util(near 100%) indicates the disk is saturated.
iostat shows you the overall picture for each disk device. It’s great for identifying that a disk is a bottleneck, but not necessarily which process is causing it. That’s where iotop comes in.
Think of iotop as top for disk I/O. It shows you, in real-time, which processes are consuming the most disk bandwidth.
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
12345 be/4 root 10.54 M/s 0.00 K/s 0.00 % 0.00 % rsync -av /data/ /backup/
67890 be/4 appuser 5.20 M/s 0.00 K/s 0.00 % 0.00 % /usr/bin/python /opt/app/main.py
11223 be/4 root 0.00 K/s 2.10 M/s 0.00 % 0.00 % mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
Run sudo iotop. The most important columns are:
DISK READ,DISK WRITE: The current read/write bandwidth being used by the process.IO>: This is a percentage indicating how much of the process’s total CPU time is spent waiting on I/O. A highIO>value means the process is largely blocked by disk operations.
iotop is excellent for pinpointing runaway processes or identifying which application is hammering your storage. If iotop shows a process with very high DISK READ or DISK WRITE, and iostat shows high %iowait or %util, you’ve found your culprit.
However, sometimes you need to go deeper. You need to see the actual I/O requests being issued to the kernel. That’s where blktrace shines. It’s a powerful, low-level tool that captures block I/O events.
To use blktrace, you first need to start tracing a device, typically like this:
sudo blktrace -d /dev/sda -o sda_trace
This will create a file named sda_trace.0 (or similar) containing the raw trace data. Then, you can process this data with blkparse:
sudo blkparse -i sda_trace.0 -o sda_parse.txt
The output of blkparse is incredibly detailed. It shows every I/O request, including its type (read, write, flush), sector, size, process ID (PID), and timestamp.
8,0 0 0 0.000000000 10000 Q W 2048 128
8,0 0 1 0.000005000 10000 Q W 2048 128
8,0 0 2 0.000010000 10000 Q W 2048 128
8,0 0 3 0.000015000 10000 Q W 2048 128
8,0 0 4 0.000020000 10000 Q W 2048 128
8,0 0 5 0.000025000 10000 Q W 2048 128
8,0 0 6 0.000030000 10000 Q W 2048 128
8,0 0 7 0.000035000 10000 Q W 2048 128
8,0 0 8 0.000040000 10000 Q W 2048 128
8,0 0 9 0.000045000 10000 Q W 2048 128
8,0 0 10 0.000050000 10000 Q W 2048 128
8,0 0 11 0.000055000 10000 Q W 2048 128
8,0 0 12 0.000060000 10000 Q W 2048 128
8,0 0 13 0.000065000 10000 Q W 2048 128
8,0 0 14 0.000070000 10000 Q W 2048 128
8,0 0 15 0.000075000 10000 Q W 2048 128
8,0 0 16 0.000080000 10000 Q W 2048 128
8,0 0 17 0.000085000 10000 Q W 2048 128
8,0 0 18 0.000090000 10000 Q W 2048 128
8,0 0 19 0.000095000 10000 Q W 2048 128
8,0 0 20 0.000100000 10000 Q W 2048 128
8,0 0 21 0.000105000 10000 Q W 2048 128
8,0 0 22 0.000110000 10000 Q W 2048 128
8,0 0 23 0.000115000 10000 Q W 2048 128
8,0 0 24 0.000120000 10000 Q W 2048 128
8,0 0 25 0.000125000 10000 Q W 2048 128
8,0 0 26 0.000130000 10000 Q W 2048 128
8,0 0 27 0.000135000 10000 Q W 2048 128
8,0 0 28 0.000140000 10000 Q W 2048 128
8,0 0 29 0.000145000 10000 Q W 2048 128
8,0 0 30 0.000150000 10000 Q W 2048 128
8,0 0 31 0.000155000 10000 Q W 2048 128
8,0 0 32 0.000160000 10000 Q W 2048 128
8,0 0 33 0.000165000 10000 Q W 2048 128
8,0 0 34 0.000170000 10000 Q W 2048 128
8,0 0 35 0.000175000 10000 Q W 2048 128
8,0 0 36 0.000180000 10000 Q W 2048 128
8,0 0 37 0.000185000 10000 Q W 2048 128
8,0 0 38 0.000190000 10000 Q W 2048 128
8,0 0 39 0.000195000 10000 Q W 2048 128
8,0 0 40 0.000200000 10000 Q W 2048 128
8,0 0 41 0.000205000 10000 Q W 2048 128
8,0 0 42 0.000210000 10000 Q W 2048 128
8,0 0 43 0.000215000 10000 Q W 2048 128
8,0 0 44 0.000220000 10000 Q W 2048 128
8,0 0 45 0.000225000 10000 Q W 2048 128
8,0 0 46 0.000230000 10000 Q W 2048 128
8,0 0 47 0.000235000 10000 Q W 2048 128
8,0 0 48 0.000240000 10000 Q W 2048 128
8,0 0 49 0.000245000 10000 Q W 2048 128
8,0 0 50 0.000250000 10000 Q W 2048 128
8,0 0 51 0.000255000 10000 Q W 2048 128
8,0 0 52 0.000260000 10000 Q W 2048 128
8,0 0 53 0.000265000 10000 Q W 2048 128
8,0 0 54 0.000270000 10000 Q W 2048 128
8,0 0 55 0.000275000 10000 Q W 2048 128
8,0 0 56 0.000280000 10000 Q W 2048 128
8,0 0 57 0.000285000 10000 Q W 2048 128
8,0 0 58 0.000290000 10000 Q W 2048 128
8,0 0 59 0.000295000 10000 Q W 2048 128
8,0 0 60 0.000300000 10000 Q W 2048 128
8,0 0 61 0.000305000 10000 Q W 2048 128
8,0 0 62 0.000310000 10000 Q W 2048 128
8,0 0 63 0.000315000 10000 Q W 2048 128
8,0 0 64 0.000320000 10000 Q W 2048 128
8,0 0 65 0.000325000 10000 Q W 2048 128
8,0 0 66 0.000330000 10000 Q W 2048 128
8,0 0 67 0.000335000 10000 Q W 2048 128
8,0 0 68 0.000340000 10000 Q W 2048 128
8,0 0 69 0.000345000 10000 Q W 2048 128
8,0 0 70 0.000350000 10000 Q W 2048 128
8,0 0 71 0.000355000 10000 Q W 2048 128
8,0 0 72 0.000360000 10000 Q W 2048 128
8,0 0 73 0.000365000 10000 Q W 2048 128
8,0 0 74 0.000370000 10000 Q W 2048 128
8,0 0 75 0.000375000 10000 Q W 2048 128
8,0 0 76 0.000380000 10000 Q W 2048 128
8,0 0 77 0.000385000 10000 Q W 2048 128
8,0 0 78 0.000390000 10000 Q W 2048 128
8,0 0 79 0.000395000 10000 Q W 2048 128
8,0 0 80 0.000400000 10000 Q W 2048 128
8,0 0 81 0.000405000 10000 Q W 2048 128
8,0 0 82 0.000410000 10000 Q W 2048 128
8,0 0 83 0.000415000 10000 Q W 2048 128
8,0 0 84 0.000420000 10000 Q W 2048 128
8,0 0 85 0.000425000 10000 Q W 2048 128
8,0 0 86 0.000430000 10000 Q W 2048 128
8,0 0 87 0.000435000 10000 Q W 2048 128
8,0 0 88 0.000440000 10000 Q W 2048 128
8,0 0 89 0.000445000 10000 Q W 2048 128
8,0 0 90 0.000450000 10000 Q W 2048 128
8,0 0 91 0.000455000 10000 Q W 2048 128
8,0 0 92 0.000460000 10000 Q W 2048 128
8,0 0 93 0.000465000 10000 Q W 2048 128
8,0 0 94 0.000470000 10000 Q W 2048 128
8,0 0 95 0.000475000 10000 Q W 2048 128
8,0 0 96 0.000480000 10000 Q W 2048 128
8,0 0 97 0.000485000 10000 Q W 2048 128
8,0 0 98 0.000490000 10000 Q W 2048 128
8,0 0 99 0.000495000 10000 Q W 2048 128
8,0 0 100 0.000500000 10000 Q W 2048 128
8,0 0 101 0.000505000 10000 Q W 2048 128
8,0 0 102 0.000510000 10000 Q W 2048 128
8,0 0 103 0.000515000 10000 Q W 2048 128
8,0 0 104 0.000520000 10000 Q W 2048 128
8,0 0 105 0.000525000 10000 Q W 2048 128
8,0 0 106 0.000530000 10000 Q W 2048 128
8,0 0 107 0.000535000 10000 Q W 2048 128
8,0 0 108 0.000540000 10000 Q W 2048 128
8,0 0 109 0.000545000 10000 Q W 2048 128
8,0 0 110 0.000550000 10000 Q W 2048 128
8,0 0 111 0.000555000 10000 Q W 2048 128
8,0 0 112 0.000560000 10000 Q W 2048 128
8,0 0 113 0.000565000 10000 Q W 2048 128
8,0 0 114 0.000570000 10000 Q W 2048 128
8,0 0 115 0.000575000 10000 Q W 2048 128
8,0 0 116 0.000580000 10000 Q W 2048 128
8,0 0 117 0.000585000 10000 Q W 2048 128
8,0 0 118 0.000590000 10000 Q W 2048 128
8,0 0 119 0.000595000 10000 Q W 2048 128
8,0 0 120 0.000600000 10000 Q W 2048 128
8,0 0 121 0.000605000 10000 Q W 2048 128
8,0 0 122 0.000610000 10000 Q W 2048 128
8,0 0 123 0.000615000 10000 Q W 2048 128
8,0 0 124 0.000620000 10000 Q W 2048 128
8,0 0 125 0.000625000 10000 Q W 2048 128
8,0 0 126 0.000630000 10000 Q W 2048 128
8,0 0 127 0.000635000 10000 Q W 2048 128
8,0 0 128 0.000640000 10000 Q W 2048 128
8,0 0 129 0.000645000 10000 Q W 2048 128
8,0 0 130 0.000650000 10000 Q W 2048 128
8,0 0 131 0.000655000 10000 Q W 2048 128
8,0 0 132 0.000660000 10000 Q W 2048 128
8,0 0 133 0.000665000 10000 Q W 2048 128
8,0 0 134 0.000670000 10000 Q W 2048 128
8,0 0 135 0.000675000 10000 Q W 2048 128
8,0 0 136 0.000680000 10000 Q W 2048 128
8,0 0 137 0.000685000 10000 Q W 2048 128
8,0 0 138 0.000690000 10000 Q W 2048 128
8,0 0 139 0.000695000 10000 Q W 2048 128
8,0 0 140 0.000700000 10000 Q W 2048 128
8,0 0 141 0.000705000 10000 Q W 2048 128
8,0 0 142 0.000710000 10000 Q W 2048 128
8,0 0 143 0.000715000 10000 Q W 2048 128
8,0 0 144 0.000720000 10000 Q W 2048 128
8,0 0 145 0.000725000 10000 Q W 2048 128
8,0 0 146 0.000730000 10000 Q W 2048 128
8,0 0 147 0.000735000 10000 Q W 2048 128
8,0 0 148 0.000740000 10000 Q W 2048 128
8,0 0 149 0.000745000 10000 Q W 2048 128
8,0 0 150 0.000750000 10000 Q W 2048 128
8,0 0 151 0.000755000 10000 Q W 2048 128
8,0 0 152 0.000760000 10000 Q W 2048 128
8,0 0 153 0.000765000 10000 Q W 2048 128
8,0 0 154 0.000770000 10000 Q W 2048 128
8,0 0 155 0.000775000 10000 Q W 2048 128
8,0 0 156 0.000780000 10000 Q W 2048 128
8,0 0 157 0.000785000 10000 Q W 2048 128
8,0 0 158 0.000790000 10000 Q W 2048 128
8,0 0 159 0.000795000 100