Occasionally I have a need to measure disk I/O in our virtual environments. This is usually because I have somebody telling me that we don’t have enough I/O to do what it is they want to do… often some sql that could be re-crafted to be easier on I/O. Usually they are wrong… but occasionally (like today) they are right on. Anyway, when I have this need, I turn to a few handy tools that have worked well for me int he past.

##iostat iostat is one of my favorite tools. I usually throw the -x flag at it and I get back something like this:

[root@host ~]# iostat -x
Linux 2.6.32-431.17.1.el6.x86_64 (host) 	07/07/2014 	_x86_64_	(2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.84    0.00    0.84    1.88    0.00   96.44

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               4.85    10.38    7.98    1.53   510.72    95.22    63.74     0.07    7.38   4.01   3.81
sdb               0.00     0.00    0.37    0.00     2.93     0.00     8.00     0.00    0.49   0.49   0.02
dm-0              0.00     0.00   10.99   11.90   473.82    95.18    24.86     0.38   15.77   1.60   3.66
dm-1              0.00     0.00    0.40    0.00     3.21     0.00     8.00     0.00    5.79   5.69   0.23

[root@host ~]# 

iostat -x n will continually run the iostat command every n seconds. I usually use one for five second intervals depending on what I am trying to get a feel for.

The key metric in iostat is %iowait - this is the amount of time spend waiting on I/O

##sar Another tool that I use from time to time is sar. It provides some alternate statistics alongside iowait. But again, %iowait is the key metric - the amount of time spent waiting on I/O

##Note: %iowait is the time that the kernel is doing nothing else, but waiting on I/O. Think about that for a minute. In other words a 100% CPU bound machine is going to show zero for %iowait, even if the system is also I/O bound.

iostat and sar are both part of the sysstat meta-package in the CentOS yum repos, and also available for Debian based systems via apt-get.

UPDATE - 2016-06-22: I have been using this a lot recently: `iostat -tx 60 2>&1 | tee iostat.log` It allows me to capture the time `-t` and output it to a log file for later perusal.

UPDATE - 2016-08-03: Also `vmstat -t 6`