VMware ESX Guest Disk IO

Published April 6th, 2009 by Paul De Audney

Knowing the state of your disk IO latency in VMware ESX can help you pre-empt performance & capacity issues before the occur. There are a few guidelines you should keep in mind. These notes are directed towards people using directly attached storage.

  • Write latency should be 0, because you have that fancy battery backed controller caching writes, right?
  • Read latency should be under 8ms.
  • Use the smallest stripe size possible for your RAID array setting. This helps keep random IO performance acceptable at the cost of some sequential performance.
  • Do not virtualise very heavy random IO workloads on shared arrays, other guest VMs wont like you for it.
  • Unless you have a very compelling reason not too, use RAID 10.

Some other notes, specific to Linux guests are:

  • Mount file systems with noatime and nodiratime, this will help reduce random IO.
  • Allocate enough memory to have some buffers.
  • Do anything possible to stop your VM swapping heavily (see point above).

As with any system, having great monitoring and performance trending allows for you to have an excellent overview of your infrastructure. Even if you don’t have external systems for performance trending, the VMware Infrastructure client with a few tweaks will display the data you want to see.

  1. Login to the VI Client.
  2. Click on an object in the left navigation tree.
  3. Click on the performance tab at the top of the main display pane.
  4. Click the “Change Chart Options” button
  5. Select the Disk chart option from the left expanding menu.
  6. Now change the counters, pick the Latency counters and Number counters, un-ticking the KBps  counters.
  7. Save the chart settings as disk-latency.

Now you can view in real time what is happening with your disk IO on the VMware ESX server. If you are more familiar with using a command line and Linux, you can SSH in to the ESX COS and use the command esxtop to view disk performance information.

  1. Launch esxtop (as root)
  2. Press “v”
  3. Press “s” and then “1″

Now you can see the per VM disk usage counters, with a 1 second sample period.

These rules of thumb are also applicable to Xen and Hyper-V.

Tags: , , , , , ,
Posted in FTW

 Leave a comment

0
Comments

Tracing I/O usage on Linux

Published February 19th, 2009 by oliver

I/O subsystems are a whole industry of their own, and many libraries could be (and probably have been) written on the subject already. The particular sub-topic I’m talking about today is when you are faced with a machine that you suspect to be suffering from heavy I/O load, and you want to find the culprit.

Sadly, this is an area that Windows has the upper hand. You can quite easily using the Performance Monitor determine which process is using the largest chunk of your disk I/O. On Linux things can be a little harder, however not all is lost.

If you are fortunate enough to be experiencing the problem on a machine running at least a 2.6.20 kernel and with Python 2.5 or later available, you can run IOTop. This prints out I/O usage data in a similar format to the standard “top” command, and it looks something like this:

IOTop Output

IOTop Output - picture sourced from http://guichaz.free.fr/iotop/iotop_big.png

Sadly at the time I needed to diagnose I/O, the machine I was using had neither a 2.6.20 kernel nor Python 2.5 so I was forced into seeking other methods to trace the I/O. Cue Blktrace. This hooks into the kernel’s debug filesystem to gather I/O stats and presents a fairly raw trace of what’s going on. You can download the source from here or find RPM packages for recent RHEL at the RPMForge Repository.

While it is possible to use blktrace directly, there is also a helper script btrace which shortcuts a lot of the most commonly used options and output formatting. You will need to mount debugfs on /sys/kernel/debug then you are ready to roll!

root@blarg:~# mount -t debugfs none /sys/kernel/debug
root@blarg:~# btrace /dev/sda
  8,0    0        1     0.000000000  2884  A   W 60060711 + 8 <- (8,1) 60060648
  8,0    0        2     0.000000244  2884  Q   W 60060711 + 8 [kjournald]
  8,0    0        3     0.000005278  2884  G   W 60060711 + 8 [kjournald]
  8,0    0        4     0.000006933  2884  P   N [kjournald]
  8,0    0        5     0.000007515  2884  I   W 60060711 + 8 [kjournald]
  8,0    0        6     0.000010068  2884  A   W 60093063 + 8 <- (8,1) 60093000
  8,0    0        7     0.000010263  2884  Q   W 60093063 + 8 [kjournald]
  8,0    0        8     0.000011588  2884  G   W 60093063 + 8 [kjournald]
  8,0    0        9     0.000012072  2884  I   W 60093063 + 8 [kjournald]

OK so there’s not much I/O happening on my workstation, but on the machine I was diagnosing recently, the output of btrace spewed out hundreds of lines per second, many of them referring to a process running the mutt mail program. It turned out one of the users had approximately 90,000 emails in one folder that mutt was constantly rescanning since the machine didn’t have a recent enough version of mutt to support header caching.

The emails were archived away and the I/O problem was resolved. Whereas previously we could only guess at what was causing the I/O load, blktrace squarely points the finger at the problem process. On later machines IOTop would have been even more straightforward. Both are valuable additions to the sysadmin toolkit.

Tags: , , , ,
Posted in FTW

 Leave a comment

1
Comment