A delicate balancing act

April 24, 2009 Technical, General

In our day to day use of computers, we try to forget as much about the boring, inane things and concentrate on the cool, useful or interesting things (like lolcats). Unfortunately for us, there are quite a few things which are boring and inane, but are also very important.

One of these things is IRQ balancing. Without going into excessive detail, IRQs (interrupt requests) are a mechanism that computer devices use to get the attention of the CPU when they need to do something. It’s like a little call for help, saying they need to get something done. You might find that your network card sends an interrupt when it has received a packet of information, or your hard disk sends an interrupt letting you know that it has some data to send to another part of the computer. Each of these things takes a bit of time away from the CPUs important tasks (like crunching numbers) so it is a thing the CPU likes to do as little as possible – just like you don’t like being interrupted by coworkers when concentrating on that difficult Sudoku puzzle.

For those of us fortunate enough to have multi-processor computers, we have an added advantage – we can give responsibility for some IRQs to one processor, and the rest to the other processor(s). This “balancing” of IRQs will ensure that we get the most efficient handling of those interrupts done and save more processing time for real work. It is possible to balance these interrupts by hand, but there is a handy package called IRQBalance that will do it all for us, automatically.

Here is an example of a well balanced system, which does a lot of passing network traffic between its interfaces:

[[email protected] ~]# cat /proc/interrupts
           CPU0       CPU1
  0:  217555173  176216279    IO-APIC-edge  timer
  1:          2          1    IO-APIC-edge  i8042
  4:        194          9    IO-APIC-edge  serial
  6:          2          1    IO-APIC-edge  floppy
  8:          1          0    IO-APIC-edge  rtc
  9:          0          1   IO-APIC-level  acpi
 12:          3          2    IO-APIC-edge  i8042
 15:         13         27    IO-APIC-edge  ide1
169:        350         26   IO-APIC-level  uhci_hcd:usb2, uhci_hcd:usb5
177:          0          0   IO-APIC-level  uhci_hcd:usb4
185:          0          2   IO-APIC-level  ehci_hcd:usb1
193:          0          0   IO-APIC-level  uhci_hcd:usb3
201:   18145851     130083   IO-APIC-level  aic79xx
209:          7          8   IO-APIC-level  aic79xx
217: 3543045844         70   IO-APIC-level  eth0
233:       9982 4165226804   IO-APIC-level  eth1
NMI:          0          0
LOC:  393789923  393790625
ERR:          0
MIS:          0

The most interesting parts of this listing are the lines referencing the network interfaces, eth0 and eth1. You can see from the cumulative IRQ count that the interrupts are evenly balanced between the two CPUs, thanks to IRQBalance.

So how does this interact with multi-core or hyperthreading CPUs? The most important thing to consider when balancing your IRQs is not the number of logical CPUs, but the number of cache domains available. The reasons behind this are quite techical but suffice it to say that you want to be balancing your IRQs over discrete cache domains.

This decision is automatically made for you by irqbalance, as you can see in this code snippet:

	/* On single core UP systems irqbalance obviously has no work to do */
	if (core_count<2)
		exit(EXIT_SUCCESS);
	/* On dual core/hyperthreading shared cache systems just do a one shot setup */
	if (cache_domain_count==1)
                one_shot_mode = 1;

Here, one_shot_mode means irqbalance will run once, balance the IRQs then exit and not continue to rebalance periodically. I ran into this problem when diagnosing a configuration management issue. We had some new Intel Core2Duo servers, and even though they had two cores per CPU, which for most purposes serves as a multi-processor environment, it was not enough for irqbalance. Configuration management would see the multi-core CPU and ensure the irqbalance service was running, but it would exit after the “one shot” run. Thus, on the next configuration management run, it would be started again, ad infinitum.

A similar situation arose recently, where I was performing some large data copies between machines over a network link. We were hitting disk and network limits as the storage subsystems were quite fast but I wanted to squeeze every drop of performance out of the machines. I realised I might be hitting IRQ saturation on one CPU if the RAID card and network interface were sending their interrupts to the same CPU. Indeed they were, and I rebalanced them by hand (check out /proc/irq/).

Lo and behold, the expected performance difference was not observed. I remembered my prior experience with the Core2Duo processors and irqbalance, which seemed to explain the result. It’s something that not many people think about, since it is quite a boring and inane subject, but nonetheless important to the efficient operation of the computer. Hopefully you’ve learned a little about it from reading this article!