A delicate balancing act
In our day to day use of computers, we try to forget as much about the boring, inane things and concentrate on the cool, useful or interesting things (like lolcats). Unfortunately for us, there are quite a few things which are boring and inane, but are also very important.
One of these things is IRQ balancing. Without going into excessive detail, IRQs (interrupt requests) are a mechanism that computer devices use to get the attention of the CPU when they need to do something. It’s like a little call for help, saying they need to get something done. You might find that your network card sends an interrupt when it has received a packet of information, or your hard disk sends an interrupt letting you know that it has some data to send to another part of the computer. Each of these things takes a bit of time away from the CPUs important tasks (like crunching numbers) so it is a thing the CPU likes to do as little as possible – just like you don’t like being interrupted by coworkers when concentrating on that difficult Sudoku puzzle.
For those of us fortunate enough to have multi-processor computers, we have an added advantage – we can give responsibility for some IRQs to one processor, and the rest to the other processor(s). This “balancing” of IRQs will ensure that we get the most efficient handling of those interrupts done and save more processing time for real work. It is possible to balance these interrupts by hand, but there is a handy package called IRQBalance that will do it all for us, automatically.
Here is an example of a well balanced system, which does a lot of passing network traffic between its interfaces:
[[email protected] ~]# cat /proc/interrupts CPU0 CPU1 0: 217555173 176216279 IO-APIC-edge timer 1: 2 1 IO-APIC-edge i8042 4: 194 9 IO-APIC-edge serial 6: 2 1 IO-APIC-edge floppy 8: 1 0 IO-APIC-edge rtc 9: 0 1 IO-APIC-level acpi 12: 3 2 IO-APIC-edge i8042 15: 13 27 IO-APIC-edge ide1 169: 350 26 IO-APIC-level uhci_hcd:usb2, uhci_hcd:usb5 177: 0 0 IO-APIC-level uhci_hcd:usb4 185: 0 2 IO-APIC-level ehci_hcd:usb1 193: 0 0 IO-APIC-level uhci_hcd:usb3 201: 18145851 130083 IO-APIC-level aic79xx 209: 7 8 IO-APIC-level aic79xx 217: 3543045844 70 IO-APIC-level eth0 233: 9982 4165226804 IO-APIC-level eth1 NMI: 0 0 LOC: 393789923 393790625 ERR: 0 MIS: 0
The most interesting parts of this listing are the lines referencing the network interfaces, eth0 and eth1. You can see from the cumulative IRQ count that the interrupts are evenly balanced between the two CPUs, thanks to IRQBalance.
So how does this interact with multi-core or hyperthreading CPUs? The most important thing to consider when balancing your IRQs is not the number of logical CPUs, but the number of cache domains available. The reasons behind this are quite techical but suffice it to say that you want to be balancing your IRQs over discrete cache domains.
This decision is automatically made for you by irqbalance, as you can see in this code snippet:
/* On single core UP systems irqbalance obviously has no work to do */ if (core_count<2) exit(EXIT_SUCCESS); /* On dual core/hyperthreading shared cache systems just do a one shot setup */
if (cache_domain_count==1) one_shot_mode = 1;
Here, one_shot_mode means irqbalance will run once, balance the IRQs then exit and not continue to rebalance periodically. I ran into this problem when diagnosing a configuration management issue. We had some new Intel Core2Duo servers, and even though they had two cores per CPU, which for most purposes serves as a multi-processor environment, it was not enough for irqbalance. Configuration management would see the multi-core CPU and ensure the irqbalance service was running, but it would exit after the “one shot” run. Thus, on the next configuration management run, it would be started again, ad infinitum.
A similar situation arose recently, where I was performing some large data copies between machines over a network link. We were hitting disk and network limits as the storage subsystems were quite fast but I wanted to squeeze every drop of performance out of the machines. I realised I might be hitting IRQ saturation on one CPU if the RAID card and network interface were sending their interrupts to the same CPU. Indeed they were, and I rebalanced them by hand (check out /proc/irq/).
Lo and behold, the expected performance difference was not observed. I remembered my prior experience with the Core2Duo processors and irqbalance, which seemed to explain the result. It’s something that not many people think about, since it is quite a boring and inane subject, but nonetheless important to the efficient operation of the computer. Hopefully you’ve learned a little about it from reading this article!