Evil hack to make arrow and SysReq keys work with a Dell iDRAC KVM and Linux desktop

March 18, 2011 Technical, General

So you’re trying to use the arrow keys in a remote server console with the iDRAC KVM. The keys not working and it’s driving you mad? Or perhaps that mission critical server is experiencing pain and you need to make use of the magic sysrq key to debug it?

Unfortunately, for a long long time, this has not worked if you are using Linux on your desktop.

Making it work

Just follow the steps below and you should be good to go.

  1. Download idrac.tar and unpack it:
  2. wget http://www.anchor.com.au/blog/wp-content/uploads/2011/03/idrac.tar
    tar xvf idrac.tar
  3. Build and install it:
  4. cd idrac
    make install

All done! It should take effect the next time you launch a virtual console.

If you want to make use of magic SysReq, make sure that it is disabled on your local workstation! (otherwise you may get a bit of a surprise…)

cat /proc/sys/kernel/sysrq

If the output is not zero, then edit /etc/sysctl.conf and make sure that it contains

kernel.sysrq = 0

then (as root) run sysctl -p

Under the hood

When you are logged into an iDRAC and you click on Launch Virtual Console, your browser downloads a dynamically generated JNLP file for a Java Web Start based KVM application. The Java Web Start tool (typically /usr/bin/javaws) processes the .jnlp file and proceeds to download the KVM software from the iDRAC and run it. The KVM software makes a connection back to the iDRAC on the standard VNC port (5900) (with the single use credentials that were provided to it in the .jnlp file).

At this point, you could easily be mistaken into thinking, Ah, VNC, that’s got to work well right. Such a simple thing and all“. Unfortunately you would be mightily wrong :(.

Whilst the iDRAC is using the standard VNC port, it appears that the implementation has been somewhat customised. Normally under the VNC protocol, a client will send key events to a remote server using standard X Window System keysym values to represent the key. It appears that, for some unknown reason, that the iDRAC client instead sends raw keycodes which it is able to read through the use of native code (this is why the problem isn’t seen on other platforms). Raw keycodes being constant can not be depended on by an application.

As you can guess by now, the keycodes in Linux have changed thus causing the assumptions to break. In the past, Linux distributions used the kbd keyboard driver. Around 2008, most Linux distributions switched to the much nicer evdev input driver system which unifies events from different device types, handles hot plug, and generally makes things sane. A consequence of this change however was  that some keyboard keycodes ended up changing. The end result is the broken arrow keys (plus other less frequently used keys).

The work around above adds more evil to the equation by making a small pre load library to change the behaviour of an X11 function which gets events to silently rewrite the keycode. It currently only remaps the arrow keys and print screen (used for SysReq). When using it, the arrow keys will cease to work in the KVM application menus but that isn’t such a worry. Anyway, have a read of the source code if you need to change anything. Good luck!