Blog Posts tagged ‘esx’

One of Anchor’s more recent product offerings is VMware-based virtual private servers. As one of my colleagues has already detailed, we take extra measures to secure the VMware host server to reduce the possibility of a compromise.

Our VPS offering uses VMware ESX, which runs on bare metal and doesn’t have a host operating system. This isn’t the full story – according to documentation it boots a Redhat Enterprise Linux 3 system, then loads the vmkernel which is where the real work is done. One of the nice things about this approach is that there’s a userspace environment in which to run support software, like good monitoring components.

We ran into an odd problem recently with an ESX host server on a dedicated network segment, namely that we couldn’t view the console for VM guests. Nothing would happen for about 30 seconds, then the VMware Infrastructure Client (VIC) would report a connection failure.

Most people using VMware now have probably used the vanilla VMware Server once or twice. It’s pretty easy to understand, and because it runs on top of your usual OS, firewalling it as simple as opening holes for legitimate clients to connect to TCP port 902. That’s not the case with VMware ESX, and without reading the manuals it’s not immediately obvious what the problem is.

As it turns out, the VIC is attempting to connect to TCP port 903 on the host server. If one assumed that the VMware hypervisor is just a modified linux system (which is what it looks like, but isn’t quite) you should be able to see a listening port in the output of netstat -tnlp, but you can’t.

[root@miyuki root]# netstat -tnlp
Proto Local Address       Foreign Address     State       PID/Program name
tcp   0.0.0.0:5666        0.0.0.0:*           LISTEN      2684/nrpe
tcp   127.0.0.1:32771     0.0.0.0:*           LISTEN      1807/cimserver
tcp   0.0.0.0:5988        0.0.0.0:*           LISTEN      1807/cimserver
tcp   0.0.0.0:5989        0.0.0.0:*           LISTEN      1807/cimserver
tcp   127.0.0.1:8005      0.0.0.0:*           LISTEN      1652/webAccess
tcp   0.0.0.0:902         0.0.0.0:*           LISTEN      1598/xinetd
tcp   0.0.0.0:199         0.0.0.0:*           LISTEN      1489/snmpd
tcp   0.0.0.0:8009        0.0.0.0:*           LISTEN      1652/webAccess
tcp   0.0.0.0:80          0.0.0.0:*           LISTEN      1765/vmware-hostd
tcp   0.0.0.0:8080        0.0.0.0:*           LISTEN      1652/webAccess
tcp   0.0.0.0:22          0.0.0.0:*           LISTEN      1498/sshd
tcp   127.0.0.1:8889      0.0.0.0:*           LISTEN      1839/openwsmand
tcp   0.0.0.0:2265        0.0.0.0:*           LISTEN      1732/osirisd
tcp   0.0.0.0:443         0.0.0.0:*           LISTEN      1765/vmware-hostd

As a little more reading revealed, the VIC makes an extra connection to port 903 for console data. This marks a significant change from the earlier model of passing everything through the same connection, and the reason for such a change is unclear.

We’ll assume there’s some performance benefit to be had there. What we find more interesting/important is that port 903 is entirely “under the radar”, as it’s implemented in vmkernel. The other traffic on the box is subject to our standard iptables rules as far as we can tell, including port 902, which is used for a lot of other management-client to host-server interaction.

tcpdump is also oblivious to port 903, so we’re guessing VMware passes traffic through the netfilter stack when it’s deemed to be convenient or necessary. If we had a spare ESX host sitting around doing nothing, I’d be interested to see if the packet counters shown in the output of ifconfig are also affected.

• PDF document describing ports used by VMware components