One of Anchor’s more recent product offerings is VMware-based virtual private servers. As one of my colleagues has already detailed, we take extra measures to secure the VMware host server to reduce the possibility of a compromise.
Our VPS offering uses VMware ESX, which runs on bare metal and doesn’t have a host operating system. This isn’t the full story – according to documentation it boots a Redhat Enterprise Linux 3 system, then loads the vmkernel which is where the real work is done. One of the nice things about this approach is that there’s a userspace environment in which to run support software, like good monitoring components.
We ran into an odd problem recently with an ESX host server on a dedicated network segment, namely that we couldn’t view the console for VM guests. Nothing would happen for about 30 seconds, then the VMware Infrastructure Client (VIC) would report a connection failure.
Most people using VMware now have probably used the vanilla VMware Server once or twice. It’s pretty easy to understand, and because it runs on top of your usual OS, firewalling it as simple as opening holes for legitimate clients to connect to TCP port 902. That’s not the case with VMware ESX, and without reading the manuals it’s not immediately obvious what the problem is.
As it turns out, the VIC is attempting to connect to TCP port 903 on the host server. If one assumed that the VMware hypervisor is just a modified linux system (which is what it looks like, but isn’t quite) you should be able to see a listening port in the output of netstat -tnlp, but you can’t.
[root@miyuki root]# netstat -tnlp Proto Local Address Foreign Address State PID/Program name tcp 0.0.0.0:5666 0.0.0.0:* LISTEN 2684/nrpe tcp 127.0.0.1:32771 0.0.0.0:* LISTEN 1807/cimserver tcp 0.0.0.0:5988 0.0.0.0:* LISTEN 1807/cimserver tcp 0.0.0.0:5989 0.0.0.0:* LISTEN 1807/cimserver tcp 127.0.0.1:8005 0.0.0.0:* LISTEN 1652/webAccess tcp 0.0.0.0:902 0.0.0.0:* LISTEN 1598/xinetd tcp 0.0.0.0:199 0.0.0.0:* LISTEN 1489/snmpd tcp 0.0.0.0:8009 0.0.0.0:* LISTEN 1652/webAccess tcp 0.0.0.0:80 0.0.0.0:* LISTEN 1765/vmware-hostd tcp 0.0.0.0:8080 0.0.0.0:* LISTEN 1652/webAccess tcp 0.0.0.0:22 0.0.0.0:* LISTEN 1498/sshd tcp 127.0.0.1:8889 0.0.0.0:* LISTEN 1839/openwsmand tcp 0.0.0.0:2265 0.0.0.0:* LISTEN 1732/osirisd tcp 0.0.0.0:443 0.0.0.0:* LISTEN 1765/vmware-hostd
As a little more reading revealed, the VIC makes an extra connection to port 903 for console data. This marks a significant change from the earlier model of passing everything through the same connection, and the reason for such a change is unclear.
We’ll assume there’s some performance benefit to be had there. What we find more interesting/important is that port 903 is entirely “under the radar”, as it’s implemented in vmkernel. The other traffic on the box is subject to our standard iptables rules as far as we can tell, including port 902, which is used for a lot of other management-client to host-server interaction.
tcpdump is also oblivious to port 903, so we’re guessing VMware passes traffic through the netfilter stack when it’s deemed to be convenient or necessary. If we had a spare ESX host sitting around doing nothing, I’d be interested to see if the packet counters shown in the output of ifconfig are also affected.