VMware ESX Server Security Lockdown


We've recently deployed a machine based on ESX as part of our master plan to take on the world with a VPS (Virtual Private server) product. There are a number of really good reasons for moving towards virtualisation; some of these include:

  • Reduction of total number of physical hardware components. This means less hardware failures, less failures mean less outages.
  • Reduction in physical number of machines
  • Increased over all data centre density
  • Reduction of over all power consumption (see why power is more important than space )

  • Much easier to rapidly deploy new services remotely
  • Easier to upgrade individual services (RAM, CPU and HDD upgrades can all be done at a click of a button)

As part of deploying any new infrastructure we need to give extensive thought to various aspects of the service including scalability, security, reliability, robustness and performance. A large percentage of our time is spent ensuring that we have absolute confidence in all of these aspects.

This article discusses many of the techniques we applied to securing ESX to ensure it meets our high standards.

Overview of our modifications

VMware ESX ships on a customised version of Red Hat Enterprise Linux 3. This is an operating system Anchor Systems has had plenty of experience with as we have deployed hundreds of dedicated servers using RHEL3.

We have made a number of changes to the system that we know will tighten the security up to a level that we are comfortable with. Combined with putting this behind our high availability network firewall pair we develop a "defense in depth" approach towards security.

Additionally we have configured and installed host integrity monitoring onto the system to track any changes to software or network services. This level of system monitoring is not provided by VMware under any add-on for ESX.

We use a firewall rule generation system that takes a set of template rules and custom include files. This gives us the flexibility to manage a large number of machines but still have unique granular firewall policies on each host. This also provides an additional feature to firewall all services off and then only allow services be used only from specific remote hosts.

Further to this, we have also configured TCP Wrapper to only allow authorised hosts access to certain VMware ESX services. When combined with the additional host based firewall policies we gain an extra measure of security. The firewall uses Linux's Iptables which provides stateful packet inspection filtering and known to be a mature and very reliable method of firewalling.

Physical Network Separation

We have configured our VMware ESX servers to make use of multiple physical network adapters. We then utilise different VLANs for various types of traffic.

As we currently only use host based storage, we have no need to configure a VMKernel interface. We have one physical interface for our management network. Additionally that physical network also carries our backup traffic off the ESX server with rate limiting. This ensures our management access is always available, even when bulk data copying is taking place during a backup window.

The second physical adapter carries all virtual machine traffic. As we have a number of virtual machines, some are under our different support packs. Therefore we have a number of tagged VLANs configured. This is used separate some types of traffic out from another and allows us to monitor a subset of traffic using our Network Security Monitoring infrastructure.

If we were to use NFS or iSCSI transport, we would add an additional physical network adapter and configure it to carry all VMkernel traffic. This would be connected to a secured VLAN. All VMkernel traffic is unencrypted, hence the need for a secure network to carry that traffic.

Firewall Changes

The default VMware ESX firewall does not ship with any host based restrictions. The firewall is implemented via the Virtual Infrastructure Client using Linux iptables which can manage the ruleset very basically. The biggest downfall we found was that the Virtual Infrastructure Client provided by VMware does not allow for host-based exceptions. For this reason we threw the default firewall management tool out the window and went with something we know and trust.

Introducing: Filtergen!

You can read more about how we manage firewalls in this article dedicated/SecurityIsFun

In addition to our default filtergen rule set that is rolled out, we have custom inbound rules that are configured in /etc/filtergen/input.d/vmware-esx:

# VI3 client login
source {
        include anchor-priv.acl
} proto { tcp udp } dport vmware-authd accept;

# ESX wants svrloc open
source {
        include anchor-priv.acl
} proto { tcp udp } dport svrloc accept;

# cimserver - ESX component
source {
        include anchor-priv.acl
} proto { tcp } dport { 427 5988 5989 } accept;

In the above configuration fragment we are limiting access to all of those VMware ESX services to only Anchor's management network.

Because we use Filtergen we can simplify our firewall rule sets by using included files to source in lists of IP address.

TCP Wrapper

A general overview of TCP Wrapper can be found at this Wikipedia article on TCP Wrapper

We use TCP Wrappers to limit which hosts based on their reverse DNS entries can access which services on the VMware ESX server.

This is used in conjunction with firewall restrictions against each specific service, with an exception for only authorised hosts. This approach gives a multi-layered security system.

For example if the host based firewall was down due to a system error, our TCP Wrapper configuration would still deny access to the services configured.

Here is a sample of two entries from the /etc/hosts.allow file with the specific VMware ESX services in it.

vmware-authd: supersecure.anchoroffice.anchor.com.au
sshd: supersecure.anchoroffice.anchor.com.au

You must remember to ensure you have the required deny entries in /etc/hosts.deny


By default we always deny all services and hosts, and configure ACLs to allow only specific hosts to the required service. The above rules will allow access for any hosts originating from supersecure.anchoroffice.anchor.com.au. This is determined by the DNS resolvers on the given machine.

SSH Changes

VMware ESX does not allow root SSH logins by default. To enable this you will need to login from the console when you have finished your install and modify the SSH daemon configuration.

You do this buy editing /etc/ssh/sshd_config and changing the line

PermitRootLogin no


PermitRootLogin without-password

and adding a new line below it

PermitEmptyPasswords no

This will allow root to log in using SSH public key authenication Save and exit the configuration and restart the SSH daemon.

service sshd restart

It is advisable to add a user account while you are working on the console, and grant sudo access. This way you can add your SSH keys to the system remotely.

Additional information is available on how we secure SSH logins in this Securing_Remote_Access_via_SSH_and_Server_Security article.

Osiris Host Integrity Monitoring

Osiris is a host integrity monitoring system that can be used to monitor changes to a network of hosts over time and report those changes back to the administrator(s). Currently, this includes monitoring any changes to the file systems. Osiris takes periodic snapshots of the file system and stores them in a database. These databases, as well as the configurations and logs, are all stored on a central management host. When changes are detected, Osiris will log these events to the system log and optionally send an email to an administrator.

In addition to files, Osiris has the ability to monitor of other system information including user lists, group lists, and kernel modules or extensions.

Our dedicated/SecurityIsFun page covers Osiris in more detail.

Other pages in similar categories