The business drivers for undertaking a certification programme vary between customers, but often involve the need to meet some form or regulatory or industry compliance requirement. Whilst it may not be the initial driver, the fundamental goal of many of these activities is to reduce business risk by improving security posture.
We’ve become accustomed to dealing with auditors to assist our customers in undertaking certification initiatives. Unfortunately, just as in any industry, the quality of service, analysis and rigour that these entities employ can vary wildly.
Auditors commonly use automated tooling as an initial mechanism when assessing the compliance of a hosted infrastructure against the certification that their customer is attempting to achieve. This is a perfectly valid approach; automation increases consistency and reduces incidences of error as well as effort expended.
These tools will often expose a number of non-compliances, many of which are perfectly valid in context of the certification against which they are testing. Unfortunately, in our experience, there are often a large number of invalid non-compliance items on these reports.
Many of these non-compliance items relate to the use of outdated software, but auditor assumptions as to what version numbers actually mean can often be faulty.
Keeping software up to date
Without doubt, outdated and unpatched software is a huge security risk. Keeping software up to date is one of the most basic tenets of any mature security posture. As software becomes more prolific in our lives, so do the number of bugs and potential security vulnerabilities that come with it.
The Common Vulnerabilities and Exposures system is the industry standard mechanism for tracking known security flaws in software. Each vulnerability is assigned an ID by the Mitre corporation, which can be later referenced by software developers and operators to assess their security exposure. Software vendors will use this data as input into their development processes to associate bug fixes with the next release that addresses them.
As consequence, consumers of affected software can then reference the software change logs or documentation to identify at what point in time and what version the software was fixed for the particular vulnerability they are concerned with. The more recent the vulnerability, the more likely it is that a more recent version of the software will address it.
With this in mind, a reasonable expectation is thus that to attain the latest security fixes, one must update to the latest available version of the affected software. Many auditor tools make the same assumption; it is unfortunate that this line of thought is naive.
Problems with updating software
One of the problems associated with updating to the latest available version of software is that vendors generally do not just release new security fixes. They’ll often (almost always) couple in new features, some of which will also change the functionality of said software.
It is these changes in functionality that represent a problem for many organisations. Updating and patching software becomes a monolithic and arduous task because it is difficult to test, validate and orchestrate at even small scales. Changes to functionality of interfaces can potentially break production services. The unfortunate consequence is that patching becomes an irregular task, which only serves to increase the organisation’s length of exposure to publicly known security vulnerabilities. From a business perspective, uptime will generally take precedence over security.
As a matter of policy, Anchor updates and patches all customer services on a weekly basis. For a large, heterogeneous fleet of services such as ours, given the prior constraints, this may seem like a mammoth task. Whilst not trivial at scale, it is actually not at all a disruptive activity for our customers — In fact, this author can count on one hand the quantity of incidents that have been caused due to Anchor’s patching practices, and all have been resolved within the bounds of the associated maintenance window.
Anchor is able to achieve this record by taking advantage of others‘ work. Wherever possible, the software we make available to customers is that provided and packaged by the operating system (OS) vendor. We rely heavily on the OS vendors and their incredible track record for releasing stable, reliable updates that address security concerns.
For many OS vendors, there exists a policy that software within a major release should remain stable; this means that there should be no or minimal changes to packaged software for the lifetime of the release. Red Hat Enterprise Linux (RHEL) is one such example with a stellar reputation. RHEL has a published application compatibility policy.
Stable software, however, should not be stagnant software. To ensure that the OS vendor’s packaged software remains secure, they will backport security fixes into their products from upstream software projects.
From Red Hat’s backporting documentation:
- identify the fixes and isolate them from any other changes
- make sure the fixes do not introduce unwanted side effects
- apply the fixes to our previously released versions
Red Hat and many other vendors will publicly document their security advisories so that users can assess the vulnerability of the software they have been provided. It is also possible to search by CVE identifier.
By taking advantage of the OS vendor’s existing security practice, Anchor is able to regularly update our customer’s infrastructure whilst ensuring stability and security.
The caveat to this approach, however, is that the software versions listed as being in use register as vulnerable and outdated to many naive auditor tools.
Ignoring auditor advice
A common consequence of an initial audit is a request to update to the most recent version of the software identified, without regard for the current security practices. A list of CVE IDs that scanning tools associate with the currently used software will also be supplied.
Again, from Red Hat’s backporting documentation:
It is at this stage that a dispute must be filed with the findings; a tiresome but necessary process. Some auditors will record current security practice as a compensating control.
Most of the time, we are successful in such discussions, but there are many times when an auditor will not accept ‘No’ for an answer.
Why not just update to the latest version?
An obvious solution to appease a difficult auditor is to simply heed their advice and update the identified software to the latest version provided by the upstream project. This can in actuality be a terrible decision that may reduce overall security in the long term.
When you deviate from the software provided by the operating system vendor, you no longer receive the benefits of their maintenance. The burden falls onto you to regularly update, test and deploy the software, dealing with any changes in functionality along the way.
This practice is difficult at any scale. As consequence, it is very likely to become an infrequent activity that will result in less updates and greater length of exposure to any known vulnerabilities.
Anchor will always recommend to customers that they stick to OS vendor provided software packages where possible. It increases security, stability and reliability whilst reducing ongoing maintenance overhead.
Sticking with OS vendor packaged software is not the only option. One of the consequences of sticking to packaged software is that there will be times when you do actually require new features in current releases that aren’t available in the installed package — Remember that the OS vendor’s objective is to maintain stability, not increase functionality. In these circumstances, a trade off may be required to achieve the desired business outcome. There’s nothing wrong with deviating from established practice if you are aware of the risks and willing to accept responsibility.
Certification programmes can be taxing on all involved, but predominantly lead to positive outcomes once completed. Auditors, whilst not often popular, can be a necessary part of the process and are ultimately there to assist wherever they can — despite common wisdom, they’re not the enemy.
It’s important to have a good understanding of your own security practices and the risks you are willing to accept. With this knowledge in hand, you’ll be well prepared to have a productive conversation with your auditor and, with some luck, survive certification with your sanity intact.