Good configuration is a seriously hard problem. Worse, it just seems to get harder and harder as time goes on. Managing a mixture of old and new computers and making sure they all do what they're supposed to do becomes harder exponentially and is known to cause hair loss, high blood pressure, social deficiencies, and in some cases, death. Anchor's ninja-sysadmin Jamie Wilkinson has the answers; and he's managed to keep our computer's configurations humming along nicely. In this series of articles, he'll let you in on some secret server kung-fu that's keeping our network of systems manageable.
Wot Anchor's Doing With ''cfengine''
Dedicated servers that are maintained manually require a lot of effort, and as the number of systems grows, the amount of effort required grows at least linearly with them; probably more than linear, due to interactions between servers -- but without the manpower to keep up that maintenance, a growing network has a lot of entropy and quickly falls into disrepair.
Every seasoned sysadmin has their toolkit of scripts that have accumulated over the years, built out of necessity to combat the eventual heat death of control over the network: checklists of configuration tweaks that get applied to each new server, either manually or progressively more automatically as the admin gets older and values their time more. In recent years, people have started to codify as best practice what the old Bastards have known for a long time: computers like boring, repetitive work, and humans don't.
So what is it?
cfengine is one tool that gives the Pimply Faced Youth a head start when it comes to devising a way to apply your tweaks to a range of systems. It takes as input a description of a goal state for the systems, and the system itself, and then makes changes to get the system into the goal state. This technique is known as "convergence".
There are several papers and websites that do a much better job of introducing automatic systems administration that I ever could, so at this point I will refer you, kind reader, to some of my favourites:
These two papers on automated systems administration are definitely worth a read if you want to get a handle on the theory, or a treatise on how to design and build your new infrastructure from the ground up.
Computer Immunology from the author of cfengine, describing the system as a reactive organism, much like your own immune system.
How it works
cfengine is a collection of programs that share the responsibility of distributing configuration programs to servers, running the configuration program, and collecting statistics about the server for use in configuration programs.
The distribution service, cfservd runs on a central host, normally known as the <em>gold</em> server. cfservd can be configured to restrict access to certain IPs, or ranges of IPs, and to certain files on the gold server.
cfengines client server model uses authenticated, encrypted communication, similar to the way SSH works -- except that both ends of the connection must verify their identity to each other. This means that the administrator must get a copy of the host keys from each machine to the gold server, and the gold server's key to each client machine. There is a mechanism in cfengine to trust a connection by its IP address and then perform a key exchange, however.
The cfagent program does the work of running a cfengine program and performing the configuration changes. It can be run manually, or periodically from cron, or via the cfexecd service. cfexecd is a wrapper program that runs cfagent periodically and mails the output to the designated sysadmin. I have found it to have no benefit over cron, and so cfexecd is run with the -F option half-hourly from cron, which means cfagent is run once. So, every half hour, cron launches cfexecd, which launches cfagent, which reads its local copy of the configuration program. cfagent then tries to connect to the gold server, and if it can, checks to see if there are new versions of the configuration program available. If it cannot connect, it continues with the existing copy of the configuration program. If the connection succeeds (i.e. both cfagent and cfservd recognise each other and allow the connection to continue) and there are new programs to download, then it retreives them all and begins the process of checking and configuring the machine.
The configuration programs are broken up into sections describing various types of checks that can be made. The running process list can be checked for a program name, filesystems can be checked for free space, file and directory existence, permissions and ownership. Files can be edited to ensure they match a certain pattern. cfagent performs these checks according to the <em>actionsequence</em> defined in the configuration program.
A key part of cfengine is its <em>classes</em>. A class describes a property of the machine, such as 'it runs Debian' or 'it has at least one IDE disk' or 'it is a webserver'. Machines can belong to many classes, and a class may contain only one machine. Each machine has a hardcoded list of classes that it belongs to, and classes that are defined by the program, and classes that are dynamically set based on the behaviour of the program. A class that says 'restart the FTP server' may be defined if the configuration file for the FTP service is modified. The hostname of a machine is also a class; in this way machines can be grouped. A webserver class may be defined as a list of hostnames -- the hosts who should be configured as webservers.
So, cfengine is a powerful tool that can simplify the maintenance of a lot of similar (and sometimes, vastly different) machines by providing a simple way of distributing common configurations and applying those changes in an intuitive way. The core set of commands are flexible enough to tackle most administrative tasks and the concept of classes is useful in keeping configurations simple.
In future articles here on Anchor's Shipyard, I'll be going into more detail about the structure of the Anchor configurations, and delving into some useful techniques for some common, and some difficult, tasks that I've encountered since rolling out the first cfengine-controlled hosts at Anchor.
Author : Jamie Wilkinson