AUTOMATE ALL THE THINGS!
Published August 30th, 2011 by David Basden
Illustration © Allie Brosh of Hyperbole and a Half
You can’t walk two metres down the street without someone going on about how cool and hip “The Cloud” is these days, being able to spin up hundreds of identical Linux VMs easily. Tools to build and configure lots of identical systems or VMs are plentiful.
But what if there is no “standard build” or even anything close, with different hardware, networking, software, distro, services, firewalls etc. every time, but you don’t want to spend all your time doing custom server builds and configuration?
Being a provider that specialises in customised hosting solutions, not only do most of our server builds have custom requirements, but we also have to configure lots of our own internal systems to deal with the eccentricities of each new server that we bring up. We have systems for monitoring, notifications, backups, fire-walling, accounting, scheduling of system updates, and that’s only a sampling of them.
We heavily use tools like puppet to automate things, but with almost every server being different, even configuring the tools designed to automate configuration starts taking serious time. For us, building a custom system and customising the systems that support it can take one of our sysadmins the best part of a day.
To solve this problem we’ve written some software that will take vague hand-waving from sales people[0] and turn it into a deterministic set of build steps that can be automated, without a sysadmin being involved at all. We now have heterogeneous server builds that are fully automated, and spun up in minutes rather than days. Rather than yet another bland VM copy, you get a beautiful and unique snowflake of awesome.
Even more fun, we’ve released them all as free software under the modified BSD license.
The first tool is called make-magic. Its job is to turn the high level requirements (I want a webserver and I want it blue) in to a detailed set of steps that can be automated (Find an online-paint shop. Buy some blue paint. Get it shipped to the data centre. etc.).
The core of make-magic can be guaranteed to always output a valid, correct list of steps for any set of input requirements. It even keeps track of which ones are done, which steps require others to be done first, and can let you know which ones can be done all at the same time.
There are some really interesting underlying problems that come up when you have to automatically generate a really specific, and provably correct set of steps from hundreds of thousands of permutations based on some high level requirements. We’ve beaten our heads against those brick walls, put it in a box, and posted it on github. You can find it at https://github.com/anchor/make-magic
Fun as it is, we still don’t want to have to go through and do those steps ourselves. This is where the second tool, mudpuppy, comes in.
mudpuppy is a Python based automation agent/client for make-magic. It allows you to write independent ‘modules’ in Python code, to automate items in a make-magic task. When run, mudpuppy will poll the make-magic server for tasks that need to be done, and will automate any items which it has a module for. Once the module has run successfully, mudpuppy will tell make-magic that the item has been completed, and will look for more work to do.
You can find mudpuppy on github at https://github.com/anchor/mudpuppy
mudpuppy isn’t the only tool that can be controlled by make-magic. make-magic has a simple, well defined JSON based HTTP API that mudpuppy uses, and which should be easy to talk to with most languages.
Earlier this month, Chris wrote about Orchestra, a set of tools for fast, reliable remote job execution on remote servers. mudpuppy is able to talk directly to Orchestra and get it to spin off jobs, get back the results, and use it to update information in make-magic. No extra parts required: They talk to each other out of the box.
As mentioned in Chris’s post, you can find Orchestra on github at https://github.com/anchor/orchestra
The combination of make-magic, mudpuppy and Orchestra (plus puppet, whatever else you might have lying around, or anything you’ve always wanted to try) give us the ability at the end of the day for us to build custom servers faster, reliably, and with ludicrously more flexibility than with any other systems that automate server builds that we’ve come across. Hopefully you’ll find them useful too.
If you’re interested in learning some more, Chris and I will be talking at the Sydney DevOps meetup on the 15th of September. There’s more details at http://devops.meetup.com/cities/au/sydney/
[0] Actually our “sales” people are pretty technical themselves, so they tend to wave their hands in the right direction.
mkdir /tmp/recovery_target
mount /tmp/recovery_fs /tmp/recovery_target