The software, which has been released as open source on github as was discussed in brief in the previously two blog posts The automation waltz and Automate all the things.

The event will be held at 7pm, next Thursday, 15th September at the Orient in Sydney and will give you the fantastic opportunity to have a beer and chat with the authors of the software as well as like minded people with both developer and systemadmin backgrounds.

Comprehensive details as well as registration for the meetup can be found at September DevOps Meetup

Look forward to seeing you then!

Illustration © Allie Brosh of Hyperbole and a Half

You can’t walk two metres down the street without someone going on about how cool and hip “The Cloud” is these days, being able to spin up hundreds of identical Linux VMs easily. Tools to build and configure lots of identical systems or VMs are plentiful.

But what if there is no “standard build” or even anything close, with different hardware, networking, software, distro, services, firewalls etc. every time, but you don’t want to spend all your time doing custom server builds and configuration?

Being a provider that specialises in customised hosting solutions, not only do most of our server builds have custom requirements, but we also have to configure lots of our own internal systems to deal with the eccentricities of each new server that we bring up. We have systems for monitoring, notifications, backups, fire-walling, accounting, scheduling of system updates, and that’s only a sampling of them.

We heavily use tools like puppet to automate things, but with almost every server being different, even configuring the tools designed to automate configuration starts taking serious time. For us, building a custom system and customising the systems that support it can take one of our sysadmins the best part of a day.

To solve this problem we’ve written some software that will take vague hand-waving from sales people[0] and turn it into a deterministic set of build steps that can be automated, without a sysadmin being involved at all. We now have heterogeneous server builds that are fully automated, and spun up in minutes rather than days. Rather than yet another bland VM copy, you get a beautiful and unique snowflake of awesome.

Even more fun, we’ve released them all as free software under the modified BSD license.

The first tool is called make-magic. Its job is to turn the high level requirements (I want a webserver and I want it blue) in to a detailed set of steps that can be automated (Find an online-paint shop. Buy some blue paint. Get it shipped to the data centre. etc.).

The core of make-magic can be guaranteed to always output a valid, correct list of steps for any set of input requirements. It even keeps track of which ones are done, which steps require others to be done first, and can let you know which ones can be done all at the same time.

There are some really interesting underlying problems that come up when you have to automatically generate a really specific, and provably correct set of steps from hundreds of thousands of permutations based on some high level requirements. We’ve beaten our heads against those brick walls, put it in a box, and posted it on github. You can find it at https://github.com/anchor/make-magic

Fun as it is, we still don’t want to have to go through and do those steps ourselves. This is where the second tool, mudpuppy, comes in.

mudpuppy is a Python based automation agent/client for make-magic. It allows you to write independent ‘modules’ in Python code, to automate items in a make-magic task. When run, mudpuppy will poll the make-magic server for tasks that need to be done, and will automate any items which it has a module for. Once the module has run successfully, mudpuppy will tell make-magic that the item has been completed, and will look for more work to do.

You can find mudpuppy on github at https://github.com/anchor/mudpuppy

mudpuppy isn’t the only tool that can be controlled by make-magic. make-magic has a simple, well defined JSON based HTTP API that mudpuppy uses, and which should be easy to talk to with most languages.

Earlier this month, Chris wrote about Orchestra, a set of tools for fast, reliable remote job execution on remote servers. mudpuppy is able to talk directly to Orchestra and get it to spin off jobs, get back the results, and use it to update information in make-magic. No extra parts required: They talk to each other out of the box.

As mentioned in Chris’s post, you can find Orchestra on github at https://github.com/anchor/orchestra

The combination of make-magic, mudpuppy and Orchestra (plus puppet, whatever else you might have lying around, or anything you’ve always wanted to try) give us the ability at the end of the day for us to build custom servers faster, reliably, and with ludicrously more flexibility than with any other systems that automate server builds that we’ve come across. Hopefully you’ll find them useful too.

If you’re interested in learning some more, Chris and I will be talking at the Sydney DevOps meetup on the 15th of September. There’s more details at http://devops.meetup.com/cities/au/sydney/

[0] Actually our “sales” people are pretty technical themselves, so they tend to wave their hands in the right direction.

In trying to solve this problem, we had assessed tools like mcollective, but came to the conclusion that they were inappropriate for our environment.  mcollective in particular was removed from consideration as it was designed for a more homogeneous environment than the one here at Anchor.

When we realised we needed a different solution, a few of us gathered around a whiteboard and started enumerating our requirements.  The result was Orchestra, which we’re releasing today under the BSD License.

Orchestra is a suite for managing the reliable execution of tasks over a number of hosts. It is intended to be simple and secure, leaving more complicated coordination and tasks to tools better suited for those jobs.

Because everybody loves this web development stuff, it was developed to provide an interface that operates asynchronously in relation to the
execution of the work being done, allowing for cheap and easy polling of job state.

And last, but not least, because having critical system services depend on potentially fragile language interpreters or their libraries is generally a really bad idea for reliability, it was completely implemented in Go - a type-safe static language that compiles to native code and includes many features derived from dynamic languages. Despite this, the work units it executes for you can be written in any language you like.

Orchestra itself is far from finished, but it’s working reliably enough that we’re already using it in production in a very limited capacity, and have been slowly extending it’s reach into new areas as appropriate.

We’d love you to take a look and see if it works for you, we’re open to suggestions and contributions for improvement. If you’re after a quick overview, doc/orchestra.tex is a good place to start, and the samples/ directory contains commented config files for the various daemons.

We don’t believe in duplicating functionality, so it’s assumed that your config, SSL certificates and scores are distributed by another automation tool – we use puppet. Utilities like daemontools or god do a great job of keeping your daemons running.

SCRIPT IT!

As a big fan of automation, there wasn’t much chance that this whole thing wasn’t going to be scripted up the wazoo. We just need to copy the filesystem data across, dump the database and load it into the new site… and we’re done. Right?

HA! Not likely. To give you an idea of the scale of this thing, it took close to 24 hours just to do an rsync scan of the repository filesystems, without actually copying any data. Then there’s the database — the events table alone contained approximately 81.5 million records, which took a great many hours to dump from the live database during pre-migration work. It doesn’t take a great mathematician to realise that copying all this data over the Internet while the site was down for business wasn’t going to fly.

Initially, we were going to rely on the bandwidth of a station wagon full of tapes (or a couple of USB drives in a FedEx jet, anyway) to do the initial copying of data. However, due to some technical problems at the old facility, the “average transfer rate” wasn’t very high (the copy to disk took several weeks to complete), and we ended up kicking off a network-based initial sync of the repository data that finished less than half an hour after the drives were plugged into the machines at the new data centre. While I’m still a fan of shipping disks around for large-scale transfers, I won’t discount using the Internet to transfer such a large data set around so quickly next time.

Incrementalism

Since a single real-time copy wasn’t practical, we’d have to look to incremental copying, where we pre-sync as much data as possible before the Big Cutover Day, and then only copy the latest changes while the site is down.

Thankfully, Github’s software design has pretty much all the hooks we needed to make this a straightforward task. For example, we didn’t have to dump the entire events table, because once a row is written it’s never changed — so we only need to dump events that were created since the last dump.

The system also keeps track of the last time a repository was changed, which means that we can ask the database for a list of repositories that have changed since the last sync, which makes for a very simple (and quick!) incremental sync. For a smaller data set we would just use rsync directly, but due to the performance limitations of the previous hosting environment, this took far, far too long to do with just rsync.

So, we can script everything, and there’s the ability to do repeated incremental syncs. What do these scripts look like?

Well, first up, there’s a lot of them. It was best to write separate scripts to synchronise each data set — one for the repositories, one for the events table, one for the rest of the database, one for gists, and so on. This meant that it was fairly trivial to develop these scripts in parallel, and they could be tested and run independently of each other.

Also, each task that had to be performed for a given data set was in its own script, so each step could be tested independently. For example, the repo sync job consisted of one script to collect the list of repos that needed resyncing and write that list to disk, another script to sync a single repository, and a third script to loop over all the repos listed by the first script and invoke the second script for each of them.

The other important properties of these scripts were:

• We relied heavily on multitasking to overcome bandwidth limitations from a single TCP stream. When you’re copying data over high capacity links, your available transfer rate is constrained more by the round-trip time between the endpoints than the available bandwidth — the longer it takes for an ACK to get back to the sender, the slower your data will flow. So, since we had eight filesystems to copy data from, we fired off eight parallel rsync processes as child processes of the individual scripts.
• Each script kept track of what it was doing and what it had done, and tried to avoid doing the same work again. The repository syncs kept track of the repositories that had already been copied by means of a timestamp file — when we did a sync, we touched a file and then used the mtime of that file (stat -c %Y ftw!) to determine the start time of the next sync. The events table was straightforward — before each dump, we just ask the destination table where it’s up to, and dump from there. Even the “main” database, which we dumped in it’s entireity each time, was dumped to a file compressed with `gzip –rsyncable` before being rsync’d across, saving a good few minutes of network transfer time on each cycle.
• If something went wrong during the sync, we knew about it immediately. We wired up a small SMS sending script to send us alerts if the script terminated improperly. This saved us a lot of waiting and watching, because we knew that we’d be told when we had to take notice of what’s going on.
• Everything was logged. The stdout and stderr of all processes was captured, and the scripts wrote their own log entries to that stream as well as to a “summary” log, like this: echo $(date) processing repo $repo |tee -a $LOGFILE. Any errors were tagged with a unique string and written in a machine-parseable format, so we could re-run any failed components of the sync to ensure that nobody was missed.
• While there were typically several scripts that had to be run in an appropriate order to make a sync happen, there was always a single script that did everything that needed doing — we never had to run more than one command to get a given sync done.

Once all of these individual scripts had been written, tested, debugged, tested a few more times, and generally fretted over until our nails were chewed to the quick, it was time to assemble the master script. I’m not about to run a dozen scripts to migrate a site when one will suffice. This was particularly important in Github’s case because to minimise downtime we wanted to run several things in parallel, then wait until they’d all finished, then run the syncs that depended on the data we’d synced in the last lot, and so on. Our scripts looked a lot like this:

task1 >logs/task1.log 2>&1 &
task1_pid=${!}

task2 >logs/task2.log 2>&1 &
task2_pid=${!}

wait $task1_pid
wait $task2_pid

task3 >logs/task3.log 2>&1 &
task3_pid=${!}

task4 >logs/task4.log 2>&1 &
task4_pid=${!}

task5 >logs/task5.log 2>&1 &
task5_pid=${!}

wait $task3_pid
wait $task4_pid
wait $task5_pid

There was also a pile of “doing this, now doing this, now doing this” logging (with timestamps) that helped us to get a feel for how long the different parts would take, and where everything was up to.

When we actually performed the cutover, the “main” sync script was running for a total of 27 minutes. Given that we’d given ourselves an hour to get everything across, we were all quite pleased with this outcome.

Putting on the brakes

Whilst all these scripts ran really well, and the background processes made everything run really fast, I must say it was a right pain in the butt to stop things mid-flight when it was necessary. Hitting Ctrl-C only stopped the foreground (controller) script, and all of the children that had been started in the background kept flying along.

Doing this again, I’d make sure all my scripts had traps on SIGINT that killed off all the child processes that they had spawned. In retrospect, this is just a variant of “one script to start everything” — you should only need to do one thing (Ctrl-C) to stop it all, as well.

Also, the timestamp files weren’t handled real well. If you did kill things off mid-run (or, heaven forbid, a script crashed out) then the timestamp files would be wrong, because we just did a straight touch at the beginning of the script. What would have been better would be something like this:

touch stamp.new
do_all_the_work
mv stamp stamp.prev
mv stamp.new stamp

This would make sure that premature death would leave the stamp as-is, while still capturing the true start time of the job (which a simple touch at the end would fail to do).

When Databases Attack

Testing the new site before we let users at it, we found that creating gists wasn’t working right. It turned out that the database dumping script didn’t have the right set of options, and the schemas of the tables weren’t quite right (no autoincrements), and that was giving gist creation conniptions. Thankfully, the bug in the script was quickly spotted and the database dump was re-run. We even managed to get the second dump and load completed before our scheduled maintenance window was finished. If our scripts hadn’t
been broken down by data set, this resyncing process would have been made a whole lot harder because we wouldn’t have been able to easily run just the parts that needed to be redone.

Once we opened the floodgates of the new site, everything ran happily for a minute or two, and then ground to a halt. The whaaaaa? Poke, prod… hmm, the database is running a bit hotter than I’d expect… whoa! 1500 queries active, all against the events table, with the disks working so hard the heads nearly came out the sides of the cases. What’s going on here?

As it turns out, schema insanity had struck again — this time, some of the indexes on the events table had failed to come across. While we know what happened with the main database dump, this one is still a mystery. How did some of the indexes fail to materialise? We’ve gone over the dumps and can’t find how they got lost. We’re putting it down to yet another case of MySQL doing dumb things without telling anyone.

Limiting the impact

As a final small improvement to the migration process, the site was able to into a “read only” mode, so that users could still browse code and pull from repositories while we were migrating. This made the migration a lot less intrusive for users, because a lot of site functions still worked, especially those made by casual users (who would be less likely to know all about the time of the migration).

Lessons Learnt

Here are a few things I’ll definitely do differently next time:

• Anywhere you’re depending on a third party to execute part of your migration, have a backup plan in case they can’t deliver — and know when you’ll have to execute your backup plan. In our case, knowing exactly how long it would have taken to copy all the data over the Internet and then calculating back, we would have known to start copying over the network a few days earlier than we did.
• Make sure that synchronisation scripts are as easy to stop as they are to start.
• Verify the database schemas completely on the destination DB server by manual inspection, as well as dumping them and comparing to what’s on the source DB server.

I wonder when we’ll get our next Github-scale migration…

I came across this rather amusing company a little while ago. They’ve been around since before the turn of the millennium but I somehow hadn’t stumbled upon them until now. What with everything turning web2.0-ey nowadays, I’d say they’ve got themselves a nice little niche.

In case you’re too busy to visit (or too much of a slacker, TL;DR!), the concept is simple: you pay a yearly subscription fee (they call it a “sockscription”), and they keep sending you pairs of nice new black socks. It’s a very cute idea that has wide-ranging appeal and applicability. It also abides by one of the general rules for sysadmins, which is to automate whatever you can; the less time you waste thinking about repeatable tasks, the more time you have for fun things.

Right now I’m lamenting the state of the Aussie dollar, as the billing amounts will be in USD. However, this is far too compelling a concept to ignore, and the current state of my sock drawer is positively unacceptable with only two pairs of plain blacks socks (both with holes, no less!). One does not wear white sports socks with black patent leather lace-ups, it’s just not done (hey, I like to dress up). I’ll have to put in an order as soon as possible, those kneesocks look like just the ticket.

かえして！ニーソックス