LCA day 4 – On freedom

Published January 23rd, 2012 by Barney Desmond

It goes without saying that Linuxconf is all about free software, as in both beer and/or speech. A number of today’s talks focused on freedom, in the context of access to data and code, and the freedom to use software (and hardware) the way you see fit.

We actually had two great keynote talks on freedom, I’d like to step back to yesterday’s talk by Karen Sandler (you can see the talk for yourself on on youtube, which I’d highly recommended). Karen was diagnosed with hypertrophic cardiomyopathy, a heart condition that means she could suddenly die at any time. Thankfully there are treatments available, one of which is a pacemaker.

Being the person she is, she immediately asked “what software does it run?”. Long story short, the manufacturer ended up stonewalling on the issue, refusing to provide code or further details even with an NDA. Noone had ever asked before, and everything was pushed back with assurances that the devices are safe, and that they’re approved by the FDA.

It might seem like a trivial matter, but it’s a big deal if you step back and consider it. This device is implanted in your body to regulate your heart. In the event of cardiac arrest, your life could be 100% dependent on it functioning properly. I think it’s safe to say that failure is unacceptable.

Okay, you say, but they work very well for a lot of people. This is true. But the devices are known to be imperfect – putting aside the issue that they may not function correctly when needed, there are clear concerns regarding malicious access by an attacker. There’s published research for this on both pacemakers and insulin pumps for diabetics.

The hard questions clearly irked a lot of people, including her doctor, who was greatly upset that she’d even be asking such things. The practical concerns did eventually win out (though she was able to get an older, less advanced device), leading to this statement:

I became a cyborg lawyer with proprietary software connected to my heart.

Switching focus to social networking, Bdale Garbee (possibly best known for free beards) has been working on FreedomBox, personal servers for social networking. The immediate need for another social network isn’t obvious – the key here is the storage and control of your own personal information. It’s your data, it should be kept on your terms.

As it stands, your data in Facebook/Google+/FoospaceEtc. could be stored anywhere in the world. For all the privacy policies and statements, you don’t know where that information is, or who really has access to it (think of legal jurisdictions). Designed with tiny “plug computers” in mind, this decentralisation should make it feasible to run your own server from home. Whether Australian internet will ever be up to the job is another matter…

There’s a lot we could go on about but for lack of time. In all, it was a very successful conference: a talk was given, ponies were node’d, a mobile phone was sent towards the stratosphere on party balloons, and Project Horus had their own successful launch. Next year we’re off to Canberra for LCA, hope to see you there!

Tags: , , , , , , ,
Posted in FTW

 Leave a comment

0
Comments

AUTOMATE ALL THE THINGS!

Published August 30th, 2011 by David Basden

Illustration © Allie Brosh of Hyperbole and a Half

You can’t walk two metres down the street without someone going on about how cool and hip “The Cloud” is these days, being able to spin up hundreds of identical Linux VMs easily. Tools to build and configure lots of identical systems or VMs are plentiful.

But what if there is no “standard build” or even anything close, with different hardware, networking, software, distro, services, firewalls etc. every time, but you don’t want to spend all your time doing custom server builds and configuration?

Being a provider that specialises in customised hosting solutions, not only do most of our server builds have custom requirements, but we also have to configure lots of our own internal systems to deal with the eccentricities of each new server that we bring up. We have systems for monitoring, notifications, backups, fire-walling, accounting, scheduling of system updates, and that’s only a sampling of them.

We heavily use tools like puppet to automate things, but with almost every server being different, even configuring the tools designed to automate configuration starts taking serious time. For us, building a custom system and customising the systems that support it can take one of our sysadmins the best part of a day.

To solve this problem we’ve written some software that will take vague hand-waving from sales people[0] and turn it into a deterministic set of build steps that can be automated, without a sysadmin being involved at all. We now have heterogeneous server builds that are fully automated, and spun up in minutes rather than days. Rather than yet another bland VM copy, you get a beautiful and unique snowflake of awesome.

Even more fun, we’ve released them all as free software under the modified BSD license.

The first tool is called make-magic. Its job is to turn the high level requirements (I want a webserver and I want it blue) in to a detailed set of steps that can be automated (Find an online-paint shop. Buy some blue paint. Get it shipped to the data centre. etc.).

The core of make-magic can be guaranteed to always output a valid, correct list of steps for any set of input requirements. It even keeps track of which ones are done, which steps require others to be done first, and can let you know which ones can be done all at the same time.

There are some really interesting underlying problems that come up when you have to automatically generate a really specific, and provably correct set of steps from hundreds of thousands of permutations based on some high level requirements. We’ve beaten our heads against those brick walls, put it in a box, and posted it on github. You can find it at https://github.com/anchor/make-magic

Fun as it is, we still don’t want to have to go through and do those steps ourselves. This is where the second tool, mudpuppy, comes in.

mudpuppy is a Python based automation agent/client for make-magic. It allows you to write independent ‘modules’ in Python code, to automate items in a make-magic task. When run, mudpuppy will poll the make-magic server for tasks that need to be done, and will automate any items which it has a module for. Once the module has run successfully, mudpuppy will tell make-magic that the item has been completed, and will look for more work to do.

You can find mudpuppy on github at https://github.com/anchor/mudpuppy

mudpuppy isn’t the only tool that can be controlled by make-magic. make-magic has a simple, well defined JSON based HTTP API that mudpuppy uses, and which should be easy to talk to with most languages.

Earlier this month, Chris wrote about Orchestra, a set of tools for fast, reliable remote job execution on remote servers. mudpuppy is able to talk directly to Orchestra and get it to spin off jobs, get back the results, and use it to update information in make-magic. No extra parts required: They talk to each other out of the box.

As mentioned in Chris’s post, you can find Orchestra on github at https://github.com/anchor/orchestra

The combination of make-magic, mudpuppy and Orchestra (plus puppet, whatever else you might have lying around, or anything you’ve always wanted to try) give us the ability at the end of the day for us to build custom servers faster, reliably, and with ludicrously more flexibility than with any other systems that automate server builds that we’ve come across. Hopefully you’ll find them useful too.

If you’re interested in learning some more, Chris and I will be talking at the Sydney DevOps meetup on the 15th of September. There’s more details at http://devops.meetup.com/cities/au/sydney/

[0] Actually our “sales” people are pretty technical themselves, so they tend to wave their hands in the right direction.

Tags: , , , , , , , ,
Posted in FTW

 Leave a comment

1
Comment

The Automation Waltz

Published August 5th, 2011 by Chris Collins

When you have a bunch of machines involved in a process, you need to ensure that various stages in this process have executed. If the target host is unavailable, you want a guarantee that the job will execute when the host becomes available again.  This is well beyond the capabilities of ssh in a for-loop.

In trying to solve this problem, we had assessed tools like mcollective, but came to the conclusion that they were inappropriate for our environment.  mcollective in particular was removed from consideration as it was designed for a more homogeneous environment than the one here at Anchor.

When we realised we needed a different solution, a few of us gathered around a whiteboard and started enumerating our requirements.  The result was Orchestra, which we’re releasing today under the BSD License.

Orchestra is a suite for managing the reliable execution of tasks over a number of hosts. It is intended to be simple and secure, leaving more complicated coordination and tasks to tools better suited for those jobs.

Because everybody loves this web development stuff, it was developed to provide an interface that operates asynchronously in relation to the
execution of the work being done, allowing for cheap and easy polling of job state.

And last, but not least, because having critical system services depend on potentially fragile language interpreters or their libraries is generally a really bad idea for reliability, it was completely implemented in Go - a type-safe static language that compiles to native code and includes many features derived from dynamic languages. Despite this, the work units it executes for you can be written in any language you like.

Orchestra itself is far from finished, but it’s working reliably enough that we’re already using it in production in a very limited capacity, and have been slowly extending it’s reach into new areas as appropriate.

We’d love you to take a look and see if it works for you, we’re open to suggestions and contributions for improvement. If you’re after a quick overview, doc/orchestra.tex is a good place to start, and the samples/ directory contains commented config files for the various daemons.

We don’t believe in duplicating functionality, so it’s assumed that your config, SSL certificates and scores are distributed by another automation tool – we use puppet. Utilities like daemontools or god do a great job of keeping your daemons running.

Tags: , , , , ,
Posted in FTW

 Leave a comment

1
Comment