We actually had two great keynote talks on freedom, I’d like to step back to yesterday’s talk by Karen Sandler (you can see the talk for yourself on on youtube, which I’d highly recommended). Karen was diagnosed with hypertrophic cardiomyopathy, a heart condition that means she could suddenly die at any time. Thankfully there are treatments available, one of which is a pacemaker.

Being the person she is, she immediately asked “what software does it run?”. Long story short, the manufacturer ended up stonewalling on the issue, refusing to provide code or further details even with an NDA. Noone had ever asked before, and everything was pushed back with assurances that the devices are safe, and that they’re approved by the FDA.

It might seem like a trivial matter, but it’s a big deal if you step back and consider it. This device is implanted in your body to regulate your heart. In the event of cardiac arrest, your life could be 100% dependent on it functioning properly. I think it’s safe to say that failure is unacceptable.

Okay, you say, but they work very well for a lot of people. This is true. But the devices are known to be imperfect – putting aside the issue that they may not function correctly when needed, there are clear concerns regarding malicious access by an attacker. There’s published research for this on both pacemakers and insulin pumps for diabetics.

The hard questions clearly irked a lot of people, including her doctor, who was greatly upset that she’d even be asking such things. The practical concerns did eventually win out (though she was able to get an older, less advanced device), leading to this statement:

I became a cyborg lawyer with proprietary software connected to my heart.

Switching focus to social networking, Bdale Garbee (possibly best known for free beards) has been working on FreedomBox, personal servers for social networking. The immediate need for another social network isn’t obvious – the key here is the storage and control of your own personal information. It’s your data, it should be kept on your terms.

As it stands, your data in Facebook/Google+/FoospaceEtc. could be stored anywhere in the world. For all the privacy policies and statements, you don’t know where that information is, or who really has access to it (think of legal jurisdictions). Designed with tiny “plug computers” in mind, this decentralisation should make it feasible to run your own server from home. Whether Australian internet will ever be up to the job is another matter…

There’s a lot we could go on about but for lack of time. In all, it was a very successful conference: a talk was given, ponies were node’d, a mobile phone was sent towards the stratosphere on party balloons, and Project Horus had their own successful launch. Next year we’re off to Canberra for LCA, hope to see you there!

Instead, we’ll have a really quick rundown of the high availability (HA) and virtualisation talks, and why it’s a good thing we sent a sysadmin along to them.

Complexity is your biggest enemy when trying to build reliable systems. Complex systems tend to be flaky, and that means they’re unpredictable. Unpredictable systems are bloody hard to support and rely upon. You won’t read this in all the you-beaut cloud services literature, but highly available systems are complex. Really, really complex.

This is all manageable, but it means your staff need to be trained with an intimate understanding of everything, top to bottom. When you’re unfamiliar with it, the HA stack on linux is like the bogeyman. It scares the living daylights out of you, and you try to pretend that if you close your eyes it’ll just go away. This is okay most of the time, but for a company like Anchor it would leave you dependent on a small team of HA gurus when things go wrong.

Thank $DEITY for the High Availability Sprint at LCA. Anchor can train you in The Way Of The Cluster if you so desire, but an enlightenment session from the jedi grandmasters is immeasurably valuable. Knowledge breeds confidence, and these things translate to a more effective sysadmin. If you’re an Anchor customer with an HA system, it means we can support you better, and respond faster when there’s a problem. Everyone wins!

To wrap up, a quick look at the presentation on Ganeti, software for management of a cluster of virtual machines.

We evaluated Ganeti for our needs a couple of years ago as a VM solution, and found that it wasn’t mature enough to really be usable. It’s clearly grown up since then, but I think it might be more interesting to discuss why it’s still no good for us.

Most people can probably look at the featureset and determine whether it’s what they need. Magical on-demand clouds of VMs are the “in thing” at the moment, what aren’t they good for? Well, it turns out they’re not much good for web-hosting.

This really became evident several months ago when we tasked a sysadmin with evaluating the various cloud management products on the market (free or otherwise). It’s kinda disappointing, but the truth is that we don’t need 100 instances of the same machine. We certainly don’t want them to be ephemeral. The other benefits touted by cloudy VMs, such as live migration and replication, are nice but ultimately not that useful for us.

In the end we developed a system that met our real needs, as plain as they are: really fast to deploy, fully automated, customisable, comprehensively supported and monitored.

It’s exciting stuff, but we’ll be waiting a bit longer to consider it robust enough to use in production. We’d kill for the integrated snapshotting (great for backups) and data integrity checking (store CRCs with your data) features.

Meanwhile, XFS reports steady progress and positions itself as the filesystem of choice for Really Big systems. Not that anyone would admit to it, but it was clear there was a little bit of rivalry between the two, especially since both talks were back-to-back in the same room.

Dave Chinner talked about how they’ve spent a lot of time working through the metadata performance issues that have caused headaches for scaling-up in the past, and reckons XFS should scale linearly, unlike the competition. Probably not something you’ll lose sleep over when deciding how to format your root filesystem, but definitely important for databases and big filestores.

In lieu of other diversions, let’s have a look at the LeoStick, which was included in the bag of goodies for LCA attendees, alongside the requisite stubby coolers and mousepads.

Unless you’ve been living under a really big rock, the Arduino is the go-to platform for hackers wanting to build embedded systems. This is thanks to ease of programming, fast prototyping, and expansion options (need a thermal probe? fingerprint scanner? CCD camera? there’s probably a single shield module with all of those things). The Leostick is particularly cute in that it comes in USB thumbdrive form-factor. As this is a pre-release board, the more cynical amongst us will note that this is a stroke of marketing genius that should result in some free beta-testing. Heh.

I know a couple of my fair colleagues are handy with a soldering iron; just quietly, this thing may or may not have had something to do with requests from the LCA organisers to stop messing with the exposed USB ports on the electronic door locks around campus.

Actually, it wouldn’t be fair to say it was that easy, so I’ll let the cat out of the bag on this one:

How Anchor's presentation slides for LCA2012 got done in time

Panel 1

T-Rex: Our talk to linux.conf.au got accepted!

Panel 2

{Close-up of T-Rex’s face, he is visibly excited}
T-Rex: It will be AWESOME

Panel 3

{Zoom out to show T-Rex and Dromiceiomimus. T-Rex is about to confidently stomp a tiny house}
Dromiceiomimus: You’ve prepared the talk months in advance, right?
T-Rex: 1337 speakers such as myself need no such preparation!

Panel 4

{Utahraptor replaces Dromiceiomimus in shot, verbally catching T-Rex just as he is about to stomp a tiny woman}
Utahraptor: But what about the slides?

Panel 5

{Now some distance apart, T-Rex and Utahraptor look directly at each other, in tense silence}

Panel 6

T-Rex: Oh uni placement dude?! Can I ask you a favor???

I kid, I kid – they did make the slides themselves, all of them. No uni students were harmed or exploited in the making of this talk.

To wrap up, one talk that covered a topic that doesn’t get much loving was Moving Day: Migrating Big Data from A to B. Mozilla had more than 40TB of data in their crash-reporting system, which demands near 100% uptime, and needed to move it all to a new datacentre – not something to be cowboyed the morning after an all-night bender.

Rigorous planning, automation and testing ensured that everything went smoothly; this talk instilled an idea of how to approach such a mammoth project with confidence.

This is something we handled when Github moved to Rackspace, but Mozilla also added a “post-mortem” phase – even if everything goes well (it did), there are lessons to be learnt from the experience, which stands you in good stead for the next time.

The basic concept is to use any SSD device as a cache in front of slower rotational media. This is similar to commercial products such as LSI’s Cachecade, but implemented as a linux device-mapper module (so you wouldn’t be able to boot from such a setup, but that’s unlikely to be a real concern).

One of the nice things about Flashcache is that it’s presented as a plain block device. As well as making for a robust and understandable system, a practical upshot of this is that you can also replicate your cache with DRBD. In large HA database setups, this would mitigate a lot of the cache warmup penalty that you suffer after a reboot or failover event.

Flashcache is also fairly configurable, and exposes a lot of stuff through procfs rather than being a black box.

At the moment you have to build it as an out-of-tree module, so of course it’s not the kind of thing we’ll be rushing into production any time soon. Based on what we’ve seen in the past, I reckon there’s a good chance we’ll see Flashcache in mainline in a year or two if there’s a concerted push on development.

In all seriousness, we do hope you’ll come along if you’re attending Linuxconf and this tickles your fancy:
Any monkey can build the same server over and over again reliably.
But what if you need reliable server builds, and every single one is a little different?

If you’d like a little more in-depth detail, the LCA website has a copy of the abstract for the talk.

In addition to presenting, Anchor is sponsoring LCA and has sent a crack team of DevOperatives to look after things. Come and say Hi if you spot us.

Simon Phipps of Sun Microsystems delivered the keynote, which mostly amounted to a scathing attack on the current state of the open source business model. People are going to be increasingly demanding for services they are interested in buying, not what the current flock of companies are selling. You don’t want to provide support for anything but RHEL or SLES? Too bad, to make money you’ll have to provide support for Fedora and SUSE.

OK, maybe it’s not as straightforward as he made out, but it’s a realistic-enough sounding proposition. At the very least I expect the rapidly changing web development environment will be increasingly demanding of the latest and greatest tools on completely supported platforms. The choice between stable OR cutting-edge will not be sufficient anymore – we want both! Don’t expect this overnight, folks.

Shortly after our arrival in Hobart we were fortunate enough to sit down to lunch with Matthew Wilcox of Intel, so we got a sneak preview of his presentation Solid State Devices. Aside from the unstated message of “Intel SSDs Rock!” (which is being actively competed against by some of the other players in the SSD market, who are making leaps and bounds in their technology) it was very interesting to see how we have finally escaped the hard drive bottleneck. Somewhat unsurprisingly this shows up limitations in the OS storage subsystem, which was never designed to handle data at the rates SSDs can deliver and store them. It’s a similar situation to when GigE and 10GigE devices came on the market initially.

Matthew Garrett (arguably the handsomest presenter at LCA, according to some sources) gave us some food for thought with Power Management That Works. Essentially the status quo is that users are given too much control over power management, when they can’t really make accurate-enough decisions for what will save the most power. This is no longer strictly a portable computer arena, as power savings are fast becoming a critical business discipline in the data centre. It is interesting to note the technologies that are currently available in recent hardware though, such as the ability to dramatically reduce the screen refresh rate in notebooks to lower power consumption caused by repeatedly scanning the frame buffer when nothing is moving on screen.

On the other end of the scale, where more power usage is desirable, was Geek My Ride with Jonathan Oxer (former president of Linux Australia) and Jared Herbohn (a.k.a. Flame), showing off their sports cars fitted out with Linux-powered entertainment, navigation, networking and engine-computer interfaces. If having an RFID tag implanted in your arm isn’t enough to feel like lord of the geeks, being able to SSH into your car surely is.

The final presentation of the conference I decided to attend was System Administration in a Large-Scale Linux Web-Hosting Environment by Dreamhost‘s Terri Haber. Sound familiar? It’s always nice to know what the other players in the market are up to, and although the US market is much larger than what we enjoy at home the same principles apply. In fact, I hoped to learn some tricks the big boys use. Dreamhost has been around for quite some time and you would expect them to get the fundamental things about hosting right.

A common trend I find in companies of this scale and age is that since they started business at a time when many of the tools we take for granted didn’t exist, they have crafted many of these tools from scratch and customised them heavily for their own business. We use the open source tool Puppet for configuration management, but Dreamhost has their own custom configuration management tool written in object-oriented Perl. Sadly this trend continued through the presentation with their home-grown solutions. I did learn one thing though – we are designing our systems right. Could we be the next Dreamhost? Who knows.

So endeth another satisfying and richly rewarding Linux.conf.au! Lessons were learnt, beards were shaved, and maybe, just maybe, we helped save another species of native animal from extinction.

The presentation on CELT: A Low-Latency, High-quality Audio Codec by Timothy Terriberry was an eye (and ear) opener. It lives up to the claims made in the title, and then some, with amazing quality. A live demo was given and I think it surprised a lot of people. I wouldn’t be surprised if this codec starts turning up in commercial products before long.

I briefly attended The Joy of Inkscape by Donna Benjamin, a tutorial on the ins and outs of this fantastic vector-based graphic illustrator. I was already familiar with the application to some extent as I’d used it for generating the diagrams in my presentation last year, but it was nice to pick up a couple of tips before darting off to another interesting presentation in the same timeslot.

The afternoon yielded some extremely satisfying learning. Jeff Arnold presented Ksplice: Rebootless Kernel Updates to a very large, expectant audience (in a very small and stuffy room). Ksplice promises to revolutionise the way we handle security updates. Given the very active community based around discovery of security vulnerabilities in open source software, we see updates for the Linux kernel quite frequently (often multiple times per month). Reboots are inconvenient for us and clients, and any possibility of reducing the number of reboots we have to perform on our growing flock of servers is welcome. If Ksplice can earn the support of commercial software vendors, it should give a large chunk of time back to us lowly sysadmins

And of course, the day wouldn’t have been complete without seeing a robot-powered clarinet running on Linux!

Tom gave a fairly moving and unconventional speech about how we think we are living in an environment of scarcity, but we are actually living in an environment of plenty. We can give so much more to other people that costs us nothing, and open source software is a great example of this (and is something Anchor is proud to support).

I enjoy creating music so the presentation on FFADO – Firewire audio for Linux by Jonathon Woithe was a nice break from the strictly sysadmin-oriented stuff I’d been attending on the first two days. Linux audio has progressed a lot recently and it is now possible to see sub 10ms latencies when using well-supported Firewire audio devices and the Realtime kernel options. We should see Linux penetrate the studio audio market increasingly as FFADO matures.

Subversion Scaling at Google by the always interesting Marc Merlin was a lot more applicable to us, as we use Subversion extensively for internal code and configuration repositories. Some of the smart fellows at Google have contributed a large number of improvements to Subversion that allow it to scale out a lot better, and add helpful collaborative tools (for example, commit hooks that force changes into a separate branch for review; reviewed changes are then merged from the branch into trunk). We’ll be looking into how we can integrate GVN into our workflow.

Asterisk, for those who don’t know already, is an open-source phone system application. Anchor made the transition to Asterisk a couple of years ago from a proprietary system and has never looked back thanks to the unprecedented flexibility, configurability and open-sourcey goodness. Using Asterisk for Fun and Profit by Avi Miller was a tutorial into using the Elastix distribution. If you haven’t checked out Asterisk, I highly suggest you download the Elastix LiveCD and give it a try.

See you at LCA2009 Day 4!

Quite seriously though, the amount of information available to you at this kind of conference can be almost overwhelming. It is a task in itself just to keep up with the presentations.

“747s on Treadmills: Experiences Scaling Uptime and Management” was a high-level look at large web-serving infrastructure rollouts by Matt Moor of Bulletproof Networks covering a project for an unnamed client that was expecting a very high amount of traffic to their charity website, peaking in November last year (surely you can guess the charity). As with many projects (and certainly with computer related technology) not everything goes 100% smoothly and indeed they did suffer hardware failures, lack of cooperation from developers and not being able to manage as much of the configuration as they would have liked to in the time available – a familiar situation.

While it was interesting hearing the approach to a common problem taken by a competitor in the market, I was left feeling unsatisfied, mostly with the lack of technical detail. I couldn’t help but compare the presentation to my own from last year where I delved head-first into configuration files, daemons and more. Maybe it is competition in the business world, individual pride of the sysadmin or just plain stubbornness. Regardless, I genuinely believe we can do things better than them and I will stick to that. Maybe there’s another LCA presentation in it!

Barney was able to attend the “Enhanced Monitoring of MySQL Servers” presentation by Arjen Lentz (at this stage still in possession of his hair). This mostly focussed on the OurDelta project, which is a collection of enhancements to MySQL created by various 3rd parties and maintained in the central repository on this site. Most of them offer significant benefits to MySQL users so it is a project to keep an eye on (and hopefully make use of) if you are a thriving MySQL DBA.

Two of the (in my humble opinion) most interesting and most important presentations of the day (perhaps even week) were those on IPv6 by AARNET’s Glen Turner and Google’s Angus Lees. Angus presented the results of a study done by his team at Google on IPv6 penetration on the Internet. In short, there is very little existing deployment (around 0.2%) and of those active deployments around half are broken. Glen painted a fairly grim view of the future of the Internet, where ISPs will profit from enterprise NAT deployments and the end user in general will suffer. We will hit the limits of IPv4 addressing within a couple of years and the logical solution is IPv6.

The impact of these presentations was not lost on us, and as a result we are now committing to deploying IPv6. I’ve made a separate summary of those presentations in a separate blog article which details a lot of the reasons for moving to IPv6, and you can read more about our deployment this on our public wiki at http://www.anchor.com.au/hosting/IPv6

Stick around and read our review of LCA2009 Day 3!