It was with regret that day 5 of LCA2009 came and went. But it was a blast!

Simon Phipps of Sun Microsystems delivered the keynote, which mostly amounted to a scathing attack on the current state of the open source business model. People are going to be increasingly demanding for services they are interested in buying, not what the current flock of companies are selling. You don’t want to provide support for anything but RHEL or SLES? Too bad, to make money you’ll have to provide support for Fedora and SUSE.

OK, maybe it’s not as straightforward as he made out, but it’s a realistic-enough sounding proposition. At the very least I expect the rapidly changing web development environment will be increasingly demanding of the latest and greatest tools on completely supported platforms. The choice between stable OR cutting-edge will not be sufficient anymore – we want both! Don’t expect this overnight, folks.

Shortly after our arrival in Hobart we were fortunate enough to sit down to lunch with Matthew Wilcox of Intel, so we got a sneak preview of his presentation Solid State Devices. Aside from the unstated message of “Intel SSDs Rock!” (which is being actively competed against by some of the other players in the SSD market, who are making leaps and bounds in their technology) it was very interesting to see how we have finally escaped the hard drive bottleneck. Somewhat unsurprisingly this shows up limitations in the OS storage subsystem, which was never designed to handle data at the rates SSDs can deliver and store them. It’s a similar situation to when GigE and 10GigE devices came on the market initially.

Matthew Garrett (arguably the handsomest presenter at LCA, according to some sources) gave us some food for thought with Power Management That Works. Essentially the status quo is that users are given too much control over power management, when they can’t really make accurate-enough decisions for what will save the most power. This is no longer strictly a portable computer arena, as power savings are fast becoming a critical business discipline in the data centre. It is interesting to note the technologies that are currently available in recent hardware though, such as the ability to dramatically reduce the screen refresh rate in notebooks to lower power consumption caused by repeatedly scanning the frame buffer when nothing is moving on screen.

On the other end of the scale, where more power usage is desirable, was Geek My Ride with Jonathan Oxer (former president of Linux Australia) and Jared Herbohn (a.k.a. Flame), showing off their sports cars fitted out with Linux-powered entertainment, navigation, networking and engine-computer interfaces. If having an RFID tag implanted in your arm isn’t enough to feel like lord of the geeks, being able to SSH into your car surely is.

The final presentation of the conference I decided to attend was System Administration in a Large-Scale Linux Web-Hosting Environment by Dreamhost’s Terri Haber. Sound familiar? It’s always nice to know what the other players in the market are up to, and although the US market is much larger than what we enjoy at home the same principles apply. In fact, I hoped to learn some tricks the big boys use. Dreamhost has been around for quite some time and you would expect them to get the fundamental things about hosting right.

A common trend I find in companies of this scale and age is that since they started business at a time when many of the tools we take for granted didn’t exist, they have crafted many of these tools from scratch and customised them heavily for their own business. We use the open source tool Puppet for configuration management, but Dreamhost has their own custom configuration management tool written in object-oriented Perl. Sadly this trend continued through the presentation with their home-grown solutions. I did learn one thing though – we are designing our systems right. Could we be the next Dreamhost? Who knows.

So endeth another satisfying and richly rewarding Linux.conf.au! Lessons were learnt, beards were shaved, and maybe, just maybe, we helped save another species of native animal from extinction.

Day 4 of Linux Conf Au began with a hangover for many after the “Penguin Dinner” for professional delegates, and an equal number of empty wallets. The open source community generously gives a lot of their time to free software, and they are equally generous when it comes to other good causes such as the plight of the Tasmanian Devils. An auction was held for an impressive nature photograph print and other valuable items, at which we were able to raise an impressive $40,000+ figure. Awesome!

The presentation on CELT: A Low-Latency, High-quality Audio Codec by Timothy Terriberry was an eye (and ear) opener. It lives up to the claims made in the title, and then some, with amazing quality. A live demo was given and I think it surprised a lot of people. I wouldn’t be surprised if this codec starts turning up in commercial products before long.

I briefly attended The Joy of Inkscape by Donna Benjamin, a tutorial on the ins and outs of this fantastic vector-based graphic illustrator. I was already familiar with the application to some extent as I’d used it for generating the diagrams in my presentation last year, but it was nice to pick up a couple of tips before darting off to another interesting presentation in the same timeslot.

The afternoon yielded some extremely satisfying learning. Jeff Arnold presented Ksplice: Rebootless Kernel Updates to a very large, expectant audience (in a very small and stuffy room). Ksplice promises to revolutionise the way we handle security updates. Given the very active community based around discovery of security vulnerabilities in open source software, we see updates for the Linux kernel quite frequently (often multiple times per month). Reboots are inconvenient for us and clients, and any possibility of reducing the number of reboots we have to perform on our growing flock of servers is welcome. If Ksplice can earn the support of commercial software vendors, it should give a large chunk of time back to us lowly sysadmins

And of course, the day wouldn’t have been complete without seeing a robot-powered clarinet running on Linux!

Just a quick writeup for day 3, as I know you are still reeling from reading all about the first two action-packed days! We were lucky enough to have Tom Limoncelli give the keynote speech on Wednesday morning. He is the revered Systems Administrator who penned Time Management for System Administrators and The Practice of System and Network Administration, both of which are compulsory reading for every sysadmin at Anchor (we each have a copy of both books!), and he works for Google (no surprises there).

Tom gave a fairly moving and unconventional speech about how we think we are living in an environment of scarcity, but we are actually living in an environment of plenty. We can give so much more to other people that costs us nothing, and open source software is a great example of this (and is something Anchor is proud to support).

I enjoy creating music so the presentation on FFADO – Firewire audio for Linux by Jonathon Woithe was a nice break from the strictly sysadmin-oriented stuff I’d been attending on the first two days. Linux audio has progressed a lot recently and it is now possible to see sub 10ms latencies when using well-supported Firewire audio devices and the Realtime kernel options. We should see Linux penetrate the studio audio market increasingly as FFADO matures.

Subversion Scaling at Google by the always interesting Marc Merlin was a lot more applicable to us, as we use Subversion extensively for internal code and configuration repositories. Some of the smart fellows at Google have contributed a large number of improvements to Subversion that allow it to scale out a lot better, and add helpful collaborative tools (for example, commit hooks that force changes into a separate branch for review; reviewed changes are then merged from the branch into trunk). We’ll be looking into how we can integrate GVN into our workflow.

Asterisk, for those who don’t know already, is an open-source phone system application. Anchor made the transition to Asterisk a couple of years ago from a proprietary system and has never looked back thanks to the unprecedented flexibility, configurability and open-sourcey goodness. Using Asterisk for Fun and Profit by Avi Miller was a tutorial into using the Elastix distribution. If you haven’t checked out Asterisk, I highly suggest you download the Elastix LiveCD and give it a try.

See you at LCA2009 Day 4!

On day two of Linux Conf I was able to attend two presentations on IPv6 – System Administration Consequences of the Endgame of IPv4 and the Deployment of IPv6 by AARNET’s Glen Turner and Google and IPv6 by Angus Lees. Both were extremely informative and made it clear to me that we need to start gearing up for IPv6. By “we”, I mean the world.

Don’t get me wrong – if you are the average home user IPv6 (or even IPv4) will mean nothing to you and the advent of IPv6 addressing en masse will likely pass you by without you even noticing. Much like the Y2K bug though, it will only be with the coordinated efforts of the best network and systems administrators around the world that we’ll be able to jump the hurdle again so gracefully.

Glen painted a fairly grim picture – by around 2010 we should expect to run out of IPv4 addresses. Admittedly there are large historical allocations which have very little usage but we can expect these to turn into lucrative commodities, bought and sold for whatever price the seller decides. IPv4 addresses will still be available, but at insanely inflated prices. The upshot of this? You can expect ISPs to start dealing in Enterprise NAT. Publicly routable IP addresses for anyone who isn’t willing to pay the price will disappear, as will end-to-end connectivity. The new cash cow for ISPs will be selling broker services for any applications requiring end-to-end connectivity such as VoIP and gaming.

In actual fact, this consumer-driven environment may spell doom for IPv6. ISPs will be able to make more money by selling services that work around the limitations in IPv4, thus it is in their best interests to not make IPv6 available. Everyone would benefit from IPv6 but it needs to be made available first. Angus Lees presented a report on the IPv6 survey conducted last year by Google. Using a variety of techniques piggy-backing on their search interface they were able to determine that around 0.2% of their users have IPv6 access. However almost half of these are not working properly. This is not enough of a userbase for which to enable IPv6 fulltime, and certainly too many non-functional deployments for which to risk breaking their search engine.

Will we all benefit from IPv6? Yes. Is it necessary to ensure the Internet continues working effectively? Definitely. Is it that hard to deploy? Maybe not.

Anchor has decided to commit to deploying IPv6, and to demystify this process we will document our progress publicly. You can view our progress on our public wiki at http://www.anchor.com.au/hosting/IPv6. If you want to make the Internet a better place by pushing for the widespread deployment of IPv6, talk to your ISP and ask them what their IPv6 deployment plans are.

OK, so I’m a little late on giving the day 2 review (and days 3-5), but in my defence I was just that busy LEARNING.

Quite seriously though, the amount of information available to you at this kind of conference can be almost overwhelming. It is a task in itself just to keep up with the presentations.

“747s on Treadmills: Experiences Scaling Uptime and Management” was a high-level look at large web-serving infrastructure rollouts by Matt Moor of Bulletproof Networks covering a project for an unnamed client that was expecting a very high amount of traffic to their charity website, peaking in November last year (surely you can guess the charity). As with many projects (and certainly with computer related technology) not everything goes 100% smoothly and indeed they did suffer hardware failures, lack of cooperation from developers and not being able to manage as much of the configuration as they would have liked to in the time available – a familiar situation.

While it was interesting hearing the approach to a common problem taken by a competitor in the market, I was left feeling unsatisfied, mostly with the lack of technical detail. I couldn’t help but compare the presentation to my own from last year where I delved head-first into configuration files, daemons and more. Maybe it is competition in the business world, individual pride of the sysadmin or just plain stubbornness. Regardless, I genuinely believe we can do things better than them and I will stick to that. Maybe there’s another LCA presentation in it!

Barney was able to attend the “Enhanced Monitoring of MySQL Servers” presentation by Arjen Lentz (at this stage still in possession of his hair). This mostly focussed on the OurDelta project, which is a collection of enhancements to MySQL created by various 3rd parties and maintained in the central repository on this site. Most of them offer significant benefits to MySQL users so it is a project to keep an eye on (and hopefully make use of) if you are a thriving MySQL DBA.

Two of the (in my humble opinion) most interesting and most important presentations of the day (perhaps even week) were those on IPv6 by AARNET’s Glen Turner and Google’s Angus Lees. Angus presented the results of a study done by his team at Google on IPv6 penetration on the Internet. In short, there is very little existing deployment (around 0.2%) and of those active deployments around half are broken. Glen painted a fairly grim view of the future of the Internet, where ISPs will profit from enterprise NAT deployments and the end user in general will suffer. We will hit the limits of IPv4 addressing within a couple of years and the logical solution is IPv6.

The impact of these presentations was not lost on us, and as a result we are now committing to deploying IPv6. I’ve made a separate summary of those presentations in a separate blog article which details a lot of the reasons for moving to IPv6, and you can read more about our deployment this on our public wiki at http://www.anchor.com.au/hosting/IPv6

Stick around and read our review of LCA2009 Day 3!

The LinuxConf Penguin Dinner was held the other night, and the key event for the night was a charity auction. The goal in this case was to raise money to help save the Tasmanian Devil. It’s a great cause and bidding was moving along nicely with the auction being run by Rusty Russell and Bdale Garbee. Things eventually reached a plateau, however, and someone really needed to up the stakes a bit. Which led to calls for Bdale to pledge his beard.

Beards are very Serious Business for hackers, and Bdale tells us he’s been growing his since 1982 when he was at college. It’s older than me! The bidding consortium that eventually won the auction also insisted that Linus Torvalds do the shaving.

Thus, it was no small event when the stage was prepared for Bdale to lose the beard.

Linus giving Bdale a shave

After much ceremony, Linus took a bow and left Bdale to take care of things.

Bdale was onto his third razor by now

It’s a surreal experience to be sitting quietly in a room with 500 other people, watching a man shave himself. As we hear it, Bdale hasn’t told his wife about this yet. He’s just hoping she’ll let him into the house again.

Arjen Lentz also agreed to shave his head for the auction, and Jared Herbohn has given up his prized “GEEK” numberplates for a year. Everyone’s been a great sport, and word has it we’ve raised about $40,000 so far (still going).

What a feeling to be back amongst the sights, sounds and ah yes the smells of Linux Conf Au once more. It doesn’t seem so long ago that I was in Melbourne enjoying the fruits of the last conference. So what does this year have to offer? I am fortunate enough to primarily enjoy presentations that also most benefit my role as a Systems Administrator. There are plenty of great presentations though on other topics such as mobile/embedded devices and multimedia. The quality is high but I don’t tend to enjoy these as much.

The first day (for me at least) offered a few ups and a few downs. “Is Parallel Programming Hard, And If So, Why?” by Paul McKenney was more of a philosophical look into the reasoning around parallel programming and despite not really diving into the “how” side of things was enjoyable.

A couple of talks on systems provisioning and automation left me desiring a bit more though. I feel like what was presented was stuff that we all should have known years ago. Still it is good for people to be pushing Kickstart and Puppet which are both worthy tools, but that doesn’t mean I can’t hunger for the next big development.

“Security-Enhanced PostgreSQL” was certainly an eye opener, but given our experiences with SELinux (around which the SEPostgreSQL project is based) lead me to believe the integration will take quite some time to be completely usable in all scenarios, and being a security product it will have to meet that criteria before it will gain widespread acceptance. Something to keep an eye on though.

The final presentation I attended was “Rails Deployment In The Enterprise” by Robert Postill. I am no developer, and both Ruby and Rails have been used worldwide with great success but somehow I have managed to avoid getting to know either of them. This presentation added fuel to the fire that is already telling everyone “if you haven’t looked into Rails yet, you really need to now” – and this is completely true.

Building web applications holds a lot in common with production line automation. We’ve really progressed beyond building the same tools and parts again and again to make websites – Rails stops the need for reinventing the wheel and I was able to appreciate that finally today. Coincidentally during the presentation I was able to create a simple blog-style app using Rails that I had been meaning to do as part of some server testing, so the impact for me was doubled. Even as a non-developer I can appreciate it, and that bodes well.

Aside from the presentations, there is a general feeling of inspiration surrounding this entire event. The air is charged with the collective intelligence and open-source passion of hundreds of enthusiasts for the same cause. I’m looking forward to not only the rest of the scheduled conference but working on problems with a fresh mind in the spare time here and being inspired to do new things.