Blog

This is the first post of a two-part set.

We’re talking about keeping your system up to date, specifically Linux ones.

Maybe you religiously check for updates and apply them every Monday. Maybe you’re a bit perverse and start your morning with a black coffee and a quick emerge world. Or maybe you’re lazy (like me) and only bother doing it whenever you login and see “66 packages can be updated”.

Whatever you do, keep doing it, and regularly. But what do you do when you have hundreds, maybe thousands of machines that need updates applied? (Hint: we’re lazy, so the answer is always “make a robot do it”)

---

We used to apply updates manually, which worked well when there weren’t so many servers to deal with, and you can check for any problems along the way. The first step to automation was realising that applying vendor-supplied updates wholesale, without oversight, isn’t really that scary (you might not think it so scary, but we have an aversion to risk and a responsibility to our customers). We maintain RHEL, Debian and Windows servers here, and all three of them have proven to be solid when it comes to not-detonating-your-system.

Well that’s easy, you just need to roll out a cronjob everywhere that runs yum -y update everyday, and the problem is solved. And aptitude update ; aptitude -y safe-upgrade for Debian. Better add the –quiet flag as well, so that cron doesn’t send spurious mail, and/or discard the output by redirecting all output to /dev/null.

You could do all this, and then you really hope nothing breaks, ’cause it’s hard to know just what’s happening on your system now.

---

These annoyances aren’t insurmountable, but they’re tedious and finicky. That’s why our resident Overkill Specialist developed tingle.

tingle abstracts away the little details of your distro’s package manager, making things consistent between systems, and adds some smarts to make thing more efficient and flexible for your environment. On a tingle-enabled system you just login and fire a single command:

tingle reboot

That’s it. Updates will be downloaded, applied, then a reboot will be scheduled to occur in 10min. If you don’t want the reboot (which is necessary for kernel updates) then tingle apply will do the trick.

We make use of tingle’s hookpoints to remove old kernel packages and run puppet after applying updates. Given the rich set of hooking options, you could easily have the server remove itself from a load-balancing cluster before applying updates, or suppress monitoring warnings while it reboots. Whatever you do, the end result is the same: systems that are more reliable and need less human intervention for mundane tasks.

---

If you’d like to try out tingle, you can find a copy in our github repo at:

https://github.com/anchor/tingle

Tune in for the next post on Monday, when we’ll talk about how we automate the use of tingle. It’s useful on its own, but we still need to solve the very real problems of making it happen, arranging a suitable time for this to happen with the customer, and of course the elephant in the room: monitoring it to make sure you’re meeting your contractual obligations (non-existent updates are the ones that cause zero downtime…)