Tag

kernel Archives - AWS Managed Services by Anchor

A most unusual temporal issue

By | Technical | No Comments

Most of us have suffered time-related woes, dealing with servers that just can’t keep their clock straight even with NTP. Well we’ve just come across a new one, and NTP isn’t going to save the day – what happens when you have multiple misbehaving clocks? We first noticed the problem when reloading the firewall on the host, a KVM server that had been virtualised from hardware. We use filtergen for ruleset generation, and one of its postreload scripts restarts fail2ban, an anti-bruteforce tool. One part of fail2ban’s initscript makes a call to sleep 1 to give things time to settle. We noticed that the invocation of sleep was taking more than a second, much more. Something wasn’t right and the automatic 15sec rollback was kicking in, having not received explicit…

Read More

An interesting HTTP injection rootkit

By | Technical | No Comments

We came across this story last week, it’s particularly interesting and relevant to us as a webhosting company. The LWN writeup describes a kernel module that maliciously intercepts HTTP requests and injects iframe tags into the HTML. This sort of behaviour isn’t terribly new to us. We see code injection attacks fairly frequently, mostly against PHP-based sites, but they’re usually done through an FTP account with a weak password. This one is interesting because it doesn’t leave obvious evidence on the system. Website code is usually hackishly modified by injecting some obfuscated PHP at the beginning of the file, which often causes visible errors on the site and is very easy to detect. That’s not necessary with this attack because it hooks into the system at a low level, looking…

Read More

Hopping mad: bitten by the leap second bug

By | Technical | No Comments

It’s been an exciting weekend for those running Linux systems, with many popular websites taken offline as a result of a kernel bug relating to the addition of a leap second at the end of June. This follows hot on the heels of an outage that saw many Instagram users tragically unable to upload pictures of their breakfast. There’s no shortage of stories in the media about the problem if you care to look around, but we were rather disappointed by the lack of coverage with much substance. We’ve also not yet found an explanation that can be understood by someone who isn’t intimately familiar with the internals of the Linux kernel. So we put on our detective hat and started looking. Update: if you are familiar with the innards…

Read More

Bugfixing the in-kernel megaraid_sas driver, from crash to patch

By | Technical | 2 Comments

Today we bring you a technical writeup for a bug that one of our sysadmins, Michael Chapman, found a little while ago. This was causing KVM hosts to mysteriously keel over and die, obviously causing an outage for all VM guests running on the system. The bug was eventually traced to the megaraid_sas driver and the patch has made it to the kernel as of version 3.3. As you can imagine, not losing a big stack of customer VMs at a time, possibly at any hour of the day, is a pretty exciting prospect. This will be a very tech-heavy post but if you’ve ever gone digging into kernelspace (as a coder, or someone on the ops side of the fence) we hope it’ll pique your interest. We’ll talk about…

Read More

Keeping your kernel output safe

By | Technical | No Comments

Keeping logs of the operation of your system(s) is really important; when something goes wrong in the middle of the night, a good log can give you all the information you need to diagnose and fix the problem before it happens again. One area of your system that’s quite crucial to keep, but which is often forgotten, is your kernel’s dmesg output. This is all of the messages that come direct from the kernel, from “filesystem mounted” to “Aiee! Penguins on the SCSI bus!” or “lp0: on fire”. While the former isn’t so important to keep for posterity, when the kernel crashes, you really do want to capture that output, but you can’t use your system’s normal logging because when the kernel dies, it usually takes userland processes like syslogd…

Read More