Tag

crash Archives - AWS Managed Services by Anchor

Inquisitio interrupta

By | Technical | 2 Comments

A customer’s Django app has been giving us hell for a little while now, something we’ve recently tracked down to dodgy signal-handling in some MySQL library code. Despite only showing up a dozen times in every 600 million queries or so, we’ve nailed it! It turns out the bug has been hanging around, on and off, for the better part of ten years now – that’s a long time! While it sounds simple on paper, it’s gone unfixed for so long because it only manifests in very specific conditions. Arriving at this conclusion was something of a surprise, but definitely worthwhile, so come join us on an adventure with surprises at every turn. The problem was first noticed by the customer, they get a mail when an exception goes uncaught….

Read More

Bugfixing the in-kernel megaraid_sas driver, from crash to patch

By | Technical | 2 Comments

Today we bring you a technical writeup for a bug that one of our sysadmins, Michael Chapman, found a little while ago. This was causing KVM hosts to mysteriously keel over and die, obviously causing an outage for all VM guests running on the system. The bug was eventually traced to the megaraid_sas driver and the patch has made it to the kernel as of version 3.3. As you can imagine, not losing a big stack of customer VMs at a time, possibly at any hour of the day, is a pretty exciting prospect. This will be a very tech-heavy post but if you’ve ever gone digging into kernelspace (as a coder, or someone on the ops side of the fence) we hope it’ll pique your interest. We’ll talk about…

Read More

A fix for the browser crashing blues

By | Technical | No Comments

Ever been constructing a verbose and complex reply to an email on a webmail site, only to have the browser crash when you were about to hit send? Or perhaps putting the finishing touches on your Wikipedia edits to the particle physics article (I know I have) and have all of your hard work lost in the blink of eye? Fear not, salvation is at hand! Although many Web 2.0 sites providing text-editing will save drafts and provide a reasonable text editor, nothing beats your favourite text editor of all time, Vim. Or Emacs. Or whatever you would prefer to use. Why shouldn’t you be able to use any text editor you like? It’s your computer and your time after all! If you are using Firefox, your problems are over….

Read More

A tale of two drives

By | Technical | No Comments

It’s no secret that we’d rather be working on Linux than Windows here at Anchor. It is, by and large, much more annoying to actually get anything done, but it also just breaks in opaque and unexplained ways. O Windowes, let me count the ways in which you are broken! This is one such problem we ran into yesterday. Hard drive failure is a fact of life when you run servers, by sheer virtue of that fact that you have hundreds of them. To mitigate the risk and reduce unscheduled downtime, we use Window’s built-in software RAID feature. It’s not an enterprise solution, but it gets the job done. What’s important is staying online and not losing data. Did I mention that trying to monitor a Windows box is a…

Read More