Cachecade is an LSI technology (Dell PERC cards are rebranded gear) that adds a read-cache tier to the RAID logic, in the form of solid-state disks. While SSDs are still too expensive for mass-scale primary storage, they’re cheap enough that you can burn a few hundred bucks and get 50gb worth of faster reads.

The real benefit of this style of read-cache should be for random block reads, where SSDs proverbially drop excrement over rotational media from a great height. The jury is still out for us – we’ve just started using cachecade on a couple of VM hypervisors and a customer DB server, but we’re hoping to see some noticeable impact even on a qualitative basis.

In truth, the performance improvements will be difficult for us to quantify on our own workloads. You can apparently get this functionality if you purchase the new LSI® MegaRAID® CacheCade™ Pro 2.0, but I’d bet that it’s not exposed through something sane (like SNMP) and you’ll be forced to use the perennially-awful MegaCLI tool to get at the data.

hay guyz, check this out, I got a bunch of old 64mb thumb drives and made a RAID out of them! now i can put all my pr0n on there roffle lolololll

RAIDed floppies? It’s been done. RAIDed tapes? Yo dawg, that’s an enterprise storage solution! Let’s talk seriously now.

I have a fileserver that my family uses, it’s just a box with a couple pairs of hard drives in it (RAID-1, thank you very much. None of this starving-student crap with an oddball assortment of drives in RAID-0). Given that the box is used exclusively for serving up SMB shares, the OS installation is tiny.

I could’ve gone with something really stripped down and optimised, but that would require effort; sysadmins are allergic to unnecessary effort. Instead I just installed Ubuntu jaunty via netinst. Laugh all you want, but I have better things to do, like sleep.

Close-up of chikage's OS drives

The old system was whining about missing one half of its RAID-1, so I decided to splurge on a pair of 4gb USB flash disk – the princely sum of $22 for the pair. I setup the md software raid volumes ahead of time, which were happily picked up by the ubuntu installer – 512MiB /boot partition and the rest handed off for LVM to manage.

I could bore you with a bunch of details, but who cares about that.

• Does it work? Yes, albeit a bit slower during bootup – total boot time from power-button to login prompt is 90 seconds.
• Does the RAID work? Nicely, thank you. You can yank a drive out and it’ll keep ticking along.
• Is there enough capacity? Plenty, the OS filesystem is 44% full.
• Won’t swapping kill it? Yes, maybe eventually. The system has 1GiB of RAM, more than enough when you consider it’s only really using about 100MiB. At least there’s a chance both drives won’t fail at exactly the same time, so I can replace one.
• Am I taking backups? Of course! If it toasts itself it’s not big deal.

What next? Hmm, if I splash out I could buy another pair of flash disks and kick it up to RAID-10 for a performance boost!

Actually, hold that thought for a moment.

I’ve got a couple of servers colocated at the datacentre, the newer of the two is an IBM x3650 named yumi. I chose the IBM because it was on special at the time, and has the capacity for expansion that I desired – it’s a 2RU box with room for two Xeons, 12 FB-DIMM RAM slots and six hot-swap hard drive bays (eight if you choose the 2.5″ option).

There are a number of things I like about the IBM compared to the Supermicros that I’ve worked with so far. The documentation doesn’t suck and the hardware is very nice to work with, almost entirely tool-less. The rackmount rails are an absolute joy. On the downside, yumi weighs about 20kg – hefting that much server over your head and onto the rails is no mean feat.

Thing’s haven’t been entirely without problems and annoyances, of course. One of the reasons to buy hardware from BigCorp is support. There’s much to be said for being able to get someone on-site to fix or replace your troublesome hardware when something goes wrong. At Anchor we’re not too concerned about that. We keep more than enough spare hardware on-hand to deal with failures, and this is just a factor of the way we run things. No hardware vendor can give us instant turnaround (we can be on-site in 10-15min), and the kinds of failures we have deal with are best dealt with ourselves.

Likewise, my colocated server is for my own use, so I’ve no need for vendor support. One of the downsides then of buying from BigCorp is that you tend to get locked-in to their way of doing things. I was planning to install a second CPU that I’d picked up through retail channels, only to discover that the heatsink is entirely non-standard, and that the motherboard requires a separate Voltage Regulator Module (not the case with Supermicro servers). There are good reasons for this, and I’m sure big enterprises don’t much care about the cost, but a $400 price difference for me was galling.

Don’t even think about buying non-IBM hard drives either – the server doesn’t come with empty hot-swap drive sleds, just blanking plates to cover the front of the chassis. Ebay provided a way out of this one, but it was something of a hassle getting them shipped over from the US. Joy of joys, IBM also uses torx screws on the drive sleds. It looks like I’m not the only one who thinks IBM engages in deviant sexual behaviour either.

After the purchase of an overpriced Xeon E5420 and half a dozen drive sleds, I turned my attention to the disk subsystem itself. I wanted a hardware RAID card, and lo and behold, there’s one available. A little more money later, and I’ve got myself a RAID-10 array across half a dozen drives. Of course it’s not written anywhere, but the controller is a specialised piece of kit made by Adaptec (it uses the aacraid driver in Linux). We’ve had some mixed experiences with Adaptec hardware at Anchor, but I don’t have much choice either (the card with it’s battery-backed cache RAM sits in a DIMM slot on the motherboards… huh?).

A couple of weeks ago yumi was rebooted for a kernel update, and then promptly failed to come up. After much prodding I noticed that the RAID card wasn’t bootable, its boot BIOS had disappeared entirely from the usual POST process! My solution was to install GRUB to a USB stick – this is sufficient to kickstart the process and then pass control to the main drives, which otherwise seem to be working fine. Needless to say, I’m rather unimpressed. I’ll probably get around to making a support request eventually, but the fact is that I’m pretty lazy. Indirectly, this is a great argument for letting other people deal with server hardware, rather than colocating your own kit.

There’s two predictable responses here, as I see it:

1. “IBM are bastards!”
2. “That hardware’s not for you“

You get to choose. I just think it’s a learning experience, and now I’m curious about some HP hardware… I expect I’m in for some more learning.

(actually, there’s another possible response: “why don’t you hack up a solution yourself? it’d be cheap, too”. anyone who’s worked with computers for a long time knows that the last thing you want to do is faff around with computers. I’d much rather be out taking photos, sewing, changing the world, etc.)

With a RAID array in normal operation we’re in a pretty safe place. We know that we can suffer failure of a drive without loss of data or disruption of service. Once a drive has failed however we’re in a slightly more precarious position. Loss of another drive or damage to the remaining drive could easily cause major problems. At this point the only thing that can protect you can against data loss if you make a mistake is your backups – you did configure backups didn’t you?

Restoring a damaged RAID array is a task that requires extra caution. 

On our range of dedicated servers and vps‘ it’s one of those things that just happens automatically and the client usually only finds out after the problem has been fixed. For our co-location customers however it’s a task that we often find ourselves involved with to lend a helping hand.

With this in mind we’ve started to put together a series of articles discussing the steps we take to restore a Linux RAID array after hard disc failure and recoving from a Windows software RAID failure We hope you find them useful.

Hard drive failure is a fact of life when you run servers, by sheer virtue of that fact that you have hundreds of them. To mitigate the risk and reduce unscheduled downtime, we use Window’s built-in software RAID feature. It’s not an enterprise solution, but it gets the job done. What’s important is staying online and not losing data.

Did I mention that trying to monitor a Windows box is a nightmare? A colleague of mine wrote a script to allow us to keep a watchful eye on Windows RAID volumes, it’s a lifesaver. A recently-deployed machine got a broken mirror, which we were able to act on immediately. We removed the dodgy mirror and prepared a replacement (we always have plenty of spares, of course). Allow me now to re-enact this scene…

Windows (sounding almost efficient): The driver has detected that device \Device\Harddisk1\DR9 has predicted that it will fail

Sysadmin: Thanks, Windows, I’ll get right on that. You didn’t say whether that was SMART, or just voodoo, but whatever, it’s good to know.

The bad drive is removed and a replacement installed in the hotswap drive bay

Sysadmin: Okay, Windows, do your stuff. “Scan for new hardware”, please.

A pause.

Sysadmin: Ahem, Windows, “Scan for new hardware” and find my drive.

Windows: ‘Ey there, chaps. Do what now, you say? AIEEEEGRH!!

The server stops responding entirely, necessitating a touch of the reset button

Needless to say, we’re rather unimpressed, and have to call the customer to let them know why it’s just dropped offline.

A quick check of the logs is in order. It’s also frustrating that there’s no sane way to scroll through log entries in Windows with something like a text editor, or to “tail” a log as it’s updated in realtime.

09:36 – The previous system shutdown at 9:21:23 AM on 8/10/2008 was unexpected.

Okay, it went down at 09:21, which is correct. Now if we look back in time a little…

09:21 – dmio: Harddisk1 write error at block 1953524618 due to disk removal

*sigh* And this is after the disk was removed cleanly…