Rather than scrapping them and foisting more e-waste on the environment, we’d prefer to send them to a good home where a geek or hacker can make use of them. If this sounds like you, we’d love you to have a gander at our ebay listings.

To whet your appetite, here’s a sample of what you can expect.

Supermicro Nyanserver 6015P-8R

Supermicro Highroller 6014H-T

Feel free to get in touch if you have any questions about the hardware.

N.B. Nyanserver may not be as nyan as shown, photograph is for demonstrative purposes only.

You can even get a deadtree copy from Amazon. If that doesn’t do it for you, there’s apparently PDFs to be had as well.

[root@tidal90 root]# uptime
15:26:10 up 1080 days, 15:47,  1 user,  load average: 0.00, 0.00, 0.00

We unearthed this beauty because the customer wants to upgrade their service to a VPS. Running Fedora Core 2 with a 2.6.10 kernel, most processes are so old that they don’t have a start-time or start-date, but a start-year, mostly in 2007.

For those interested in the title of this post, When Sysadmins Ruled the Earth is a fun short story by Cory Doctorow.

LEVIATHAN is, I assure you, teh hardk0rez - dual xeon 5500-series, 6gb RAM and 12TB usable storage in RAID-10

I was pushing for PHYREXIAN DREADNOUGHT personally, but LEVIATHAN is acceptable too; the upkeep effort of backup servers is pretty high after all.

Preparation

This should go without saying. The more time you have to prepare for the migration, the better. You do not want to leave it until the last minute. My philosophy when approaching the migration is always to leave the least amount of work possible to do at the time of the actual migration. Clients will generally want to schedule any server downtime for late at night, when you are not going to be operating at your best (despite how many coffees or energy drinks you may have consumed). If you can log in to the machine, run a prepared script which takes of everything and have the migration completed for you, you will end up with a happy client and be happy yourself. You will be in the datacentre for less time and get to bed earlier, both of which are good things.

Make good use of scripting

Following on from my last point, I strongly encourage you to script as much as possible. The migration I just performed entailed moving a server from one datacentre and network provider to another which meant a change in address space. Thus, firewalls, IP address configuration files, Apache vhosts, ACLs and more had to change. Ahead of time I determined which files would need to be modified and created a script which took a backup of each of these files before overwriting them with corrected versions. Any failure would cause the script to stop and print the problem which could be easily diagnosed manually.

The more automation and failsafes you can build into your script, the better. Since you will be creating it with plenty of time up your sleeve and your brain operating at full capacity you can build up the script with your full arsenal of tricks. At 3am in a cold datacentre with noisy airconditioning you can hardly expect to have your full faculties with you, so make life easier for yourself by leaving as little actual work to do at this point.

Fully acquaint yourself with the server

You will only know what needs to be changed on the server if you are familiar with it. Of course, you should have plenty of good documentation already on it but if not, log in and get the lie of the land. Have a plan for how you will find out facts about the system – make use of grep and well structured regexes for finding out configuration details, slocate (if there is a locate database present) for finding critical files, and your usual toolkit of sysadmin techniques.

Document as you go

At Anchor, documentation is critically important. We have an internal wiki system in which we make detailed notes on every server and a great number of technical articles (a lot of which we have shared with you in our public wiki). Every migration plan is carefully documented from start to finish. In more complicated scenarios a full change proposal is created and officially ratified, but at the very least you should create a checklist:

• people involved (and their contact details, if necessary)
• time frame
• a detailed list of items that need to be prepared or information that needs to be acquired before the migration takes place
• actions that will be undertaken just before the migration starts
• the list of actual migration steps, including details of what any scripts will be doing
• post-migration actions which need to be done immediately after the migration – e.g. checking that all your monitoring is showing OK for all hosts and services
• a list of “cleanup” items which can be completed after the migration, but not time critical, e.g. removing stale references to servers from your internal documentation

Have as many people check over your documentation as possible, preferably those who have knowledge of the systems so that they can find anything you have missed. The more eyes on your documentation, and heads thinking about it, the better the chances that you will have a plan that covers all aspects.

One of the most important things from my point of view with documentation is to forward a copy to the client, and keep them involved in the process. Not only does it give them confidence in your abilities to conduct the migration successfully, but it gives them an idea of the work that you have had to put in, gives transparency to the process and gives you another point of view on the migration – there may be other steps important to them which you may have missed for example lowering TTLs on domains that are solely client-controlled.

Keep the client “in the loop”

Following on from my previous point, as well as giving the client a copy of your migration documentation, it is important to let them know what is going on. Send them a courtesy email every day or two, a call or whatever your deem appropriate to let them know how you are going with preparations and any information you need from them.

On the day of the migration, double-check everything with them – times, contact details, the migration plan, and so on. Make sure they are still happy to go ahead and that they are happy with your plans. Give them a courtesy call or message when you are about to start the migration, when you are finishing, but most importantly whenever you have any unexpected problems. Nothing upsets clients more than having things go pear-shaped and not being informed about it. Even if you don’t know what the problem is, let them know that you are diligently working on it and will keep them up to date with developments.

Plan for when things go wrong

In a perfect world, you would prepare adequately and everything would go flawlessly (as it did for me last night, luckily). However every slightly obsessive-compulsive systems administrator knows that things can and will go wrong every now and then despite your best efforts.

Make an escape plan for every point where things can go wrong during the migration. Given you won’t have infinite time available, prepare most for the most likely failure scenarios. Make a rollback plan which will abort the migration, and decide how many failures will cause you to take this rollback plan on the night. Confirm this with the client.

Make sure that no change you make cannot be reverted (which most times will necessitate backups). There is nothing worse than discovering you have irrevocably destroyed data in the process of making a critical change.

Approach everything with an obsessive-compulsive attitude

The best plans will have considered everything and left no detail to chance. It can be tiresome to be painstakingly thorough in your plans, but ultimately it will pay off. At the same time though, you don’t have to do everything in one sitting – make notes in your migration plan on what you still need to do and follow it up later. Don’t foolishly believe you will remember everything on the migration day, or even an hour from now – WRITE IT DOWN!

Remember, even though the preparation may be slightly tiresome, you are just making life easier for yourself at migration time. Hopefully if you follow these general tips I’ve prepared, they will make your next migration a lot easier.

Actually, hold that thought for a moment.

I’ve got a couple of servers colocated at the datacentre, the newer of the two is an IBM x3650 named yumi. I chose the IBM because it was on special at the time, and has the capacity for expansion that I desired – it’s a 2RU box with room for two Xeons, 12 FB-DIMM RAM slots and six hot-swap hard drive bays (eight if you choose the 2.5″ option).

There are a number of things I like about the IBM compared to the Supermicros that I’ve worked with so far. The documentation doesn’t suck and the hardware is very nice to work with, almost entirely tool-less. The rackmount rails are an absolute joy. On the downside, yumi weighs about 20kg – hefting that much server over your head and onto the rails is no mean feat.

Thing’s haven’t been entirely without problems and annoyances, of course. One of the reasons to buy hardware from BigCorp is support. There’s much to be said for being able to get someone on-site to fix or replace your troublesome hardware when something goes wrong. At Anchor we’re not too concerned about that. We keep more than enough spare hardware on-hand to deal with failures, and this is just a factor of the way we run things. No hardware vendor can give us instant turnaround (we can be on-site in 10-15min), and the kinds of failures we have deal with are best dealt with ourselves.

Likewise, my colocated server is for my own use, so I’ve no need for vendor support. One of the downsides then of buying from BigCorp is that you tend to get locked-in to their way of doing things. I was planning to install a second CPU that I’d picked up through retail channels, only to discover that the heatsink is entirely non-standard, and that the motherboard requires a separate Voltage Regulator Module (not the case with Supermicro servers). There are good reasons for this, and I’m sure big enterprises don’t much care about the cost, but a $400 price difference for me was galling.

Don’t even think about buying non-IBM hard drives either – the server doesn’t come with empty hot-swap drive sleds, just blanking plates to cover the front of the chassis. Ebay provided a way out of this one, but it was something of a hassle getting them shipped over from the US. Joy of joys, IBM also uses torx screws on the drive sleds. It looks like I’m not the only one who thinks IBM engages in deviant sexual behaviour either.

After the purchase of an overpriced Xeon E5420 and half a dozen drive sleds, I turned my attention to the disk subsystem itself. I wanted a hardware RAID card, and lo and behold, there’s one available. A little more money later, and I’ve got myself a RAID-10 array across half a dozen drives. Of course it’s not written anywhere, but the controller is a specialised piece of kit made by Adaptec (it uses the aacraid driver in Linux). We’ve had some mixed experiences with Adaptec hardware at Anchor, but I don’t have much choice either (the card with it’s battery-backed cache RAM sits in a DIMM slot on the motherboards… huh?).

A couple of weeks ago yumi was rebooted for a kernel update, and then promptly failed to come up. After much prodding I noticed that the RAID card wasn’t bootable, its boot BIOS had disappeared entirely from the usual POST process! My solution was to install GRUB to a USB stick – this is sufficient to kickstart the process and then pass control to the main drives, which otherwise seem to be working fine. Needless to say, I’m rather unimpressed. I’ll probably get around to making a support request eventually, but the fact is that I’m pretty lazy. Indirectly, this is a great argument for letting other people deal with server hardware, rather than colocating your own kit.

There’s two predictable responses here, as I see it:

1. “IBM are bastards!”
2. “That hardware’s not for you“

You get to choose. I just think it’s a learning experience, and now I’m curious about some HP hardware… I expect I’m in for some more learning.

(actually, there’s another possible response: “why don’t you hack up a solution yourself? it’d be cheap, too”. anyone who’s worked with computers for a long time knows that the last thing you want to do is faff around with computers. I’d much rather be out taking photos, sewing, changing the world, etc.)