Practical and hands-on hiring for sysadmins

By December 4, 2012Technical

Hiring good staff is hard. As a company that revolves around employing Very Smart People, we know this very well. (And if you’re a Very Smart Person that can get stuff done, please get in touch)

Plenty of companies have written about their hiring processes and the brilliant methods (“tricks” if you will) they’ve found for picking the best candidates and sifting out the rest. They’re a great read, but we think it’s really about finding what works best for you. We think Joel has a lot of great ideas, but we ignore the bits that are specific to hiring coders.

We’d like to share some new stuff that we’ve been doing, including a very practical hiring test that we’re not aware anyone else is doing. We don’t honestly know how well it’ll work if you’re not a pretty-big-Australian-hosting-company-that-does-a-lot-of-tailored-solutions-that-Just-Work-for-customers, but we reckon it’s an improvement on the usual methods for finding people who know what they’re doing and can hit the ground running.

Sieving the field

Just to make sure we’re on the same page, let’s talk process. It’s roughly the same for every company: sift through your applicants and make a hire/no-hire call. Doing this in stages means you can save time by pruning the lesser candidates early on.

  1. Sift the resumes: the tone and style of a resume tends to give us a very good idea of whether the person is what we’re looking for – appropriately styled writing (without spelling errors!), attention to detail, clear and to the point.
  2. Phone screening: phone interviews are scheduled for promising candidates. As well as giving us an idea of their strengths and weaknesses, it also lets us assess their oral communication skills, which will be used on a daily basis.
  3. In-person interview and testing: the ultimate decider, no surprises here.

Practical testing

If a candidate makes it past the phone screening and they’re applying for a technical role with experience (Level 2 Support or similar), this is where the fun begins. To test their real-world prowess we’ve developed a few little obstacle courses to challenge them before the face-to-face interview.

These are mostly normal linux systems, our bread and butter, with a bit of a twist. They’re slightly broken in various ways, and the candidate is given some realistic scenarios to resolve. They’re based on typical support requests that we’ve received in the past, nothing too mean.

To ensure we can get some measurement of ability, the requests start easy and ramp up in difficulty and complexity. The initial tasks, like modifying Apache’s config or installing some packages, should be a piece of cake for anyone with a little sysadminning experience. Candidates who can tackle the toughest challenges will have seen it all before and can spot the telltale signs of problems.

Administering the test

One of the important things about the obstacle course is that it’s an open-book test – the candidate is free to proceed at their own pace and use Google as much as they like. The time limit will sort out candidates that can’t get the job done in a timely fashion.

If you want to try this for yourself, the technical aspects are very simple. We built a couple of vanilla Debian and RHEL VMs, with a standard set of packages, then went about carefully breaking things. To test a candidate, we simply make a snapshot clone of one of the VMs and setup a sudo-capable account for them. Once they’re done, we can login and inspect their work, then discard the snapshot altogether once assessment is completed.

Getting results

As part of the testing, we don’t just look at whether they did the job correctly. We’re also interested in how they got there, and what sort of process they followed. Fixing a broken server by debootstrap'ing a new OS install into a chroot started from init, while technically brilliant, is not the most maintainable solution, and we’d be asking some very tough questions in the follow-up interview.

We’re also looking, again, for communication skills. The candidate will need to respond to the customer’s requests via email, and make notes about what they did so that other sysadmins can pick up where they left off if need be.

So far we’ve had some very good results from candidates running the obstacle course. It tests real skills and provides a lot of insight into how people think and work, and gives strong evidence as to whether we’re dealing with something who can really Get Things Done.

How to spot a no-hire candidate

Let us tell you about the candidate who was asked to fix a broken website using a certain rather popular CMS. The candidate whose resume looked good and passed the phone screening with no glaring problems. The candidate that seemed competent enough, so we gave them the obstacle course.

We had our doubts when we read their reply to the customer – “we’ve decided that rather than work through on the current system, we’ll give you a free upgrade to the very latest system for maximum security and speed”. An unorthodox solution, to be sure, and it would be difficult to demonstrate in the testing environment. Doubly so because the customer is away on holidays and can’t answer questions for you.

Things became much clearer once we read their for-internal-use notes. They’d opted to “cut their losses” early, spent a lot of time pondering the virtues of creating our own hosted platform for a certain rather popular CMS, then left the system in an even worse state than when they started, saying that they were “busy with other things”.

The face-to-face interview would’ve weeded this one out, but at the cost of about 3-4 man-hours. Chalk one up for the obstacle course!

We’re looking for sysadmins, don’t you know? Step right up and try the obstacle course!

Leave a Reply