Well, actually, it’s the sort of pony you probably don’t want, but it’s something of a “pour encourager les autres” sort of thing: the 2008 pwnie awards.

http://royal.pingdom.com/2008/11/14/the-worlds-most-super-designed-data-center-fit-for-a-james-bond-villain/

This story’s been around for a little bit, but it popped up in the geeky media a few days ago. Seeing it, we were inspired to share our own little piece of home with you.

Rent prices for inner-city office space have been rising steadily over the last few years, and smaller businesses in particular are starting to feel the squeeze. Rather than sitting around hoping for the real estate economy to cool down a bit, we made the decision several months ago to move. We’ve handled a few office moves since we opened up shop in 2000, most of them have come off without a hitch and (more importantly) with no impact to our support operations.

As much fun as moving is, doing it every few years is a hassle. This time around we wanted to move in somewhere that’d (hopefully!) last us for quite a while with room to grow.

It’s probably not fully apparent from the schematic, but what we’ve moved into is actually a converted church. The styling is very tastefully Gothic, with high ceilings and plenty of usable space. Sorry there’s no photos yet; the place is embarrassingly messy, and there’s crates everywhere! For reasons that aren’t really worth explaining, we call it The Chalet.

To save you the effort of figuring it out for yourself, the nave is roughly nine metres wide and 33 metres from end to end; this comprises our main work and rec areas. The transept measures 24 metres east-to-west across the nave, and provides useful storage and amenities. Attached just off the northwest corner is a spire, which off-duty staff can bunk in.

With its position on naturally elevated terrain, the east-facing side has a good line of sight to our datacentre. We’ve decided to take advantage of this to install our own microwave antennae for direct connectivity to our core network; giving us a very reliable, high speed service. We’ve kept our ADSL2+ connection with Internode for redundant connectivity, but we’ve not yet been hit with such nasty weather than the wireless link was unusable.

One of the nice things about the location is that it’s quiet. The thick stone walls provide very effective isolation, and being out of the CBD means you dodge the continuous traffic noise. Also, an open-plan office can easily suffer from being too cramped and noisy, like pubs where you have to shout to be heard by the person next to you. High, vaulted ceilings make that a non-issue here, lending a very soothing and peaceful atmosphere.

For the curious amongst you, we’ve decked the place out much to our liking. Something of a cyberpunk-goth aesthetic for lack of better description (everything you’ve heard about sysadmins shying away from light, it’s true). This doesn’t mean we don’t know how to have fun, of course. We’ve brought the pool table and foosball kit along, and with any luck there’s a pinball machine in the pipeline.

One of our dedicated server customers recently had a problem with the machine keeling over and dying for a few days in a row, for no apparent reason. This necessitated a remote reboot of the server to get it running again (we cut the power to both power supplies for a few seconds). The immediate suspicion was faulty hardware, but this should rarely be the case as we put our hardware through a thorough “burn in” period before it’s ever deployed. In addition to this, it was happening pretty regularly in the middle of the day.

After spotting this pattern, a quick look at our trending graphs showed us the problem very clearly. The machine was steadily using all available physical memory. Once this ran out, the system starts pushing data off to swapspace, which is on the disks. Once this runs out the OS simply has no more memory to give. It’ll start killing off processes in a desperate attempt to reclaim some memory, but this is rarely successful.

(more…)

Some of the problem-solving we do has a forensic component (not in the legal sense). When something breaks down, we perform what’s known as Root Cause Analysis. It’s well and good to fix the symptoms so things are running again, but it’s more important to fix the problem so it doesn’t happen again.

Often enough, we’ll login to a user’s account and check out what they’ve run recently. They might have edited a config file, removed some necessary wrapper scripts (this happens on our shared hosting servers), or blindly duplicated something that really shouldn’t be copied. Whatever the case, these are good places to start looking when you need to fix something.

(more…)

We’ve discovered some interesting things about Windows, and they never fail to cause some head-scratching. We had cause to go rooting through a customer’s wordpress installation recently to hunt down the cause of PHP errors, and discovered two WTFs here.

The first was the breakage of various scripts in the wp-admin directory. Through means unknown, every array definition was broken by the addition of a file path. If you grok PHP, you’ll recognise that this isn’t syntactically valid:

$defaults = array(

'show_option_all'../../../wordpress/wp-includes/ => '',
'show_option_none'../../../wordpress/wp-includes/ => ''

);

Python is our preferred in-house language, but breadth of knowledge is more important for a sysadmin. Cleaning up the PHP was a snap, but it’s a mystery as to how this happened in the first place; according to the customer it “just stopped working”. It looks a bit like someone got busy with a site-wide find-and-replace. This isn’t implausible, but it seems far less likely given that this is on a Windows machine.

(more…)

Recently we had a client on one of our webservers doing something which we consider to be bad form, inconsiderate and downright annoying – running scripts from cron every five minutes which take longer than five minutes to run. The end result? Processes that pile up, consuming resources and frequently inconveniencing other customers (if it is running on a shared webserver).

If the scripts in question are competing for resources, you can frequently end up in a dead-lock or live-lock situation as well. It seems some developers are blissfully unaware of how to write their scripts with enough safety features built in. Fortunately, we at Anchor have encountered these situations enough times to know how to best combat them.

In this scenario our client was using Perl in scripts which measured the response time of remote websites, then logged the data and presented graphs for statistical and trending analysis. OK – this is a wheel that has been reinvented far too many times but that is beside the point. Not only that, but since they were running in series, one slow-responding website in the config file could slow down the entire run and cause the script to take longer than the allotted 5 minutes.

The developer offered to look at the script and improve its efficiency, but we still wanted a failsafe mechanism to stop processes piling up – regardless of the tweaks the developer would make. For this, the Sys::RunAlone Perl module is very handy:

#!/usr/bin/perl
import Sys::RunAlone

# your code body goes here

__END__
# END OF FILE

It’s really that simple. You just need to install the Perl module from CPAN or your method of choice. With the code above in your script, Sys::RunAlone ensures through its locking mechanisms that your script can only ever run a single instance at a time, thus saving webservers everywhere from potential devestation.

If you have hundreds of dedicated servers in a remote datacentre, and a need to operate on the console of some of those servers on a semi-regular basis (and I KNOW you do), then you’ll understand the frustration of having to physically put someone in front of those machines. You need to take into account travel time, and waiting time while the server is doing anything until the next point it requires input from the human.

This can all be very frustrating and time wasting, since whoever is designated to operate the console is taken away from their regular tasks, and datacentres being what they are it is unlikely they’ll be able to make the best use of their time by multi-tasking with other jobs.

Enter the Wireless IPKVM. No doubt you are familiar with KVMs, and IPKVMs are simply a network-operable extension of that concept. Add a Wireless bridge to the equation and you have a truly portable and remotely-accessible KVM! There are integrated Wireless IPKVM solutions available, but we found that they relied on a proprietary protocol, required a matching transceiver pair, didn’t support standard 802.11g or WPA2 or they were just too expensive.

Here we have the following:

• JayCar kit box
• AdderLink IPEPS IPKVM
• Linksys Business Wireless Bridge

The wireless bridge requires its own power supply unless you are using Power over Ethernet, so we included that in the kit box. The IPKVM is powered from either PS/2 or USB connections to the server you connect it to. We have a Cat5 cable connecting the IPKVM to the wireless bridge; sadly I couldn’t find a shorter cable at the time but it also acts as a spacer between the top of the wireless bridge and the kit box.

The pieces of foam you can see in the picture act as spacers between internal components and the kit box. Once the lid is on, none of the components can move around in the box. We also drilled some cooling holes in the box around the wireless bridge, and made openings for the connectors to the IPKVM and wireless bridge power adaptor. The wireless bridge antennae stick out at the back through another couple of holes. Due to the space constraints I had to allow the wireless bridge power plug and RJ45 connector to stick out a little – in future revisions I may get a slightly larger kit box but there isn’t a perfect size available unfortunately.

Configured to connect to our WPA2-protected wireless network, we were able to log into the IPKVM instantly. Response time is good, and the only gripe at the moment is that if you remotely power cycle the machine it is connected to, the IPKVM is rebooted also. Without including another power supply for the IPKVM in the kit box, we will have to live with this however.

Now that we have had success with this little project, in the next revision I may make some of the following improvements:

• take all the covers of the internal components for space saving and improved cooling
• align the antennae in the middle of the case so cables coming from the wireless bridge are hidden
• reuse PCB mount points to screw the boards into the kit box, or use some sort of glue
• unified power supply for the wireless bridge and IPKVM

Here are a few photos of the finished product with the lid on: