How not to handle errors: phpmyadmin edition

By | Technical | No Comments

The user does something that fails, due to server-side problems, and you wish to inform the user of this problem so it can be fixed. You redirect to a page with a URL something like /error.php?message=Something+has+gone+wrong.+Please+fix+it+and+try+again The user fixes the error on the server, and (since staring at the location bar isn’t a whole lot of fun) simply hits ‘reload’ in their browser. This causes much wailing and gnashing of teeth, as the problem apparently persists. I’ll give phpMyAdmin partial credit for at least HTML escaping their errors, but… the stupid, it BURNS!

Read More

A new home for the Anchor NOC

By | Technical | One Comment

http://royal.pingdom.com/2008/11/14/the-worlds-most-super-designed-data-center-fit-for-a-james-bond-villain/ This story’s been around for a little bit, but it popped up in the geeky media a few days ago. Seeing it, we were inspired to share our own little piece of home with you. Rent prices for inner-city office space have been rising steadily over the last few years, and smaller businesses in particular are starting to feel the squeeze. Rather than sitting around hoping for the real estate economy to cool down a bit, we made the decision several months ago to move. We’ve handled a few office moves since we opened up shop in 2000, most of them have come off without a hitch and (more importantly) with no impact to our support operations. As much fun as moving is, doing it every few years is…

Read More

Winning the war on PHP memory leakage

By | Technical | No Comments

One of our dedicated server customers recently had a problem with the machine keeling over and dying for a few days in a row, for no apparent reason. This necessitated a remote reboot of the server to get it running again (we cut the power to both power supplies for a few seconds). The immediate suspicion was faulty hardware, but this should rarely be the case as we put our hardware through a thorough “burn in” period before it’s ever deployed. In addition to this, it was happening pretty regularly in the middle of the day. After spotting this pattern, a quick look at our trending graphs showed us the problem very clearly. The machine was steadily using all available physical memory. Once this ran out, the system starts pushing…

Read More

Root cause analysis using shell history timestamping

By | Technical | No Comments

Some of the problem-solving we do has a forensic component (not in the legal sense). When something breaks down, we perform what’s known as Root Cause Analysis. It’s well and good to fix the symptoms so things are running again, but it’s more important to fix the problem so it doesn’t happen again. Often enough, we’ll login to a user’s account and check out what they’ve run recently. They might have edited a config file, removed some necessary wrapper scripts (this happens on our shared hosting servers), or blindly duplicated something that really shouldn’t be copied. Whatever the case, these are good places to start looking when you need to fix something.

Read More

Bug report: “all” does not mean all, for some values of “all”

By | Technical | No Comments

We’ve discovered some interesting things about Windows, and they never fail to cause some head-scratching. We had cause to go rooting through a customer’s wordpress installation recently to hunt down the cause of PHP errors, and discovered two WTFs here. The first was the breakage of various scripts in the wp-admin directory. Through means unknown, every array definition was broken by the addition of a file path. If you grok PHP, you’ll recognise that this isn’t syntactically valid: $defaults = array( ‘show_option_all’../../../wordpress/wp-includes/ => ”, ‘show_option_none’../../../wordpress/wp-includes/ => ” ); Python is our preferred in-house language, but breadth of knowledge is more important for a sysadmin. Cleaning up the PHP was a snap, but it’s a mystery as to how this happened in the first place; according to the customer it “just…

Read More

The long running process dilemma

By | Technical | No Comments

Recently we had a client on one of our webservers doing something which we consider to be bad form, inconsiderate and downright annoying – running scripts from cron every five minutes which take longer than five minutes to run. The end result? Processes that pile up, consuming resources and frequently inconveniencing other customers (if it is running on a shared webserver). If the scripts in question are competing for resources, you can frequently end up in a dead-lock or live-lock situation as well. It seems some developers are blissfully unaware of how to write their scripts with enough safety features built in. Fortunately, we at Anchor have encountered these situations enough times to know how to best combat them. In this scenario our client was using Perl in scripts which…

Read More

A Portable Wireless IP KVM Solution

By | Technical | One Comment

If you have hundreds of dedicated servers in a remote datacentre, and a need to operate on the console of some of those servers on a semi-regular basis (and I KNOW you do), then you’ll understand the frustration of having to physically put someone in front of those machines. You need to take into account travel time, and waiting time while the server is doing anything until the next point it requires input from the human. This can all be very frustrating and time wasting, since whoever is designated to operate the console is taken away from their regular tasks, and datacentres being what they are it is unlikely they’ll be able to make the best use of their time by multi-tasking with other jobs. Enter the Wireless IPKVM. No…

Read More

Firewall Hero III: Legends of Packet Filtering

By | Technical | No Comments

Customer support can be fantastically rewarding sometimes. When your combination of skill, tenacity and knowledge produce a solution that’s straightforward and effective, the feeling of satisfaction is hard to match. It doesn’t even have to be something big, we’ll take those small victories gladly. Our work sometimes looks a bit like magic, and we don’t mind one bit. A few weeks ago one of our customers reported problems with streaming media from their server. Clients were taking about 20-30sec to connect, which was of course unacceptable, and they were suspecting something was wrong on our side, perhaps congestion or some over-zealous border firewall. The redundant connectivity we purchase is well above requirements even in the face of failure, and we don’t oversell bandwidth, so the former wasn’t a possibility. We…

Read More

A tale of two drives

By | Technical | No Comments

It’s no secret that we’d rather be working on Linux than Windows here at Anchor. It is, by and large, much more annoying to actually get anything done, but it also just breaks in opaque and unexplained ways. O Windowes, let me count the ways in which you are broken! This is one such problem we ran into yesterday. Hard drive failure is a fact of life when you run servers, by sheer virtue of that fact that you have hundreds of them. To mitigate the risk and reduce unscheduled downtime, we use Window’s built-in software RAID feature. It’s not an enterprise solution, but it gets the job done. What’s important is staying online and not losing data. Did I mention that trying to monitor a Windows box is a…

Read More