Click Here, You Idiot

Published March 17th, 2009 by Lachlan Cooper

At Anchor we’re mostly systems administrators, not web developers, so the content of the sites we host isn’t usually any of our concern. Occasionally, though, a particular site design happens to catch someone’s eye and provides a chuckle for everyone in the office.

As a sort of counterpoint to our previous post about good design, there’s a certain popular… style… of website design that will probably be quite familiar to some of our readers. It’s really best described by providing a (very entertaining) example:

http://www.clickhereyouidiot.com/

1
Comment

Using browser search keywords

Published March 17th, 2009 by Lachlan Cooper

One thing I spend a lot of my working (and playing) time with each day is my trusty web browser. Naturally I want to get things done as efficiently as possible, so it makes sense to reduce the time I need to spend on the most common tasks. One of these is searching.

Traditionally, a search engine was used by typing in the site’s address or clicking a bookmark, waiting for the page to load, clicking to select the search field, typing the query, and hitting Enter. That’s two separate page loads, and a whole lot of unnecessary mouse movement and clicking or typing.

More recently, browsers have started providing a built-in search box, into which you can enter a query and get results with only one page load. That’s a big improvement, but the box takes up a fair amount of useful screen space. If you use more than one search engine, constantly clicking to select between them also makes the process a lot slower than it could be.

Fear not: there is a solution which reduces the time taken for the entire search process to just a second or two! Mozilla Firefox calls this feature smart keywords.
Read the rest of this entry »

0
Comments

Filebucketing to the MAXXXXX

Published March 12th, 2009 by oliver

Every now and then we see an example of application failure so astounding it literally brings tears to our eyes. We have a client whose legacy application is unfortunately still running on an ancient version of Oracle Weblogic and which must be maintained until the new, flashy .NET version of their site is complete.

We were alerted this morning to a problem with some of the Weblogic content – the pages were timing out. Diagnostics were fairly fruitless – packet captures showed nothing useful, and the logging from Weblogic left much to be desired. We started considering more outlandish possibilities such as I/O load causing issues, recently applied updates and so on. Even rebooting was considered (given it is running on Windows).

The first clue of note was the open file list from the Weblogic processes – one such example stood out:

C:\weblogic\state\Sa0V\b1gR\O1Ok\WqYN\9kiv\IQT2\SHGx\C3ri\aE1z\L1YH\X5QW\
gdkB\B2PB\pPPw\uHDK\p1a7\I0l5\94sU\kQ43\+533\5517\5738\7484\6253\_-10\
6273\1519\_6_8\888_\8888\_700\2_702_8\888_

For the sake of your screen, I have manually wrapped this Godzilla-like filename.



Perhaps you are familiar with file bucketing already, but if not, typically the directory structure used will have a relatively sane scheme for locating files and only extend a few levels deep. What we saw in this instance was a completely new breed of monster. Admittedly the absolute path of this file is less than 200 characters out of a limit of more than 32,000 but the naming strategy and depth of the structure has us flummoxed.

But this was only the tip of the proverbial iceberg. When we requested Windows to show us the properties of this state folder it took over an hour to completely calculate the file and folder totals, and the result is impressive:

Web logic makes efficient use of the filesystem

Web logic makes efficient use of the filesystem

Yes you read that right – over 10 million nested directories. By this stage we had already moved the state directory out of the way and created a new one, and restarted Weblogic. It seemed happy and quite responsive after that. My suspicion is that someone developing this application at some point ran into a limitation with their filebucketing algorithm, and resolved to solve the problem once and for all, evidently by making it possible to efficiently filebucket every file in the known universe.

0
Comments

Tales of Hardware – IBM RSA-II slimline

Published March 10th, 2009 by Barney Desmond

In a recent post I mentioned that there are some nice things about using Supermicro hardware here at Anchor. There’s a bit of a dark lining to that silver cloud, however – we’ve had the worst luck trying to get their IPMI stuff to work in any sane way.

IPMI is short for Intelligent Platform Management Interface. Different companies have different names for the technology – Dell has this in their DRAC cards, HP calls this iLO, Sun calls it ALOM. IPMI basically gives you access to a bunch of diagnostic information and management controls for the server. The real killer feature is remote console. Because IPMI is largely independent of the rest of the system, you can get unimpeded access to everything on the server, regardless of what state it’s in. Just like The Matrix, you need to experience how great it is when you rescue a downed server from a BSOD, or a dopey prompt holding up the BIOS while booting.

You see that? IPMI just saved you from:

  1. Walking all the way to the datacentre
  2. Going through six proximity card doors and an airlock
  3. Getting to your suite
  4. Opening the racks
  5. Finding the server
  6. Hooking up a keyboard and monitor
  7. Hitting F1 to get past that pesky “keyboard not found” error
  8. Walking back to the office – congratulations, you just wasted your lunchtime

Read the rest of this entry »

0
Comments

WordPress 2.7, now with fewer absurd bugs

Published March 10th, 2009 by Barney Desmond

I went ahead and upgraded the installation of wordpress we use for this blog from 2.6 to 2.7 – you won’t notice anything mind you, but we get a completely different admin interface under the hood. Keeping things up to date is always a good idea from a security standpoint, but I also wanted to address an odd issue that wasn’t present in my own personal installation of 2.7.

I’d noticed a little while ago that the font-colour controls in the editor didn’t seem to work. I could select the text and apply the colour, but the change disappeared once I saved the changes. Looking at the HTML, something odd was afoot:

<span style="#990000">lorem ipsum dolor</span>

Definitely not the expected behaviour, the “color:” was being stripped out of the style attribute. A little further searching revealed the treachery: wp-include/kses.php

kses.php performs data sanitisation, mostly to do with handling dodgy/exploitable code in posts and comments. The editor implements colours in the expected manner using inline CSS, but the sanitiser strips it out! I imagined that surely this was well-tested, but trawling google for likely problem reports was largely fruitless.

There were a couple of posts from people who’d run into this same issue, but the fix was largely unsatisfactory.

Why unsatisfactory? Because you add the desired css attributes to an array in kses.php called “$allowed_protocols” – a rather nasty hack to my mind. In WordPress’ defence, I didn’t perform a vanilla installation and verify the issue was still there with no user modifications – it would have been substantial effort for a problem that could be fixed by keeping up to date anyway. No hard feelings, guys.

0
Comments

Tales of Hardware – IBM x3650

Published March 10th, 2009 by Barney Desmond

All the servers Anchor buys are from Supermicro. Most people won’t have heard of them, but they’re a sizeable hardware vendor that also does some OEM gear. Supermicro certainly doesn’t carry the mindshare of other big brands like HP, Dell, et al., but we chose them because their stuff is reliable and affordable – we focus on the things that actually matter, rather than some enterprise-y idea of sticking with big brands that you trust – “noone ever got fired for buying IBM” they say.

Actually, hold that thought for a moment.
Read the rest of this entry »

0
Comments

Safely handling RAID failure

Published March 9th, 2009 by Davy Jones

With hard discs being by far the most common point of failure in servers RAID does wonders for protection against loss of data.

With a RAID array in normal operation we’re in a pretty safe place. We know that we can suffer failure of a drive without loss of data or disruption of service. Once a drive has failed however we’re in a slightly more precarious position. Loss of another drive or damage to the remaining drive could easily cause major problems. At this point the only thing that can protect you can against data loss if you make a mistake is your backups – you did configure backups didn’t you?

Restoring a damaged RAID array is a task that requires extra caution. 

On our range of dedicated servers and vps‘ it’s one of those things that just happens automatically and the client usually only finds out after the problem has been fixed. For our co-location customers however it’s a task that we often find ourselves involved with to lend a helping hand.

With this in mind we’ve started to put together a series of articles discussing the steps we take to restore a Linux RAID array after hard disc failure and recoving from a Windows software RAID failure We hope you find them useful.

0
Comments

Advanced web application monitoring

Published March 6th, 2009 by Davy Jones

We’ve been using Nagios to monitor an ever-increasing number of services on all of the servers that we own at Anchor for a number of years. For the most part the things we monitor have a focus on those that a systems administrator (us in other words) has to deal with. This includes things like CPU load, memory usage, disc space availability, swap usage, server load, availability of core applications such as web servers, data base servers, mail servers. On a given server we typically monitor anywhere from 5 to 25 different attributes.

The end goal of all this monitoring is to ensure that the services on the servers we run are always working.

We can take this a step further though, rather than just monitor the components of the server that are required to keep the websites running, we can monitor in quite detailed ways many of the components of the websites themselves.

At the end of the day, having a monitoring system tell you that a server is healthy and all of the applications are working only goes so far. Ultimately what’s important (to our clients) is that the website is behaving the way that they expect it to.

Since we don’t build any of the websites that we host unfortunately we can’t put in place systems to monitor the innards of an application. To do so requires an intimate knowledge of how the application was built. We do however have a very powerful monitoring system and if the developers of the websites put the hooks into their code, we can monitor these hooks so that both Anchor and the developers can be alerted to the problems.

With these hooks in place, in many cases Anchor will be able to fix the problems, but if we can’t at least we notify the client that there’s a problem (even if it is at 2am in the morning).

In our world, the more monitoring we put in place the greater the uptime of services and the happier our clients are. On this one though, we need our clients’ help. For all Anchor customers on a Fully Managed support pack we do our side of the monitoring free of charge, and for everyone else we can do anything – for a small fee of course.

To help people understand what can be achieved with web application monitoring along with some implementation ideas, we’ve put together this article on website monitoring.

0
Comments

Wanted: Full-time Customer Support Guru

Published March 4th, 2009 by Davy Jones

Due to recent growth Anchor is looking for full time customer service guru for immediate start, please find full details of the job listed in-line below:

Anchor Systems is a leading Linux based web hosting provide in the Australian market. We are growing organisation and require intelligent, passionate people who want to learn everything there is to know about Linux system administration and web hosting in general.

Our customer service representatives:

* show initiative and are self motivated who are capable of working in a dynamic fast-paced environment that requires quick learning and intelligent solutions

* have an excellent command of the English language as well as good phone manner

* get a buzz out of assisting end users

* are keen, creative and able to accurately identify problems

* endeavour to provide the absolute best possible customer experience

* have a little experience with either Linux or Windows problem diagnosis

* are familiar with end user applications such as Outlook, Entourage, Internet Explorer, Firefox, WS-FTP, CuteFTP, SSH

* have an attention to detail bordering on obsessive compulsive and have an attitude of doing things right, the first time

Responsibilities in this position include:

* Order processing: Domain name registration, renewals and transfers. Adding DNS records, provisioning of new shared hosting accounts, managing SSH/FTP accounts, addition of databases and users. SSL certificate procurement.

* First Level technical support: end user email configuration, updating DNS records, password resets, follow up calls, FTP/SSH support as well as field other general support enquiries.

* Basic system administration: Webserver configuration, basic changes to apache/IIS configuration. Database administration, covering both MySQL and PostgreSQL including user account addition, database
imports and exports. Mail configuration on both postfix and sendmail.

* career opportunity to move into a Linux system administration role
* casual work environment
* dual head workstation
* pool table, foosball table and beer.

If you feel you would be suitable for this role, Please send your CV preferably in plain text, PDF or if absolutely necessary, word format to jobs@anchor.com.au

1
Comment