That said, I can say with some confidence that we have the pleasure of working with some of the very best software and web developers both locally here in Australia as well as abroad.

Further to this, I can also add quite unreservedly that software developers really don’t make good system administrators.. And can you really blame them?

Allow me to elaborate a little bit here; As you may have already guessed from the above few paragraphs, software development is tough.  Being a good software developer is even tougher. Under the pretty exterior of most websites there an awful lot of work that goes into making the sites work.  Pulling this together requires a fair amount of consideration through-out all aspects of the software development process, from getting requirements and designing the application through to writing the code, testing, debugging and forever trying to squash that final elusive bug.  It takes someone with a fairly specific skill-set to be able to do all this and to do it well.

Something that I’ve noticed however, is software developers are sometimes expected to take on the role of server management and look after the on-going running and maintenance of the machine.  Whilst I can appreciate there’s a similarity between what a software developer and a system administrator does, “hey, they both do ‘computer stuff’”, the tasks which are completed by each roles are worlds apart.  A software developer really only cares about getting his or her application working within a specific environment the quickest way possible.  This can sometimes mean that there are some rather drastic changes to the machine configuration with little consideration to the potentially negative implications. This is pretty understandable,  as far as they’re concerned, once they get the environment working with their application then they can just continue hacking away on their code.  Given they are probably under other tight deadlines or would just simply be preferring to get on with what they’re actually being paid to do without much consideration for the longevity and maintainability of the operating system environment.

This is something we see a lot of; from developers downloading source tarballs then compiling and installing software system-wide to running bleeding edge versions of software which just aren’t suited to being in production.

To give an example of an incident recently which has prompted this post, we had a client call up complaining that they couldn’t get their postgresql database to start. Whilst this was not on our fully managed service, we are always willing to help out or clients on a professional consulting basis.  Upon logging in we attempted to start postgresql and witnessed it failing without too many clues as to what’s doing on. Further investigation revealed the following in the postgresql startup logs:

FATAL:  database files are incompatible with server
DETAIL:  The database cluster was initialized with CATALOG_VERSION_NO 200812281, but the server was compiled with CATALOG_VERSION_NO 200904091.

Further digging revealed that postgresql had recently been updated.. 14 hours ago to be precise. Subsequent to this the database engine had been stopped and then failed to start again. The client in question actually uses this machine as a mail exchange for his clients and uses a postgresql back-end to manage the mail tables.  This means that for the duration of the outage, no email was working for any of the clients on the machine.  Yes, for 14 hours.  Ouch.

Once we had found the problem, all we needed to do was roll back to the previous version start up postgres and everything would be hunky-dory, right? Well.. Easier said than done.

In this case, the software developer had installed what appears to be a development version of postgresql which was (as the error message alludes to) released in January 2008.  That’s ok, we should just be able to reinstall the previous version from the RPM on the machine, right?  Wrong. Didn’t exist.

At this point in time we started to do a quick google and checking the postgresql website to see if they perhaps, just maybe, had a copy of this daily development release somewhere on the website.  No joy there…

I know! We take backups for any clients who chose to use our managed backup solution, and this client has opted for this service!  As part of our managed backups we roll-out an automated process to take a dump of all the databases and store locally on the disk!  Given this happens at midnight each night and the database stopped running at 8pm we’ll just be able to restore from the database dumps right?  Wrong.  We didn’t install postgresql and there is no process in place to do this.

So at this point in time, the dataset was still there but effectively useless and mail services were still down.  Fortunately, we were able to save the day by restoring all the binary files from this specific version of postgresql from backups and thus restore services for the client.  Whilst the motivation behind using this specific version is unknown, the software developer has since moved on and there is zero documentation.  This situation really shouldn’t have happened in the first place. This type of problem is actually something that we see more often then you would imagine.  We often have developers requesting specific versions of software to use in a production environment.  Obviously, we would strongly, strongly discourage the use of development versions within production (they’re called DEVELOPMENT versions for a reason, they simply haven’t been around long enough to be considered stable, reliable software). However, from time to time a specific feature or bug fixes within a specific development version which dictates we must install such a version.  This is something we can certainly get working…  And, most importantly, keep the machine in a maintainable state! This means having supporting documentation as to the decisions made as well as making sure that routine maintenance tasks will not break the existing, carefully crafted configuration.

I also have another fond memory of a web developer who was having some niggling problems with tomcat and permissions and figured that the best way to solve the problem was using:

chown tomcat / -R

So, it got the web application working, but broke virtually every other service on the machine. Can anyone say hosed file system permissions?

…Or how about the Windows machine which has 4, yes, 4 separate instances of MSSQL installed on it..  I digress.

Without wanting to turn this into a big marketing spiel, it is important to keep in mind that like software development, system administration can be a tough game too.   Obviously in the above examples using hind-sight we can easily identify the problems in what was done previously on the machines.  That said, at Anchor we are a team of system administrators who have been running complex systems for a long time now and have the experience to make sure that all the appropriate precautions are taken to make sure we don’t end up in these situations above.

Further to this we have numerous systems in place to pro-actively check services including database servers, 24/7. In the event of failure both audible and visual alerts are generated with notifications outside of hours being sent via SMS message service. Even in the event that this happened on a fully managed machine it would never have resulted in 14 hours down time.  All said, I am not just trying to blow our own horn about how fantastically brilliant we are (ok, maybe, just a little), but what I am trying to get across is system administration is something that really requires an all or nothing attitude towards. If your website or associated hosting infrastructure is critical to your business’ success then making sure the commitment to system management is commensurateable is absolutely imperative to success. Either through outsourcing via our fully managed support pack or by hiring a dedicated system administrator.  There really is no place for laissez-faire and utilising a software developer part-time for this role is only likely to cost more in the longer term.

$USER1$/check_disk -w 5G -c 1G -p /data/foo

If you don’t actually test that this works (by artificially filling your disk and seeing what happens), you may be dismayed to find that you only get alerted when the disk has 5MB of free disk space. Why is this?

Because Nagios, despite the fact that nobody has sweated the megabytes for about a gazillion years, doesn’t support ‘G’ as a suffix for thresholds. Oh, it’ll make a good show of pretending — after all, the output formatting options have ‘GB’ as an option — but nope, for your thresholds it’s “5000M” all the way.

ROCK ON!

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            tcp flags:ACK/ACK
ACCEPT     all  --  anywhere             anywhere            state ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            state RELATED
ACCEPT     udp  --  anywhere             anywhere            udp spt:domain dpts:1024:65535
ACCEPT     icmp --  anywhere             anywhere            icmp any
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ftp
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:smtp
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:pop3
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:imap
ACCEPT     udp  --  anywhere             anywhere            udp dpt:imap
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:mysql
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:mysql
ACCEPT     tcp  --  anywhere             anywhere            tcp dpts:terabase:samsung-unidex
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ndmp
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:dnp
LOG        all  --  anywhere             anywhere            LOG level debug prefix `DROPPED = '
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ftp-data 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:ftp-data dpt:ftp-data
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:ftp dpt:ftp

Lovely that it has all those ports and whatnot opened up, but what’s with the ACCEPT policies?

Webmin: Now with FAILWALL management!

I should have been in marketing.

I’ll cut to the chase – sometimes you have to really shoehorn technologies to do what you want. Fortunately I love doing this, and the technologies of today’s article are virtualised Windows 2008 on Xen, and Oracle XE 10g. Neither likes to play ball, for a few reasons:

• Generally speaking, when you virtualise an OS you want to have para-virtualisation drivers enhancing the hardware support. Open Source Xen has PV drivers, but they are not signed with a legitimate certificate. Windows 2008 does not play nicely with unsigned or test-cert-signed drivers.
• Oracle is just a messy, messy, nasty thing. Yes, paid versions undoubtedly support all manner of loadbalancing and HA options, but the free one does not.

Adding HA to Windows 2008 on Xen

The basic procedure was as follows:

• Install the telnet server within Windows (making sure to lock it down in the firewall to only be accessible by the host machines)
• Create a special admin account and password used for triggering a shutdown
• Create an Expect script which logs into the VM via telnet, and issues the shutdown command
• Create a modified version of the Heartbeat Xen resource agent which calls the expect script to shut down the VM (and wait a safe period of time) before “xm shutdown” is called. Without this, “xm shutdown” will simply power off the VM (in absence of working PV drivers).

The VM was already running on a DRBD volume between the two HA Xen servers, so I was able to just create a standard set of Heartbeat resources to control DRBD primary/secondary mode and the startup/shutdown of the HA WIndows VM. For your benefit (if you want to recreate it) here is the expect script:

#!/usr/bin/expect -f
#
# Script which "automates" shutting down a Windows VM

# Don't log telnet output and commands to stdout, and set a reasonable timeout.
log_user 0
set timeout 3

# Log in via telnet and issue commands. Fairly straightforward.
spawn -noecho /usr/bin/telnet 192.168.1.1
sleep 0.5

# login as the "shutdown" user
expect {
 -re "login: $" {send "shutdown\r"}
 timeout exit
}
sleep 0.5
expect {
 -re "password: $" {send "mysecretpassword\r"}
 timeout exit
}
sleep 0.5
expect {
 -re ">$" {send "shutdown /s /t 0\r"}
 timeout exit
}
sleep 0.1
expect {
 -re ">$" {send "exit\r"}
 timeout exit
}
exit

The rest is fairly self-explanatory if you understand Heartbeat.

Oracle XE 10g

This was more of a learning process, since usually you just install Oracle and leave it the hell alone. Not so for me.

• Install Oracle on both nodes using (fortunately) the RPMs they provide
• Configure Oracle on both nodes including creating the databases, using the same password for SYSDBA
• Shutdown both instances of Oracle
• Create the DRBD resource, and mount it on the primary node
• On the primary node, move the contents of /usr/lib/oracle/xe/oradata and /usr/lib/oracle/xe/app/oracle/flash_recovery_area onto the mounted DRBD
• On the secondary node, delete the aforementioned paths
• Bind mount the oradata and flash recovery area from the mounted DRBD volume into the correct places in the directory tree.
• Start Oracle

After I had created a Heartbeat resource group which contained the DRBD resource, the DRBD filesystem mount, the aforementioned bind mounts and the Oracle service itself I was quite pleased to see that Oracle plays quite nicely with our shoehorned HA setup. You’ll want to make sure you have a properly fixed Oracle init script though, as the supplied one is fairly bad.

After making Oracle and Windows 2008 work nicely in HA, I’m almost certain any service no matter how bad can be shoehorned in a similar way to give you decent availability even when it was n’t originally intended.

“Industry analysts predict that Linux and Windows will soon dominate the operating system space. How you respond to this is critical.”

Meanwhile, industry analysts predict that more than 98% of the population will be consuming oxygen by 2010.

• The most commonly used bootloader for Linux systems, GRUB, doesn’t yet have capabilities to boot from GPT partitions (at least not in the stable release)
• The most commonly used partitioner, fdisk, doesn’t support GPT-partitioned disks (and hence no disk larger than 2TB)
• GNU parted, which does support GPT, insists on performing all partition resize operations itself (including resizing the contained filesystem). Since it doesn’t yet understand LVM, it can’t resize any partition that contains an LVM PV.

Today I ran into what appears to be a bug in the CentOS 5.3 installation partitioner, which left my 12TB RAID volume only partitioned to 8TB when I had supplied the –grow parameter in the Kickstart script. Since parted can’t resize LVM partitions, and there don’t appear to be any other tools out there at the moment for GPT partitioning on Linux, I’m left in a less than ideal position.

GNU parted can’t resize the partition because it can’t understand LVM. Fortunately I can just use it to create another partition with the remaining space and add it to the existing LVM volume group but this is really just a hack, and one that disturbs my obsessive-compulsive sysadmin nature. Were it not for the flexibility of LVM, we would be in a bit of a mess.

Sadly, it seems the large filesystem support that will soon become essential for everyone is largely lacking in adequate support.

One big win for systems administration at Anchor is our use of APC Rack Power Distribution Units. These have been documented elsewhere in our blog and wiki but suffice it to say that having remote control over your power ports is a Very Good Thing. Situations where you have servers or other devices with multiple power supply units complicates things slightly, but not that much, especially with the aforementioned Rack PDUs in place.

APCs in particular allow you to configure what are called Multicast Groups. Essentially you tell a couple of the Rack PDUs to talk to each other and share information, and WHAMMO you can turn off and turn on a bunch of ports on separate Rack PDUs simultaneously! So rather than turning off the power to one PSU then rebooting the other, you can conduct a reboot of the power to both PDUs with a single command.

The confusion comes during the configuration of the Multicast Group option. Multicast is a very under-utilised feature of IPv4 (which has now partially been rectified in IPv6), in fact a large chunk of the IPv4 address space is allocated to multicast (and is technically called the Class D space). As with all other portions of IP address-space, this has been carefully portioned into sections and allocated to various purposes. You can see the full list here:

http://www.iana.org/assignments/multicast-addresses/

Being a good sysadmin I consider standards to be of paramount importance, so naturally I wanted to configure our Rack PDUs with multicast addresses suitable for the purpose. There are many existing references on the Internet for how to pick sane and standards-obeying addresses from the multicast range. However, when attempting to follow standards and good reason, I was confronted with this error message:

Multicast IP Address is out of range. Valid values are 224.0.0.3 - 224.0.0.254.

Uh, what? I was under the impression that the range 224.0.0.0/24 was already heavily allocated to entities and purposes other than APC Rack PDUs! So much for following the standard, APC.

We were alerted this morning to a problem with some of the Weblogic content – the pages were timing out. Diagnostics were fairly fruitless – packet captures showed nothing useful, and the logging from Weblogic left much to be desired. We started considering more outlandish possibilities such as I/O load causing issues, recently applied updates and so on. Even rebooting was considered (given it is running on Windows).

The first clue of note was the open file list from the Weblogic processes – one such example stood out:

C:\weblogic\state\Sa0V\b1gR\O1Ok\WqYN\9kiv\IQT2\SHGx\C3ri\aE1z\L1YH\X5QW\
gdkB\B2PB\pPPw\uHDK\p1a7\I0l5\94sU\kQ43\+533\5517\5738\7484\6253\_-10\
6273\1519\_6_8\888_\8888\_700\2_702_8\888_

For the sake of your screen, I have manually wrapped this Godzilla-like filename.

Perhaps you are familiar with file bucketing already, but if not, typically the directory structure used will have a relatively sane scheme for locating files and only extend a few levels deep. What we saw in this instance was a completely new breed of monster. Admittedly the absolute path of this file is less than 200 characters out of a limit of more than 32,000 but the naming strategy and depth of the structure has us flummoxed.

But this was only the tip of the proverbial iceberg. When we requested Windows to show us the properties of this state folder it took over an hour to completely calculate the file and folder totals, and the result is impressive:

Web logic makes efficient use of the filesystem

Yes you read that right – over 10 million nested directories. By this stage we had already moved the state directory out of the way and created a new one, and restarted Weblogic. It seemed happy and quite responsive after that. My suspicion is that someone developing this application at some point ran into a limitation with their filebucketing algorithm, and resolved to solve the problem once and for all, evidently by making it possible to efficiently filebucket every file in the known universe.

Hard drive failure is a fact of life when you run servers, by sheer virtue of that fact that you have hundreds of them. To mitigate the risk and reduce unscheduled downtime, we use Window’s built-in software RAID feature. It’s not an enterprise solution, but it gets the job done. What’s important is staying online and not losing data.

Did I mention that trying to monitor a Windows box is a nightmare? A colleague of mine wrote a script to allow us to keep a watchful eye on Windows RAID volumes, it’s a lifesaver. A recently-deployed machine got a broken mirror, which we were able to act on immediately. We removed the dodgy mirror and prepared a replacement (we always have plenty of spares, of course). Allow me now to re-enact this scene…

Windows (sounding almost efficient): The driver has detected that device \Device\Harddisk1\DR9 has predicted that it will fail

Sysadmin: Thanks, Windows, I’ll get right on that. You didn’t say whether that was SMART, or just voodoo, but whatever, it’s good to know.

The bad drive is removed and a replacement installed in the hotswap drive bay

Sysadmin: Okay, Windows, do your stuff. “Scan for new hardware”, please.

A pause.

Sysadmin: Ahem, Windows, “Scan for new hardware” and find my drive.

Windows: ‘Ey there, chaps. Do what now, you say? AIEEEEGRH!!

The server stops responding entirely, necessitating a touch of the reset button

Needless to say, we’re rather unimpressed, and have to call the customer to let them know why it’s just dropped offline.

A quick check of the logs is in order. It’s also frustrating that there’s no sane way to scroll through log entries in Windows with something like a text editor, or to “tail” a log as it’s updated in realtime.

09:36 – The previous system shutdown at 9:21:23 AM on 8/10/2008 was unexpected.

Okay, it went down at 09:21, which is correct. Now if we look back in time a little…

09:21 – dmio: Harddisk1 write error at block 1953524618 due to disk removal

*sigh* And this is after the disk was removed cleanly…