I got woken up by a SMS for low diskspace the other night on one of our customer’s servers. Okay, so that’s a lie, I never sleep, but the SMS is real.

Oh great, they’re making whoopie on their mailing lists again and making some stupidly huge logfile.

Little did I know just how huge that file was. How about 735gb huge, in the space of 12hrs? This customer is already a bit of an oddball, what with 1.4TiB of usable space in their server. “Oh that’s nothing”, you say. Sure, I’ve got a few TiB of kitten pictures on my machine at home, just like you, but to put things in perspective: 300GiB of space would be “big” for most Anchor customers. SCSI disks cost about $1.70/Gb, compared to about 10c/Gb for SATA.

There was no mailout. No big processing job, and no flood of activity. With a little digging I was able to nail it down to an apache errorlog file. That was a surprise, except for the PHP errors all throughout – some things never change.

[Fri Oct 02 02:39:57 2009] [error] [client 63.82.71.139] PHP Warning: fgets(): supplied
argument is not a valid stream resource in /home/wright/public_html/script.php on
line 15, referer: XXX

Nice work there, guys. You need to learn to check your return values from failure-prone functions.

Strangely, there were no actual active connections, but the process list showed two apache processes going balls to the wall, writing the same error message to the log file ad infinitum. By my reckoning that was over 9000 lines per second – nothing a quick service-restart couldn’t fix, thankfully.

And to actually fix the problem? It’s tempting to dump the file, but we don’t like doing that; it’s just a bit too cowboy for us. I settled for a forced logrotate run, taking about 4hrs and squishing it down to just 4.3GiB – Crisis (and sleep) Averted.

Not quite as zippy as most of the stuff we deploy, but no less admirable for it.
http://www.humanclock.com/webserver.php

I’m currently porting apache 1.3.14 to the venerable TI-89, it should be finished in about 5-6 weeks.

I was shoulder-surfing a colleague today while they were trying to diagnose a webserver problem for a client. I noticed, certainly not for the first time, that the Apache error log message was filled with messages like “robots.txt not found” and “favicon.ico not found”. Surely these must be amongst the most frequently logged errors (if not the top two).

Multiplied by many hundreds of servers, with millions of hits per day, and you have a significant amount of disk space being taken up by these trivial messages. What’s more, any time you spend scrolling through the hordes of messages like these is time taken away from debugging the real problem, if it exists.

So please, be kind to your sysadmin and include a robots.txt and favicon.ico for your website. It makes sense from a search engine point of view and makes your website just a little bit prettier, so why not?

One of our dedicated server customers recently had a problem with the machine keeling over and dying for a few days in a row, for no apparent reason. This necessitated a remote reboot of the server to get it running again (we cut the power to both power supplies for a few seconds). The immediate suspicion was faulty hardware, but this should rarely be the case as we put our hardware through a thorough “burn in” period before it’s ever deployed. In addition to this, it was happening pretty regularly in the middle of the day.

After spotting this pattern, a quick look at our trending graphs showed us the problem very clearly. The machine was steadily using all available physical memory. Once this ran out, the system starts pushing data off to swapspace, which is on the disks. Once this runs out the OS simply has no more memory to give. It’ll start killing off processes in a desperate attempt to reclaim some memory, but this is rarely successful.

(more…)