I got woken up by a SMS for low diskspace the other night on one of our customer’s servers. Okay, so that’s a lie, I never sleep, but the SMS is real.

Oh great, they’re making whoopie on their mailing lists again and making some stupidly huge logfile.

Little did I know just how huge that file was. How about 735gb huge, in the space of 12hrs? This customer is already a bit of an oddball, what with 1.4TiB of usable space in their server. “Oh that’s nothing”, you say. Sure, I’ve got a few TiB of kitten pictures on my machine at home, just like you, but to put things in perspective: 300GiB of space would be “big” for most Anchor customers. SCSI disks cost about $1.70/Gb, compared to about 10c/Gb for SATA.

There was no mailout. No big processing job, and no flood of activity. With a little digging I was able to nail it down to an apache errorlog file. That was a surprise, except for the PHP errors all throughout – some things never change.

[Fri Oct 02 02:39:57 2009] [error] [client 63.82.71.139] PHP Warning: fgets(): supplied
argument is not a valid stream resource in /home/wright/public_html/script.php on
line 15, referer: XXX

Nice work there, guys. You need to learn to check your return values from failure-prone functions.

Strangely, there were no actual active connections, but the process list showed two apache processes going balls to the wall, writing the same error message to the log file ad infinitum. By my reckoning that was over 9000 lines per second – nothing a quick service-restart couldn’t fix, thankfully.

And to actually fix the problem? It’s tempting to dump the file, but we don’t like doing that; it’s just a bit too cowboy for us. I settled for a forced logrotate run, taking about 4hrs and squishing it down to just 4.3GiB – Crisis (and sleep) Averted.

I came across this a little while ago; it’s a good little presentation with some interesting points I’d not considered before.

http://www.slideshare.net/techdude/how-to-kill-mysql-performance

If you’re an Anchor customer, I should point out that the ARCHIVE storage engine isn’t available in Redhat’s version of MySQL, which is a damned nuisance.

Hot on the heels of PHP 5.3’s snazzy new namespaces, here comes goto. I kid you not, see for yourself:

http://au2.php.net/goto

Even Java, which we have little fondness for here, does not give you goto. This has to be one of the worst misfeatures yet in 5.3 (as opposed to the native MySQL driver, which only causes customers to ask for stupid things, like installing pre-alpha code on their production webserver).

In case it weren’t clear enough, they go so far as to use an XKCD strip that highlights how bad an idea this is. I suspect this is some kind of sick joke…

I really like cryptography and security. I was lucky enough to take it as a subject at UNSW before I graduated.

I found this earlier this evening; it’s a little old (~18 months), but that doesn’t make it any less relevant, so it’s a good read. There’s the odd inaccuracy here and there, but it’s solid stuff, and relevant to anyone writing webapp code to handle authentication.
http://www.matasano.com/log/958/enough-with-the-rainbow-tables-what-you-need-to-know-about-secure-password-schemes/

The article focuses heavily on one particular authentication methodology, because it’s something a lot of people do. After you read it, you’ll understand that it’s something a lot of people do poorly. It assumes a reasonable degree of knowledge about what you’re doing (ay, there’s the rub), but the best lesson to be learnt there is that you shouldn’t be writing that code! Just stop! Someone’s already done it better than you, and it’s been checked by a lot more people than you and your colleague, who reckons it “looks pretty secure”. If you’ve got time to read all the comments on the article you’ll find a rollercoaster of people saying, “wouldn’t it be secure if I just..?” Just don’t do it. Don’t write my security code, bro!

Just using a proper authentication mechanism means you can spend your valuable time on more important things.

• Like not building SQL queries in a piecemeal manner, using unsanitised user-supplied data
• Like not making your whole site world-writeable so you can handle file uploads (you don’t have to do this on our shared hosting servers)
• Like fixing your newsletter signup process so it requires double opt-in, saving you from filling your database with spurious accounts and causing massive headaches when you do a campaign mailout
• Like optimising your database schema for bitchin’ performance – do you retain a DBA on a six-figure salary to do this for you? Nah I didn’t think so, this is cheaper and much more fun
• Like checking that your code doesn’t spew 14 PHP Warns/Errors to the apache logs for every page access, and fill the partition
• Like trying to get nice round corners on your page elements

One of our dedicated server customers recently had a problem with the machine keeling over and dying for a few days in a row, for no apparent reason. This necessitated a remote reboot of the server to get it running again (we cut the power to both power supplies for a few seconds). The immediate suspicion was faulty hardware, but this should rarely be the case as we put our hardware through a thorough “burn in” period before it’s ever deployed. In addition to this, it was happening pretty regularly in the middle of the day.

After spotting this pattern, a quick look at our trending graphs showed us the problem very clearly. The machine was steadily using all available physical memory. Once this ran out, the system starts pushing data off to swapspace, which is on the disks. Once this runs out the OS simply has no more memory to give. It’ll start killing off processes in a desperate attempt to reclaim some memory, but this is rarely successful.

(more…)