Anchor’s been doing this for what must be about ten years now. That’s way longer than I’ve been employed here, but what with our tech-director-and-co-founder being busy stuntjumping his scooter over rows of parked cars, it’s fallen to me to write this one up.

We use apache’s mod_suexec to run PHP scripts as though they’re CGI scripts, and it works great. There’s lots of guides out there about how to do this for yourself, but for us one of the most important things when deploying a solution is to ensure that it Just Works, for everyone. When we do it right, nothing breaks and noone notices a thing, because it works exactly as everyone expects. That’s what we’ve done.

It’s also one of the reasons that apache isn’t going anywhere in a hurry. Newer tech like nginx is an absolute performance-demon compared to apache, but the barrier to entry is way too high if your goal is “throw the site on the server and make it work”. Everyone writes apps assuming they’ll be deployed to apache. As a hosting company, if you’re not offering something compatible you’d better have a good ace up your sleeve.

So, we give you a rundown on how Anchor does PHP; it’s secure and it just works. There are many others like it, but this one is ours. If you’re an existing customer, we hope you’ve never had to think about it. If you’ve got any questions we’d love to hear them.

Oh great, they’re making whoopie on their mailing lists again and making some stupidly huge logfile.

Little did I know just how huge that file was. How about 735gb huge, in the space of 12hrs? This customer is already a bit of an oddball, what with 1.4TiB of usable space in their server. “Oh that’s nothing”, you say. Sure, I’ve got a few TiB of kitten pictures on my machine at home, just like you, but to put things in perspective: 300GiB of space would be “big” for most Anchor customers. SCSI disks cost about $1.70/Gb, compared to about 10c/Gb for SATA.

There was no mailout. No big processing job, and no flood of activity. With a little digging I was able to nail it down to an apache errorlog file. That was a surprise, except for the PHP errors all throughout – some things never change.

[Fri Oct 02 02:39:57 2009] [error] [client 63.82.71.139] PHP Warning: fgets(): supplied
argument is not a valid stream resource in /home/wright/public_html/script.php on
line 15, referer: XXX

Nice work there, guys. You need to learn to check your return values from failure-prone functions.

Strangely, there were no actual active connections, but the process list showed two apache processes going balls to the wall, writing the same error message to the log file ad infinitum. By my reckoning that was over 9000 lines per second – nothing a quick service-restart couldn’t fix, thankfully.

And to actually fix the problem? It’s tempting to dump the file, but we don’t like doing that; it’s just a bit too cowboy for us. I settled for a forced logrotate run, taking about 4hrs and squishing it down to just 4.3GiB – Crisis (and sleep) Averted.

http://www.slideshare.net/techdude/how-to-kill-mysql-performance

If you’re an Anchor customer, I should point out that the ARCHIVE storage engine isn’t available in Redhat’s version of MySQL, which is a damned nuisance.

http://au2.php.net/goto

Even Java, which we have little fondness for here, does not give you goto. This has to be one of the worst misfeatures yet in 5.3 (as opposed to the native MySQL driver, which only causes customers to ask for stupid things, like installing pre-alpha code on their production webserver).

In case it weren’t clear enough, they go so far as to use an XKCD strip that highlights how bad an idea this is. I suspect this is some kind of sick joke…

I found this earlier this evening; it’s a little old (~18 months), but that doesn’t make it any less relevant, so it’s a good read. There’s the odd inaccuracy here and there, but it’s solid stuff, and relevant to anyone writing webapp code to handle authentication.
http://www.matasano.com/log/958/enough-with-the-rainbow-tables-what-you-need-to-know-about-secure-password-schemes/

The article focuses heavily on one particular authentication methodology, because it’s something a lot of people do. After you read it, you’ll understand that it’s something a lot of people do poorly. It assumes a reasonable degree of knowledge about what you’re doing (ay, there’s the rub), but the best lesson to be learnt there is that you shouldn’t be writing that code! Just stop! Someone’s already done it better than you, and it’s been checked by a lot more people than you and your colleague, who reckons it “looks pretty secure”. If you’ve got time to read all the comments on the article you’ll find a rollercoaster of people saying, “wouldn’t it be secure if I just..?” Just don’t do it. Don’t write my security code, bro!

Just using a proper authentication mechanism means you can spend your valuable time on more important things.

• Like not building SQL queries in a piecemeal manner, using unsanitised user-supplied data
• Like not making your whole site world-writeable so you can handle file uploads (you don’t have to do this on our shared hosting servers)
• Like fixing your newsletter signup process so it requires double opt-in, saving you from filling your database with spurious accounts and causing massive headaches when you do a campaign mailout
• Like optimising your database schema for bitchin’ performance – do you retain a DBA on a six-figure salary to do this for you? Nah I didn’t think so, this is cheaper and much more fun
• Like checking that your code doesn’t spew 14 PHP Warns/Errors to the apache logs for every page access, and fill the partition
• Like trying to get nice round corners on your page elements

After spotting this pattern, a quick look at our trending graphs showed us the problem very clearly. The machine was steadily using all available physical memory. Once this ran out, the system starts pushing data off to swapspace, which is on the disks. Once this runs out the OS simply has no more memory to give. It’ll start killing off processes in a desperate attempt to reclaim some memory, but this is rarely successful.

Curiously, on days that the server didn’t die we saw a chunk of memory being released early in the morning. This pointed to a maintenance cronjob doing something, but what?

This should more accurately be described as a graph of non-critical memory allocation; buffers and caches can be quickly discarded to free up memory for applications that really need it. Note the sawtooth wave pattern with a period of 1 day.

Swapspace usage can be seen peaking multiple times over the course of a few days, corresponding to times when the server resembled a glorious fireball

We were now well-prepared to catch the system in the act. We kept an eye on trending graphs and waited for the memory usage get out of hand. When we logged in, what we saw was surprising. Apache was consuming vast amounts of memory, highly irregular for a server which has the primary function of a mail server. Having a look at apache’s logs, we noticed a lot of failed requests to Zpush, an open-source implementation of the MS Exchange protocol for syncing data to mobile clients (iPhones, in this case), written in PHP.

PHP Fatal error:  Allowed memory size of 16777216 bytes exhausted (tried
to allocate 727553 bytes) in /home/zpush/zpush/include/mimeDecode.php
on line 292

PHP is usually a pretty sturdy beast. It sanitises the environment between requests, effectively sandboxing different users’ code from each other. It’s certainly possible to write pathologically broken code, but generally this shouldn’t affect anyone badly except for yourself.

For whatever reason, zpush was exceeding its local memory limits in the backend MIME parser and dying. This is a condition that PHP should recover from gracefully, but that clearly wasn’t happening. With zpush being regularly called by the customer’s iPhones, this was leaking a little bit of memory every single time. Because apache wasn’t dying outright, the memory never got freed.

A whole lot of things suddenly made sense. Apache will eventually recycle its processes (after 4,000 requests), but on a lightly-used server like this one, apache was only dealing with zpush requests. Spread over 8 child processes, apache could in theory take some 32,000 hits before it gets close to saving itself. This also had a clean explanation for the early morning respite: logrotate will signal apache to do a graceful restart after processing the logs for the day, which will restart the child processes, freeing the memory.

In this instance, the most straighforward solution was to disable zpush. If this weren’t an option then further investigation would be needed. Increasing the PHP memory limit beyond 16MB in php.ini woud do the job, but we’d really like to know why it’s trying to allocate so much memory.