Anchor’s been doing this for what must be about ten years now. That’s way longer than I’ve been employed here, but what with our tech-director-and-co-founder being busy stuntjumping his scooter over rows of parked cars, it’s fallen to me to write this one up.

We use apache’s mod_suexec to run PHP scripts as though they’re CGI scripts, and it works great. There’s lots of guides out there about how to do this for yourself, but for us one of the most important things when deploying a solution is to ensure that it Just Works, for everyone. When we do it right, nothing breaks and noone notices a thing, because it works exactly as everyone expects. That’s what we’ve done.

It’s also one of the reasons that apache isn’t going anywhere in a hurry. Newer tech like nginx is an absolute performance-demon compared to apache, but the barrier to entry is way too high if your goal is “throw the site on the server and make it work”. Everyone writes apps assuming they’ll be deployed to apache. As a hosting company, if you’re not offering something compatible you’d better have a good ace up your sleeve.

So, we give you a rundown on how Anchor does PHP; it’s secure and it just works. There are many others like it, but this one is ours. If you’re an existing customer, we hope you’ve never had to think about it. If you’ve got any questions we’d love to hear them.

Oh great, they’re making whoopie on their mailing lists again and making some stupidly huge logfile.

Little did I know just how huge that file was. How about 735gb huge, in the space of 12hrs? This customer is already a bit of an oddball, what with 1.4TiB of usable space in their server. “Oh that’s nothing”, you say. Sure, I’ve got a few TiB of kitten pictures on my machine at home, just like you, but to put things in perspective: 300GiB of space would be “big” for most Anchor customers. SCSI disks cost about $1.70/Gb, compared to about 10c/Gb for SATA.

There was no mailout. No big processing job, and no flood of activity. With a little digging I was able to nail it down to an apache errorlog file. That was a surprise, except for the PHP errors all throughout – some things never change.

[Fri Oct 02 02:39:57 2009] [error] [client 63.82.71.139] PHP Warning: fgets(): supplied
argument is not a valid stream resource in /home/wright/public_html/script.php on
line 15, referer: XXX

Nice work there, guys. You need to learn to check your return values from failure-prone functions.

Strangely, there were no actual active connections, but the process list showed two apache processes going balls to the wall, writing the same error message to the log file ad infinitum. By my reckoning that was over 9000 lines per second – nothing a quick service-restart couldn’t fix, thankfully.

And to actually fix the problem? It’s tempting to dump the file, but we don’t like doing that; it’s just a bit too cowboy for us. I settled for a forced logrotate run, taking about 4hrs and squishing it down to just 4.3GiB – Crisis (and sleep) Averted.

I’m currently porting apache 1.3.14 to the venerable TI-89, it should be finished in about 5-6 weeks.

Multiplied by many hundreds of servers, with millions of hits per day, and you have a significant amount of disk space being taken up by these trivial messages. What’s more, any time you spend scrolling through the hordes of messages like these is time taken away from debugging the real problem, if it exists.

So please, be kind to your sysadmin and include a robots.txt and favicon.ico for your website. It makes sense from a search engine point of view and makes your website just a little bit prettier, so why not?

After spotting this pattern, a quick look at our trending graphs showed us the problem very clearly. The machine was steadily using all available physical memory. Once this ran out, the system starts pushing data off to swapspace, which is on the disks. Once this runs out the OS simply has no more memory to give. It’ll start killing off processes in a desperate attempt to reclaim some memory, but this is rarely successful.

Curiously, on days that the server didn’t die we saw a chunk of memory being released early in the morning. This pointed to a maintenance cronjob doing something, but what?

This should more accurately be described as a graph of non-critical memory allocation; buffers and caches can be quickly discarded to free up memory for applications that really need it. Note the sawtooth wave pattern with a period of 1 day.

Swapspace usage can be seen peaking multiple times over the course of a few days, corresponding to times when the server resembled a glorious fireball

We were now well-prepared to catch the system in the act. We kept an eye on trending graphs and waited for the memory usage get out of hand. When we logged in, what we saw was surprising. Apache was consuming vast amounts of memory, highly irregular for a server which has the primary function of a mail server. Having a look at apache’s logs, we noticed a lot of failed requests to Zpush, an open-source implementation of the MS Exchange protocol for syncing data to mobile clients (iPhones, in this case), written in PHP.

PHP Fatal error:  Allowed memory size of 16777216 bytes exhausted (tried
to allocate 727553 bytes) in /home/zpush/zpush/include/mimeDecode.php
on line 292

PHP is usually a pretty sturdy beast. It sanitises the environment between requests, effectively sandboxing different users’ code from each other. It’s certainly possible to write pathologically broken code, but generally this shouldn’t affect anyone badly except for yourself.

For whatever reason, zpush was exceeding its local memory limits in the backend MIME parser and dying. This is a condition that PHP should recover from gracefully, but that clearly wasn’t happening. With zpush being regularly called by the customer’s iPhones, this was leaking a little bit of memory every single time. Because apache wasn’t dying outright, the memory never got freed.

A whole lot of things suddenly made sense. Apache will eventually recycle its processes (after 4,000 requests), but on a lightly-used server like this one, apache was only dealing with zpush requests. Spread over 8 child processes, apache could in theory take some 32,000 hits before it gets close to saving itself. This also had a clean explanation for the early morning respite: logrotate will signal apache to do a graceful restart after processing the logs for the day, which will restart the child processes, freeing the memory.

In this instance, the most straighforward solution was to disable zpush. If this weren’t an option then further investigation would be needed. Increasing the PHP memory limit beyond 16MB in php.ini woud do the job, but we’d really like to know why it’s trying to allocate so much memory.