Recently, I read an article from a fairly prominent “cloud computing” vendor, which contained a line that basically said “Let the cloud worry about your scalability and performance problems”. I nearly snorted my late-mid-morning can of mother out my nose when I read it. Here’s why.

“Let the cloud worry about your scalability” is nothing more than a thinly disguised version of “just throw more hardware at it”. This is a “solution” beloved of salespeople everywhere, because it’s plausible, real easy to say, and makes a whole pile more money for the company providing the hardware. However, while it can be an appropriate solution in the right circumstances, and with appropriate evidence of its effectiveness in those particular circumstances, it usually isn’t the only option, it often isn’t the best option, and sometimes it isn’t an effective option at all.

The dirty little secret of hosting is that your scaling ability is solely determined by your application — the technologies it uses and its internal architecture. Yes, you can probably get more performance or concurrent users out of throwing more hardware at it this time, but sooner or later more memory or faster CPUs isn’t going to do anything useful.

I suppose, in some perverse way, just telling developers that “the cloud will provide” could be construed as a kindness. In the same way that we give high school kids the simplified approximation of motion that is Newton’s Laws, rather than the complicated and fiddly reality of relativity, saying “let the cloud scale you to being the next Facebook” might be a useful approximation to let developers ignore extraneous details and focus on getting things “right enough”.

The vast majority of sites, even those who aspire to be the next Twitter, will never get to anywhere near that scale. Even if it is the goal (and plenty of sites manage to occupy a satisfying — and dare I say it, profitable — niche without needing a second datacentre full of equipment), a new site is only going to get that big by focusing on satisfying users and creating compelling applications.

Spending your time writing Yet Another Key-Value Store is an awesome way to spend a lazy weekend, but when you’re burning your rent cheque and credit rating trying to get your “next big thing” site off the ground, every minute spent not awesomising your user experience is putting you 35c closer to having to go back to working for The Man.

For whatever reason, though, it makes me uncomfortable to lie to people about things like this, even if I might think it’s in their interest. I know, first hand, the shock and pain that comes from finding out that your site, beloved by millions, is suddenly overloaded and unreliable — and, even worse, that throwing hardware at the problem won’t do a damned thing. It’s an awful feeling.

While you can’t be worrying about scaling to a million users when your site has a grand total of three users (one of which is your mum), you have to prepare for it when things starts to take off, and have a plan in place to deal with it.

Sooner or later, you’re going to have to sit down, find the pain points in your current architecture, and work out how to solve them. If you’re not comfortable doing that yourself, then you need smart systems people who know how. I can guarantee you that “the cloud” isn’t going to advise you on how to restructure your file storage so it will horizontally scale to a petabyte of data. Don’t rely on it to scale you out of trouble.

I came across this a little while ago; it’s a good little presentation with some interesting points I’d not considered before.

http://www.slideshare.net/techdude/how-to-kill-mysql-performance

If you’re an Anchor customer, I should point out that the ARCHIVE storage engine isn’t available in Redhat’s version of MySQL, which is a damned nuisance.

Reducing the memory usage in your VPS/VM can be a great way to free up some resources to handle more requests, users or some other metric of win.

By default at Anchor we provision our Red Hat & Cent OS VPS servers with a trim memory usage profile by disabling a lot of unneeded services at install time. We do this by using Trogdor (our hardware/software burninator) and Puppet.

So just what services do we disable, if they exist on the new VPS?

• gpm
• netfs
• pcmcia
• sgi_fam
• yum-updatesd
• pcscd
• rhnsd
• xfs
• hald
• hcid or sdpd
• hpiod or hpssd.py
• dbus-daemon
• cupsd

You can also reap performance gains by changing how you serve content. For example you can use a cut down high performance web server (nginx or lighttpd) to serve all static content, such as images. Then use an Apache process to handle your dynamic requests.

Tuning Apache is deserving of an article all to itself, however some hard and fast rules are:

• Disable all unnecessary modules.

• Work out the per process memory usage, and set your max clients to a suitable number taking into account the available memory and other system daemons.

• Disable htaccess if you do not need it.

PHP is much the same as Apache, disable what you do not need. This goes for any service or application with many optional components.

Any company with multiple servers in their authentication domain will know of LDAP. Sadly on the Linux platform, OpenLDAP (although arguably the most widely used and well known of the few LDAP servers available) is still not particularly reliable, especially when it comes to replication. The overheads involved in querying even a local OpenLDAP server are much higher than, say, the plaintext files such as /etc/passwd.

Enter nsscache. Created by two boffins at Google (one of whom graduated from Anchor Systems), nsscache gives the reliability and speed of plaintext files (or BDB if you desire) and the scalability of OpenLDAP. Anchor recently started using it and we are confident it will dramatically boost the reliability and lookup speed of all of our LDAP systems.

In terms of performance, we are seeing update times of a second or so for a partial update (only changed or new entries), and around 10 seconds for a full update (replacing the entire on-disk files with a fresh copy of the entire source database) for a bit under 20000 records.

I’d definitely recommend taking a look at it. It currently only operates on Python 2.4 or later (although a patch is nearing release that allows it to work with Python 2.3).