Weathering the ClickFrenzy Storm

It was touted to be the online sale to stop the nation. In reality, for most, it turned out to be the online sale which broke the Internet.

At a time when bricks and mortar shops are seeing a decline in sales — this was a real opportunity to regain some ground against international competition. And for some, they did. But for many of the retailers getting in on the action – it was nothing more than an abysmal failure. It doesn’t take too much to just see how widespread the performance and reliability problems were last night; social media was quick with the complaints rampant on the #ClickFrenzy hashtag and both #ClickFrenzyFail and #ClickFail were trending. On Facebook, a group “ClickFrenzy fail” was created and quickly gathered thousands of likes and comments from people complaining that the site hosting the event had simply dropped off the Internet.

Not only was the ClickFrenzy site not available, but many of the retailers participating in this campaign were also affected. There were various reports that many of the big retailers such as Myer, Priceline and Kogan.com.au were quickly taken offline.

But it wasn’t all doom and gloom for the retailers who paid good money to be involved in this campaign! A site which continued to buzz thoughout the campaign was Booktopia’s website, hosted by Anchor. As publicly commented by Tony Nash, CEO of Booktopia:

“Congrats to Steve, Wayne and everyone at Anchor for showing the rest of the country how to put on an online sale. Wayne just told me that response times on the site increased by 10 milliseconds at the peak of the traffic. We absolutely blew our huge competitors away.”

So there are at least some take-aways from this event which are worthwhile sharing to protect yourself from this kind of event:

1) Deal with someone who has done it before. On a world-wide scale, Internet infrastructure in Australia is relatively small. Compared with the US networks, servers and over-all deployments are typically an order of magnitude smaller. So why are we trying to re-invent the wheel? Why not deal with people who have experience deploying on a worldwide scale? Here at Anchor we’ve done BIG sites. Think millions of users big.

2) Load testing. Load testing. Load testing. Until you know just how much capacity your infrastructure can reasonably handle — you just don’t know how close you are sailing to the line. Can your web infrastructure support a 2 fold increase in traffic or a 100 fold increase in traffic? Without load testing — you’ll never really know. We personally recommend using something like Blitz.io for load testing. This has the capacity to throw literally hundreds of thousands of concurrent connections at your web infrastructure. This translates to millions of users.

3) Deploy Web infrastructure in a way that is rapidly scalable. There are many, many, many ways that this can be approached. Through using a combination of vertical or horizontal scaling techniques or using something like Amazon’s “Auto-scale” to burst into “the cloud”. Naturally, you need to understand the strengths, limitations and extent that your scalability approaches will work — and at what point they will break and require a re-think or re-factor. Your local sysadmin team or web hosting company should have this level of expertise.

4) Speak with your web hosting company! Ask the hard questions! Such things as: “How quickly can you upgrade our infrastructure?”, “What other options are there to improve performance apart from just throwing more hardware at the problem?”, “How much network capacity can you service?”, “What happens if/when this gets exhausted?” Make sure that your development team and system administration team work in a symbiotic fashion. Embrace the devops mantra!