On early evening of Tuesday the 30th of March the entire building in which Anchor’s offices are located at 81 York Street Sydney lost power. Our phone system is the only critical piece of infrastructure located in our office that is required to provide service as normal. Until power is restored our phones appear to be engaged to all callers.

If you require support please email us on support@anchor.com.au

If you need to speak with us please email us and we will call you back.

If you are on IRC you can find us at:

• Server: irc.oftc.net
• Channel: #anchor

Important note:

The loss of power does not have any effect on any hosted services since all hosting equipment is stored in a separate specialist facility with redundant power systems.

What happened?

Energy Australia identified and confirmed the fault to be within their network overnight and have since been carrying out emergency repairs. At this stage we expect power to be restored today (Wednesday the 31st of March) but no specific ETA has been given.

How are we working around this?

All of our level 2 staff have the ability to work remotely. Access to 100% of our systems including server management, email support, monitoring systems, customer information. All level 2 staff are currently working remotely until such time as power is restored in the office.

Updates

10:00 AM – We have one of two phases of power available. Power to phones and office workstations have been restored. We’re still without lights but otherwise business as usual.

1:00 PM – Full power restored and the lights are on again.

At Anchor we do not believe in black box solutions.  Sharing is caring and we like to share. In this post we specifically want to share our triumph with Project StarBug, better known to the wider world as GitHub. For the uninitiated, GitHub is ‘Social Networking meets Source Code management’, or in GitHubs own words ‘Git is a fast, efficient, distributed version control system ideal for the collaborative development of software. GitHub is the easiest (and prettiest) way to participate in that collaboration: fork projects, send pull requests, monitor development, all with ease.’.

Some readers may protest this point, stating that GitHub is hosted in the USA while Anchor is located in Australia. How then has Anchor architected, implemented and (going forwards) manage GitHub’s infrastructure with such a geographical encumbrance?

All will be revealed in a blog entry in three of many parts.

Part 1: (This Post) Designing for success (Otherwise known as: Making GitHub’s dream a reality and nightmares a thing of the past)

Part 2: Speed matters

Part N: (To be announced)

For obvious reasons, we cannot expose GitHub’s architecture in full, however we are sharing some of the more interesting technologies/architecture we have implemented, and the rationale for doing so. Essentially what we have done to make GitHub’s dreams a reality.

Geographical encumbrance

It is a credit to GitHub’s management that they were willing to look the world over for the right team to support them. While they do not want to be harried by anything outside the GitHub application (i.e. Hardware, O/S, Management, etc), they still needed to ensure that the right company was employed to look after these components.

Why Anchor? Anchor’s flexibility to manage a solution on third-party hosted hardware (anywhere in the world) and versatility in developing an architecture to suit this scenario were part of the rationale. Anchor’s reputation for needing to know how technology works (again, no black boxes) and then working out how to improve it was a major contribution.

Enough fluff, now to the meat;

One can imagine that the architecture required to support GitHub is complex mix. We won’t lie; there are many moving parts. Some of the key criteria for designing the solution included:

Scalability

GitHub states it growth as “400 new users and 1000 new repositories every day”. Post migration GitHub will be running on infrastructure spread across 15+ physical hosts/servers. It is essential that the infrastructure can grow with the user base, from 10’s  to 100’s of servers, without the need to re-architect everything. Without a doubt, growing without the associated pain is a major objective for GitHub as it moves forward.

Interesting Note: GitHub’s new physical infrastructure (at migration) consists of:

• 15+ physical servers
• 10+ virtual servers
• 128 physical processor cores
• Over 288GBs RAM
• 1TB+ of storage

GitHub’s software architecture is modular by nature and scalability friendly. Components outside the core software, however, were not as readably scalable. This has been achieved with the following improvements;

• Distributed Storage Architecture (with real-time slaves). Distribution of GitHub’s source code repos across multiple partitions and multiple nodes (including redundant slaves) provided improvements in performance, scalability and reliability. By removing the limitation of using a single filesystem volume for storage, the issue of dealing with large scale storage has been avoided. New partitions can be rapidly added on demand with little to no fuss.

The graphic below illustrates a simplified request to the distributed file storage repo:

GitHub Distributed Repo Storage

• (Sensible) Virtualisation. Previously, GitHub’s infrastructure was entirely virtualised. While virtualisation has its merits, there are reasons to avoid it. Services that aren’t I/O-heavy can be virtualised, while components with high I/O requirements are run on dedicated (“bare metal”) servers. For GitHub, this means file storage and databases are not virtualised. Otherwise, virtualisation is used to provide a mix of server consolidation, rapid deployment and service redundancy/HA.
• Horizontal scalability (on-demand, via automated build infrastructure). The ability to add additional components to the infrastructure in an automated fashion reduces scale-out time and removes user error from builds/configuration. In addition, this also turns the server build/deployment procedure into a measurable deliverable. Over time this can be review and improved (Thank you W. Edwards Deming).

Reliability

As with most businesses, High Availability (or business continuance) is essential to a success. To achieve this a combination of DRBD, virtualisation, heartbeat and load balancing has been employed.

• Mirroring Data; DRBD is utilised for several purposes.

1. It is used to ensure the redundant (read: slave) storage partitions and nodes are in sync with the active counterparts.
2. DRBD is also key in providing HA functionality across the virtualised environment.

Several Xen hosts are deployed with the following scenario; Server 1 runs VM A(active) B(active) C(offline DRBD mirrored) D(offline DRBD mirrored), and Server 2 runs VM A(offline DRBD mirrored) VM B(offline DRBD mirrored) VM D(active) VM E(active). This provides active failover if either of the virtualisation hosts fail.

The graphic below illustrates the replicated, highly-available storage architecture:

GitHub Storage HA/Replication

• Consistency; via automated builds and configuration management. With any horizontally-scaled solution, consistency amongst similar components is essential. One of the most notable achievements across the entire architecture is the complete integration of automated build infrastructure. A new/additional component of the solution can be rapidly built and added to the overall system regardless of the architecture (physical or virtual).
• Redundancy; A simple way to ensure greater uptime and lower the risk of service interruption is to introduce as much redundancy as possible. GitHub is a great example of this practice. Data links, Ethernet/switching, server and components all have a redundant twin ready to swing into action should the primary fail.

Conclusions

The implementation of any new architecture for an already mature product is never easy. Anchor engineers have been working tirelessly with GitHub staff to ensure the any growing pains are transparent to the users. In the next entry, we will be sharing some of our insights in regard to migrating GitHub from their existing host and infrastructure to the new Anchor developed model. Until then, we hope you enjoy the new faster GitHub, more of the time (well, all/any of the time) than ever before.

Many many years ago before working for Anchor I built websites and I remember it sometimes being difficult to make things work the same way in every different browser available.

These days I’m a Mac user and granted Safari is not the most widely used browser but it’s been a while since I came across a site that failed so badly as this one.

In their defence, having told me that my browser might not work – they do provide a link to “proceed” anyway, the link takes me to this page:

Nice touch, and guess where clicking “here” took me to – back to the first page.

I know there will be plenty of people that will say I shouldn’t be surprised, what do you expect etc etc but seriously, it’s just not that hard to make a web page work in something other than IE anymore. WTF!!!!

But wait, there’s more, to get to this most helpful loop I had to go through this, and yes, that’s exactly how this page rendered in my browser.

Neither are most of the other companies on the list.

An emerging trend nowadays is that a lot of businesses have trouble differentiating themselves from their competitors. At this point they usually need to innovate, so they have a “novel” product to sell, or change the way they do business to make themselves more appealing to customers (of course another option is to start slashing prices in an attempt to soak up some of the market, but this simply isn’t sustainable, making it a risky gamble).

Anchor is no exception to this – we firmly believe that our success is in no small part thanks to our quality of customer support and technical prowess. One suggestion that often arises when discussing how we can Do It Better is somehow building better relationships with our customers. The better we understand their goals, the more effectively we can help them.

A lot of other business turn this into a selling point, by offering you a dedicated account manager or point of contact. It’s easy to see why; it makes people feel special, and that makes them feel good – feel-good customers are happy customers. For the business, it’s fairly cheap to do, too. We do this a bit too, but in a less formal manner. A staff member with a lot of experience with one customer tends to pick up their support requests anyway, which works out nicely.

In a perfect world, all else being equal, having someone who knows all your history and systems inside out is much better than having different people work on your business each time you make contact. In the real world, however, it’s never that simple. We’re all for promoting stronger customer relationships, but it’s important to recognise that this can add weaknesses if not done carefully.

Things to think about

• When your tech goes to the pub, what do you do?
• When your tech leaves the company, what do you do?
• Documentation is the key, someone who doesn’t need to document things regularly probably won’t
• A lack of documentation means that anyone else who does have to work on your stuff is working completely blind
• Having someone write notes about their clients when they’re leaving is too late, they’ll forget too much stuff
• Pub/hit-by-a-bus problems don’t give the luxury of pre-departure documentation, anyway
• If you’re running multiple technologies, perhaps different people will be the best to work on the different parts of your system anyway
• Is your “dedicated support person” actually a tech, or just an account manager who acts as a gopher between you and the real technicians? If so, they’re just overhead, and could quite possibly make things harder by slowing down communication and not properly communicating things back and forth
• Some companies will advertise “dedicated support tech” because they’ve only got one person in their support department!

For these reasons we do not believe in providing a “dedicated support person”. What we do offer, however, is access to a team of well trained and experienced system administrators who are all capable of assisting with all your hosting infrastructure. That said, we’ll do our best to get the most appropriate person onto your problem, whether that’s someone who knows your particular setup or someone who is an expert at the technology you’re using or even the particular problem that you’re having. To ensure there is continuity of information across the team we use a ticketing system to make sure that everyone readily has access to all historical information about an issue, and finally, we have a very healthy appreciation of the value in having comprehensive documentation about everything we do.

We use a Barracuda to keep spam out of our email at Anchor. Having overcome some early teething issues and generally handling it with care it does do quite a good job of keeping spam out of our email to the point that it doesn’t really bother you – most of the time.

Perhaps that’s why the delightful email I received from Cosmotel Web Hosting caught my eye this morning – I just don’t get that much spam these days. Note the URL’s use of the words “emailmarketing”, I guess to some that is another name for spam.

My quarantine box always has a good collection of spam covering the ever enlightening topics of how to last longer on the job, how to make my schlong – well, long I guess, and of course all manner of exciting prescription medicines. It would be fair to say that the majority of this doesn’t originate from Australia and those generating could benefit from re-evaluating their ethics.

What surprised me about receiving spam from another hosting company is that as a web hosting provider you spend a not insignificant amount of time blocking spam, dealing with customer complaints about getting too much spam and getting your own mail servers out of spam abuse lists from the occasional overzealous sales cadet. Surely as a hosting provider you’re more aware of the problems with spamming and the illegality of it than the average punter? Surely you would think more than twice before hitting the send button?

For those that aren’t clued up on the legal problems with spam, our guide to responsible email marketing will run you through the Spam Act of 2003. Yes 2003! that’s 6 years since spamming in Australia became illegal (technically a little under 5 as the act only came into effect in April 2004).

Looking on the bright side, this mornings colourful email promising me 99.95% uptime (really, only 21.6 minutes of downtime per month, from a website that appears to be hosted on a DSL link, perhaps we’re wasting money on our bgp implementation and 4 upstreams) for $58/year did make me ask the question – is our government actually doing anything to enforce the Spam Act of 2003?

My Google searches soon led me to the ACMA website where I discovered that they appear to be quite active. They have a plugin not just for the Outlook mail client, but Outlook Express as well. Great, time to bin my Apple and switch to a Windows PC. Dig a little further and to their credit ACMA have made available some very usable alternatives for non-Outlook users to report spam. You can register for an email address to forward messages to our report spam via a web form. I’m impressed.

What happens to it once a complaint is received? According to ACMA the emails go into a database and are used in investigations and proceedings against spammers. They quote some quite impressive statistics on data collection and enforcement activities.

Will the report from an unhappy camper receiving probably one of the less harmful types of spam from another player in their industry be investigated? I’ll let you know if I hear back from ACMA.

A nice quote from CosmoTel’s website: “CosmoTel has different work ethics to our competitors” – yes you certainly do!

p.s It seems I’m not the only one that isn’t happy about the Cosmotel spam or questioning why a web hosting company is spamming!

With hard discs being by far the most common point of failure in servers RAID does wonders for protection against loss of data.

With a RAID array in normal operation we’re in a pretty safe place. We know that we can suffer failure of a drive without loss of data or disruption of service. Once a drive has failed however we’re in a slightly more precarious position. Loss of another drive or damage to the remaining drive could easily cause major problems. At this point the only thing that can protect you can against data loss if you make a mistake is your backups – you did configure backups didn’t you?

Restoring a damaged RAID array is a task that requires extra caution. 

On our range of dedicated servers and vps‘ it’s one of those things that just happens automatically and the client usually only finds out after the problem has been fixed. For our co-location customers however it’s a task that we often find ourselves involved with to lend a helping hand.

With this in mind we’ve started to put together a series of articles discussing the steps we take to restore a Linux RAID array after hard disc failure and recoving from a Windows software RAID failure We hope you find them useful.

We’ve been using Nagios to monitor an ever-increasing number of services on all of the servers that we own at Anchor for a number of years. For the most part the things we monitor have a focus on those that a systems administrator (us in other words) has to deal with. This includes things like CPU load, memory usage, disc space availability, swap usage, server load, availability of core applications such as web servers, data base servers, mail servers. On a given server we typically monitor anywhere from 5 to 25 different attributes.

The end goal of all this monitoring is to ensure that the services on the servers we run are always working.

We can take this a step further though, rather than just monitor the components of the server that are required to keep the websites running, we can monitor in quite detailed ways many of the components of the websites themselves.

At the end of the day, having a monitoring system tell you that a server is healthy and all of the applications are working only goes so far. Ultimately what’s important (to our clients) is that the website is behaving the way that they expect it to.

Since we don’t build any of the websites that we host unfortunately we can’t put in place systems to monitor the innards of an application. To do so requires an intimate knowledge of how the application was built. We do however have a very powerful monitoring system and if the developers of the websites put the hooks into their code, we can monitor these hooks so that both Anchor and the developers can be alerted to the problems.

With these hooks in place, in many cases Anchor will be able to fix the problems, but if we can’t at least we notify the client that there’s a problem (even if it is at 2am in the morning).

In our world, the more monitoring we put in place the greater the uptime of services and the happier our clients are. On this one though, we need our clients’ help. For all Anchor customers on a Fully Managed support pack we do our side of the monitoring free of charge, and for everyone else we can do anything – for a small fee of course.

To help people understand what can be achieved with web application monitoring along with some implementation ideas, we’ve put together this article on website monitoring.

Due to recent growth Anchor is looking for full time customer service guru for immediate start, please find full details of the job listed in-line below:

Anchor Systems is a leading Linux based web hosting provide in the Australian market. We are growing organisation and require intelligent, passionate people who want to learn everything there is to know about Linux system administration and web hosting in general.

Our customer service representatives:

* show initiative and are self motivated who are capable of working in a dynamic fast-paced environment that requires quick learning and intelligent solutions

* have an excellent command of the English language as well as good phone manner

* get a buzz out of assisting end users

* are keen, creative and able to accurately identify problems

* endeavour to provide the absolute best possible customer experience

* have a little experience with either Linux or Windows problem diagnosis

* are familiar with end user applications such as Outlook, Entourage, Internet Explorer, Firefox, WS-FTP, CuteFTP, SSH

* have an attention to detail bordering on obsessive compulsive and have an attitude of doing things right, the first time

Responsibilities in this position include:

* Order processing: Domain name registration, renewals and transfers. Adding DNS records, provisioning of new shared hosting accounts, managing SSH/FTP accounts, addition of databases and users. SSL certificate procurement.

* First Level technical support: end user email configuration, updating DNS records, password resets, follow up calls, FTP/SSH support as well as field other general support enquiries.

* Basic system administration: Webserver configuration, basic changes to apache/IIS configuration. Database administration, covering both MySQL and PostgreSQL including user account addition, database
imports and exports. Mail configuration on both postfix and sendmail.

* career opportunity to move into a Linux system administration role
* casual work environment
* dual head workstation
* pool table, foosball table and beer.

If you feel you would be suitable for this role, Please send your CV preferably in plain text, PDF or if absolutely necessary, word format to jobs@anchor.com.au

This morning as I made my way up the escalators from Wynard station in the city something caught my eye that had kind of been on my mind this last week. It was the very well recognised Australian Made logo, only it was tattooed on a young girls arm. As proud an Aussie as I am, and admittedly not the tattoo type of person, and as cool as it did look it was hard to avoid the cliched “she’ll regret that one day” thoughts.

Australian Made

The tattoo experience got me thinking though, can web hosting be Australian Made? and what is the real difference between Australian hosting and overseas hosting?

As an Australian based provider we obviously only use Australian labour, that’s a big part of what we do. Our offices are in Australia, and I think are owned by an Australian company. The power we use to run our services comes from Australia and I’m sure there are more than a few other local suppliers. But there are some big ones that aren’t so local.

The hardware we run our services on, 100% imported, the software that we rely on – well aside from the bits we wrote ourselves, you’d have to say it’s atleast 90% imported, our data centre – in Australia but owned by a British company. Our network connectivity – only partially Australian owned.

In fairness the rules associated with using the Australian Made logo only seem to require a significant percentage of the products to originate or be substantially modified in Australia. In truth there’s probably not that many things we do in this country these days that are Australian from start to finish. So I think it’s fair to say we’re Australian as we could be, but not entirely Australian Made.

What does this all mean in a hosting sense? Is there really a difference between a local hosting provider and an overseas one? What does an Australian hosting service have going for it? I’ve tried to deal with this in my article on Australian Made Web Hosting.