It all began, as these things do, on a sunny spring morn. The ticket came in, saying “the control panel says our licence is invalid or expired, even though we paid for a new licence a couple of months ago”. As this tends to cause customer-facing outages, it was a fairly important problem that needed fixing.

(Sidenote: Is it really such a clever idea to run a piece of software that has a feature that is deliberately designed and intended to stop the software from working at the deranged whim of the monkeys who sold it to you? I think not)

Digging into the problem, we could find no obvious cause of the fault — firewall open, packets flowing, manual renewal of the licence via the little button in the web UI seemed to work… all very strange.

As the problem adversely effected the customer’s ability to continue to provide money to the vendor, we thought the vendor might be somewhat keen to help rectify the problem, so as to ensure the ongoing supply of said money. So, we contacted their support department.

“You don’t have an extended super-dooper-bend-over-and-take-it support plan; please pay us $90″, replied the support department, with ‘nary a “how do you do” to soften the blow.

“But wait, we’re trying to ensure the customer can continue to pay you money!”, we replied, on the assumption that the support drone on the other end of the e-mail program was just functionally illiterate (isn’t it great the level of service you get for your money)

“We know. We don’t care. Pay up.” was the curt reply.

Well, doesn’t that just obtain the baked goods. In order to get assistance with paying them money, the vendor wants us to pay yet more money. The logic defies all attempts at analysis or explanation.

This number is:

(888) 250 8847

This number is manned during our standard business hours and will allow you to get directly in contact with any of our qualified system administrators without any holding or difficult to navigate phone systems.

In addition to this, if you have a dedicated hosting environment or purchase a support pack which gives you 24×7 emergency access to Anchor you will also be supplied with an emergency support number.

We would recommend that all US clients record both of these numbers to ensure that they are always able to get in contact with us when required.

OR is the online retailing industries event of the year, with companies such as PayPal, Google, IBM, Macquarie Telecom, NetRegistry, and of course – Anchor in attendance.

The event will be held at the Sydney Conference and Exhibition Centre.

Check out Online Retailer website for further information!

Anchor will be exhibiting at the event, So why not come along, and enter the draw to win an iPad – one will be drawn each day!

We are also offering FREE setup on our virtual private servers, and a hassle free migration for new customers. Head over to our stand and ask us for further information.

Hope to see you there!

To give you an idea of how much of a problem the data centre is:

• The average Australian home uses between 20 and 30 kWh/day of electricity
• We use around 200kWh/day in our office
• Our data centre is using circa 4800kWh/day!
• We do this from a foot print of around 300 square metres of floor space.

Our approach

• Measure our usage so we can see what  works and what doesn’t.
• Source 100% green power to cover our usage
• Reduce power consumption through increased use of energy efficient server models and virtualisation

Measuring Usage

We operate entirely on managed APC power rails, from these we collect and graph in real time the power usage of every rack we have deployed. We also aggregate the data to provide a view of total power consumption across each data centre presence.

We used the data we collected from the managed power rails to calculate power usage figures on per server level. The nature of our operations is such that server specifications vary from one machine to the next so we collected data across large set and came up with average power consumption per server figures:

• SuperMicro servers (circa 2007 – 2010 generation 1 & 2 RU models): 210 Watts/server
• Dell servers (R410, R510, 2010 onwards): 160 Watts/server
• Dell servers (low voltage models): 140 Watts/server
• Virtual machines: 10-20 Watts/server

These figures relate to physical machines with specifications varying between:

• HDDs: 2 – 8 (SAS/SATA/SSD)
• Memory: 2 – 64 GB
• CPU: 1-2 (Single to Quad Core)

Green Power

The first step we took was to purchase 100% Green Power to cover the power consumption of everything in our data centre with the exception of customers co-located equipment (we took that view that with no control over their provisioning decisions this should remain the clients responsibility). Green Power is a government initiative that guarantees that all energy comes from certified, domestic renewable power generation activities such as wind, solar and hydro.

Reducing demand through virtualisation

As indicated by the figures collected above, virtualisation of servers represents the most drastic reduction in power consumption available. Aside from the operational benefits associated with virtualisation, replacing ageing server hardware with virtualised environments has not only reduced power usage but also provided significant gains in server performance.

Energy efficient Dell servers

Anchor now uses Dell servers across the board for new deployments (you can buy our old power hungry SuperMicro’s on eBay). We’ve seen a minimum of 20% reductions in power consumption by optioning new servers with:

• Intel L series processors (low voltage variants)
• 2.5″ drives in preference to 3.5″
• SSD storage in preference to SAS/SATA where appropriate

As SSD storage volumes increase and price approaches that of traditional storage media deployment of SSD will increase and bring with it further reductions in power consumption. Energy efficient variant of components has been used with a view to avoiding performance degradation, for example, applications known to be CPU bound continue to be deployed on higher power CPUs and so forth.

The cost of going green

Right now the upfront costs of making the changes outlined are material, but when taken into consideration with

• Reduced overhead of physical server management
• Improved efficiency of new server deployments
• Reduced costs of power and data centre space
• Ongoing increases in power and data centre costs

and aggregated over the life of the equipment, these changes make business sense today and offer significant cost savings into the future.

Energy Assesment

This was our first step, to know where power was being used:

The section titled “Office” represents usage from power outlets, predominantly office workstations and internal servers. We still haven’t found where our shop is, but according to our energy auditor it only uses 0.4% of power so we didn’t look too hard for it.

In NSW, the Energy Efficiency for Small Business Program can assist with the cost of energy assessements as well as contributions of up to $5000 towards the cost changes that you make to reduce your energy consumption.

100% Green Power

Our daily power consumption when we first moved in was over 200kWh/day, that’s nearly 10 times the typical Australia home. Whilst we took steps to reduce this we couldn’t eliminate it so we purchased 100% Green Power.

Lighting

The office started with 140 fluorescent tubes, of these we disabled around 50 tubes in areas that either received adequate levels of natural light, weren’t being used, or the people sitting below them were creatures of the dark. In many fittings we found that one of the two tubes provided an adequate and sometimes a more comfortable level of light for the area being lit.

The remaining 90 or so we replaced with Phillips LED tubes reducing power consumption by 50%.

Turning off workstations

Everyone now either turn off or put their office workstations into sleep mode (most of the time). We built a script to enable reporting on which workstations had been left on for enforcement.

No bottled water

Having someone deliver water on the back of a diesel powered truck when we had a tap in the office seemed quite counter-intuitive. We realised most of what we enjoyed about the water dispenser was that it was cold and filtered. We replaced the water that was previously being delivered by truck to a plumbed dispenser with a water filter.

Re-usable take away coffee cups

 
We bought a few hundred of these from Keep Cup, gave most of them away to our customers and kept the rest for office use.

Indoor plants

Our initial attempts at office plants was somewhat of a failure, most of the plants died due to lack of water. So we automated the plant supply and watering, 6 months on our new batch of plants are all still alive.

Other changes we’re considering:

• Gradual shift to increased use of low power workstations or laptops over traditional workstations
• Installation of a power monitoring device such as a Wattson or SEGmeter
• Talk to our landlords about reducing water consumption in bathroom amenities

If you have read this blog before, you will understand how much our SysAdmins love beer. This is why we have come to a company-wide consensus that we will swap our power rails for beer! Let us know if you have some decent beer and we can trade you a power rail! (APC Model: AP7852)

Contact us at 1300 883 979 or support@anchor.com.au – Get in fast, first in, first served! (We are parched!)

Lots of rails

If you require support please email us on support@anchor.com.au

If you need to speak with us please email us and we will call you back.

If you are on IRC you can find us at:

• Server: irc.oftc.net
• Channel: #anchor

Important note:

The loss of power does not have any effect on any hosted services since all hosting equipment is stored in a separate specialist facility with redundant power systems.

What happened?

Energy Australia identified and confirmed the fault to be within their network overnight and have since been carrying out emergency repairs. At this stage we expect power to be restored today (Wednesday the 31st of March) but no specific ETA has been given.

How are we working around this?

All of our level 2 staff have the ability to work remotely. Access to 100% of our systems including server management, email support, monitoring systems, customer information. All level 2 staff are currently working remotely until such time as power is restored in the office.

Updates

10:00 AM – We have one of two phases of power available. Power to phones and office workstations have been restored. We’re still without lights but otherwise business as usual.

1:00 PM – Full power restored and the lights are on again.

Some readers may protest this point, stating that GitHub is hosted in the USA while Anchor is located in Australia. How then has Anchor architected, implemented and (going forwards) manage GitHub’s infrastructure with such a geographical encumbrance?

All will be revealed in a blog entry in three of many parts.

Part 1: (This Post) Designing for success (Otherwise known as: Making GitHub’s dream a reality and nightmares a thing of the past)

Part 2: Speed matters

Part N: (To be announced)

For obvious reasons, we cannot expose GitHub’s architecture in full, however we are sharing some of the more interesting technologies/architecture we have implemented, and the rationale for doing so. Essentially what we have done to make GitHub’s dreams a reality.

Geographical encumbrance

It is a credit to GitHub’s management that they were willing to look the world over for the right team to support them. While they do not want to be harried by anything outside the GitHub application (i.e. Hardware, O/S, Management, etc), they still needed to ensure that the right company was employed to look after these components.

Why Anchor? Anchor’s flexibility to manage a solution on third-party hosted hardware (anywhere in the world) and versatility in developing an architecture to suit this scenario were part of the rationale. Anchor’s reputation for needing to know how technology works (again, no black boxes) and then working out how to improve it was a major contribution.

Enough fluff, now to the meat;

One can imagine that the architecture required to support GitHub is complex mix. We won’t lie; there are many moving parts. Some of the key criteria for designing the solution included:

Scalability

GitHub states it growth as “400 new users and 1000 new repositories every day”. Post migration GitHub will be running on infrastructure spread across 15+ physical hosts/servers. It is essential that the infrastructure can grow with the user base, from 10’s  to 100’s of servers, without the need to re-architect everything. Without a doubt, growing without the associated pain is a major objective for GitHub as it moves forward.

Interesting Note: GitHub’s new physical infrastructure (at migration) consists of:

• 15+ physical servers
• 10+ virtual servers
• 128 physical processor cores
• Over 288GBs RAM
• 1TB+ of storage

GitHub’s software architecture is modular by nature and scalability friendly. Components outside the core software, however, were not as readably scalable. This has been achieved with the following improvements;

• Distributed Storage Architecture (with real-time slaves). Distribution of GitHub’s source code repos across multiple partitions and multiple nodes (including redundant slaves) provided improvements in performance, scalability and reliability. By removing the limitation of using a single filesystem volume for storage, the issue of dealing with large scale storage has been avoided. New partitions can be rapidly added on demand with little to no fuss.

The graphic below illustrates a simplified request to the distributed file storage repo:

GitHub Distributed Repo Storage

• (Sensible) Virtualisation. Previously, GitHub’s infrastructure was entirely virtualised. While virtualisation has its merits, there are reasons to avoid it. Services that aren’t I/O-heavy can be virtualised, while components with high I/O requirements are run on dedicated (“bare metal”) servers. For GitHub, this means file storage and databases are not virtualised. Otherwise, virtualisation is used to provide a mix of server consolidation, rapid deployment and service redundancy/HA.
• Horizontal scalability (on-demand, via automated build infrastructure). The ability to add additional components to the infrastructure in an automated fashion reduces scale-out time and removes user error from builds/configuration. In addition, this also turns the server build/deployment procedure into a measurable deliverable. Over time this can be review and improved (Thank you W. Edwards Deming).

Reliability

As with most businesses, High Availability (or business continuance) is essential to a success. To achieve this a combination of DRBD, virtualisation, heartbeat and load balancing has been employed.

• Mirroring Data; DRBD is utilised for several purposes.

1. It is used to ensure the redundant (read: slave) storage partitions and nodes are in sync with the active counterparts.
2. DRBD is also key in providing HA functionality across the virtualised environment.

Several Xen hosts are deployed with the following scenario; Server 1 runs VM A(active) B(active) C(offline DRBD mirrored) D(offline DRBD mirrored), and Server 2 runs VM A(offline DRBD mirrored) VM B(offline DRBD mirrored) VM D(active) VM E(active). This provides active failover if either of the virtualisation hosts fail.

The graphic below illustrates the replicated, highly-available storage architecture:

GitHub Storage HA/Replication

• Consistency; via automated builds and configuration management. With any horizontally-scaled solution, consistency amongst similar components is essential. One of the most notable achievements across the entire architecture is the complete integration of automated build infrastructure. A new/additional component of the solution can be rapidly built and added to the overall system regardless of the architecture (physical or virtual).
• Redundancy; A simple way to ensure greater uptime and lower the risk of service interruption is to introduce as much redundancy as possible. GitHub is a great example of this practice. Data links, Ethernet/switching, server and components all have a redundant twin ready to swing into action should the primary fail.

Conclusions

The implementation of any new architecture for an already mature product is never easy. Anchor engineers have been working tirelessly with GitHub staff to ensure the any growing pains are transparent to the users. In the next entry, we will be sharing some of our insights in regard to migrating GitHub from their existing host and infrastructure to the new Anchor developed model. Until then, we hope you enjoy the new faster GitHub, more of the time (well, all/any of the time) than ever before.

These days I’m a Mac user and granted Safari is not the most widely used browser but it’s been a while since I came across a site that failed so badly as this one.

In their defence, having told me that my browser might not work – they do provide a link to “proceed” anyway, the link takes me to this page:

Nice touch, and guess where clicking “here” took me to – back to the first page.

I know there will be plenty of people that will say I shouldn’t be surprised, what do you expect etc etc but seriously, it’s just not that hard to make a web page work in something other than IE anymore. WTF!!!!

But wait, there’s more, to get to this most helpful loop I had to go through this, and yes, that’s exactly how this page rendered in my browser.