More details on Github’s page, come along if you like hanging around hardcore technical people (and beer taps).

Since we first got our hands into their systems about two years ago, Github has grown. A lot. For our sysadmins it’s been a process of gradual but continuous change, and it’s an eye opener to step back and take in the sheer scale of it all. We’d like to share that with you, and also talk about what the changes mean at a higher level.

Taking stock

Late in 2009, our project lead Matt Palmer penned a few technical posts about the size of the new architecture. Comprising some 17 physical servers and a dozen or so VMs, this was a huge upgrade for the whole architecture at the time.

Today Github has 48 pieces of hardware, and about twice as many VMs, with more coming online all the time. There’s about five times as much repo data being hosted, too. And that’s after they figured out how to dedupe all those copies of XBMC that people keep forking.

We’d like to think that this is a testament to the solidarity of our design. It’s not flawless by any means, but it’s proven to be very robust, manageable, and scalable enough to keep pace with Github’s growth.

Self sufficiency

So where to now? A recent count says Github has over a million users, about an eight-fold increase, and their team has grown similarly from five to 47 54 55 56! (for the sake of comparison, Anchor has gone from 18 to 30 staff – our sysadmins are denser).

Github now employs three full-time sysadmins in addition to their customer-facing technical support team, which means around-the-clock coverage. Anchor’s management services let customers leverage our size to get a high level of service at an affordable cost, but as a massively technical company working on a large scale, it makes even more sense for Github to bring it all in-house.

This is something that’s been in the pipeline for a while, and it’s been a pleasantly painless process of handing over the reins. Github’s sysadmins now handle the regular stream of diskspace alerts and other blips on the radar, and new VM services that we know nothing about appear on a regular basis (seriously guys, what’s a “cheddar”?), all thanks to the fully-automated build systems that we developed when the project started.

We wish the Github crew good luck and smooth sailing for the future. They’ve been a great bunch to work with and we look forward to more Octocat adventures.

Anchor will be putting on drinks from 6pm.

Details:
6pm, Monday, Jan 18th
James Squire Brewhouse
22 Promenade King St Wharf, Sydney

I just had a Github customer report that they couldn’t access their repos via SSH, despite it all working properly yesterday, and “not having changed anything”. A bit of debug logging and an inspired leap of intuition on the part of another sysadmin in the office, and the answer was quickly found.

First off, the symptoms:

• Debug logging showed that the user was connecting successfully, presenting six SSH keys (none of which were the key of interest) before disconnecting;
• The SSH key was in the user’s SSH agent (you can verify this with a quick ssh-add -l);
• There were more than six keys in the SSH agent

This last symptom is the key point. As an anti-brute-force measure (I assume), SSH won’t allow a user to connect and present more than MaxAuthTries credentials (whether they be passwords or keys) before being forcibly disconnected. The default value for this parameter (if you haven’t realised already) is six.

Whilst this makes a lot of sense for passwords (and a lesser, but still valid, measure for keys) it does mean that you effectively have a hard limit of six keys in your agent simultaneously (at least without using SSH configs to specify a single key to present to the server). Any more than six keys, and you run the very real risk that the key you need to give to a particular server will be number seven in your agent, and all your authentications will fail miserably.

Bumping the value of MaxAuthTries to a much larger value works fine for Github — password auth is disabled, and if you can manage to brute force a key you’re welcome to what you can get — but you certainly can’t rely on inflating MaxAuthTries everywhere to get you out of trouble, so: keep those SSH agents lean, or at least specify IdentityFile for all your servers.

SCRIPT IT!

As a big fan of automation, there wasn’t much chance that this whole thing wasn’t going to be scripted up the wazoo. We just need to copy the filesystem data across, dump the database and load it into the new site… and we’re done. Right?

HA! Not likely. To give you an idea of the scale of this thing, it took close to 24 hours just to do an rsync scan of the repository filesystems, without actually copying any data. Then there’s the database — the events table alone contained approximately 81.5 million records, which took a great many hours to dump from the live database during pre-migration work. It doesn’t take a great mathematician to realise that copying all this data over the Internet while the site was down for business wasn’t going to fly.

Initially, we were going to rely on the bandwidth of a station wagon full of tapes (or a couple of USB drives in a FedEx jet, anyway) to do the initial copying of data. However, due to some technical problems at the old facility, the “average transfer rate” wasn’t very high (the copy to disk took several weeks to complete), and we ended up kicking off a network-based initial sync of the repository data that finished less than half an hour after the drives were plugged into the machines at the new data centre. While I’m still a fan of shipping disks around for large-scale transfers, I won’t discount using the Internet to transfer such a large data set around so quickly next time.

Incrementalism

Since a single real-time copy wasn’t practical, we’d have to look to incremental copying, where we pre-sync as much data as possible before the Big Cutover Day, and then only copy the latest changes while the site is down.

Thankfully, Github’s software design has pretty much all the hooks we needed to make this a straightforward task. For example, we didn’t have to dump the entire events table, because once a row is written it’s never changed — so we only need to dump events that were created since the last dump.

The system also keeps track of the last time a repository was changed, which means that we can ask the database for a list of repositories that have changed since the last sync, which makes for a very simple (and quick!) incremental sync. For a smaller data set we would just use rsync directly, but due to the performance limitations of the previous hosting environment, this took far, far too long to do with just rsync.

So, we can script everything, and there’s the ability to do repeated incremental syncs. What do these scripts look like?

Well, first up, there’s a lot of them. It was best to write separate scripts to synchronise each data set — one for the repositories, one for the events table, one for the rest of the database, one for gists, and so on. This meant that it was fairly trivial to develop these scripts in parallel, and they could be tested and run independently of each other.

Also, each task that had to be performed for a given data set was in its own script, so each step could be tested independently. For example, the repo sync job consisted of one script to collect the list of repos that needed resyncing and write that list to disk, another script to sync a single repository, and a third script to loop over all the repos listed by the first script and invoke the second script for each of them.

The other important properties of these scripts were:

• We relied heavily on multitasking to overcome bandwidth limitations from a single TCP stream. When you’re copying data over high capacity links, your available transfer rate is constrained more by the round-trip time between the endpoints than the available bandwidth — the longer it takes for an ACK to get back to the sender, the slower your data will flow. So, since we had eight filesystems to copy data from, we fired off eight parallel rsync processes as child processes of the individual scripts.
• Each script kept track of what it was doing and what it had done, and tried to avoid doing the same work again. The repository syncs kept track of the repositories that had already been copied by means of a timestamp file — when we did a sync, we touched a file and then used the mtime of that file (stat -c %Y ftw!) to determine the start time of the next sync. The events table was straightforward — before each dump, we just ask the destination table where it’s up to, and dump from there. Even the “main” database, which we dumped in it’s entireity each time, was dumped to a file compressed with `gzip –rsyncable` before being rsync’d across, saving a good few minutes of network transfer time on each cycle.
• If something went wrong during the sync, we knew about it immediately. We wired up a small SMS sending script to send us alerts if the script terminated improperly. This saved us a lot of waiting and watching, because we knew that we’d be told when we had to take notice of what’s going on.
• Everything was logged. The stdout and stderr of all processes was captured, and the scripts wrote their own log entries to that stream as well as to a “summary” log, like this: echo $(date) processing repo $repo |tee -a $LOGFILE. Any errors were tagged with a unique string and written in a machine-parseable format, so we could re-run any failed components of the sync to ensure that nobody was missed.
• While there were typically several scripts that had to be run in an appropriate order to make a sync happen, there was always a single script that did everything that needed doing — we never had to run more than one command to get a given sync done.

Once all of these individual scripts had been written, tested, debugged, tested a few more times, and generally fretted over until our nails were chewed to the quick, it was time to assemble the master script. I’m not about to run a dozen scripts to migrate a site when one will suffice. This was particularly important in Github’s case because to minimise downtime we wanted to run several things in parallel, then wait until they’d all finished, then run the syncs that depended on the data we’d synced in the last lot, and so on. Our scripts looked a lot like this:

task1 >logs/task1.log 2>&1 &
task1_pid=${!}

task2 >logs/task2.log 2>&1 &
task2_pid=${!}

wait $task1_pid
wait $task2_pid

task3 >logs/task3.log 2>&1 &
task3_pid=${!}

task4 >logs/task4.log 2>&1 &
task4_pid=${!}

task5 >logs/task5.log 2>&1 &
task5_pid=${!}

wait $task3_pid
wait $task4_pid
wait $task5_pid

There was also a pile of “doing this, now doing this, now doing this” logging (with timestamps) that helped us to get a feel for how long the different parts would take, and where everything was up to.

When we actually performed the cutover, the “main” sync script was running for a total of 27 minutes. Given that we’d given ourselves an hour to get everything across, we were all quite pleased with this outcome.

Putting on the brakes

Whilst all these scripts ran really well, and the background processes made everything run really fast, I must say it was a right pain in the butt to stop things mid-flight when it was necessary. Hitting Ctrl-C only stopped the foreground (controller) script, and all of the children that had been started in the background kept flying along.

Doing this again, I’d make sure all my scripts had traps on SIGINT that killed off all the child processes that they had spawned. In retrospect, this is just a variant of “one script to start everything” — you should only need to do one thing (Ctrl-C) to stop it all, as well.

Also, the timestamp files weren’t handled real well. If you did kill things off mid-run (or, heaven forbid, a script crashed out) then the timestamp files would be wrong, because we just did a straight touch at the beginning of the script. What would have been better would be something like this:

touch stamp.new
do_all_the_work
mv stamp stamp.prev
mv stamp.new stamp

This would make sure that premature death would leave the stamp as-is, while still capturing the true start time of the job (which a simple touch at the end would fail to do).

When Databases Attack

Testing the new site before we let users at it, we found that creating gists wasn’t working right. It turned out that the database dumping script didn’t have the right set of options, and the schemas of the tables weren’t quite right (no autoincrements), and that was giving gist creation conniptions. Thankfully, the bug in the script was quickly spotted and the database dump was re-run. We even managed to get the second dump and load completed before our scheduled maintenance window was finished. If our scripts hadn’t
been broken down by data set, this resyncing process would have been made a whole lot harder because we wouldn’t have been able to easily run just the parts that needed to be redone.

Once we opened the floodgates of the new site, everything ran happily for a minute or two, and then ground to a halt. The whaaaaa? Poke, prod… hmm, the database is running a bit hotter than I’d expect… whoa! 1500 queries active, all against the events table, with the disks working so hard the heads nearly came out the sides of the cases. What’s going on here?

As it turns out, schema insanity had struck again — this time, some of the indexes on the events table had failed to come across. While we know what happened with the main database dump, this one is still a mystery. How did some of the indexes fail to materialise? We’ve gone over the dumps and can’t find how they got lost. We’re putting it down to yet another case of MySQL doing dumb things without telling anyone.

Limiting the impact

As a final small improvement to the migration process, the site was able to into a “read only” mode, so that users could still browse code and pull from repositories while we were migrating. This made the migration a lot less intrusive for users, because a lot of site functions still worked, especially those made by casual users (who would be less likely to know all about the time of the migration).

Lessons Learnt

Here are a few things I’ll definitely do differently next time:

• Anywhere you’re depending on a third party to execute part of your migration, have a backup plan in case they can’t deliver — and know when you’ll have to execute your backup plan. In our case, knowing exactly how long it would have taken to copy all the data over the Internet and then calculating back, we would have known to start copying over the network a few days earlier than we did.
• Make sure that synchronisation scripts are as easy to stop as they are to start.
• Verify the database schemas completely on the destination DB server by manual inspection, as well as dumping them and comparing to what’s on the source DB server.

I wonder when we’ll get our next Github-scale migration…

Why ldirectord rocks

The reasons for Github using ldirectord are fairly straightforward:

• I have a lot of experience with ldirectord. Never underestimate the value of knowing where the bodies are buried. In ldirectord’s case, there aren’t many skeletons, but “better the devil you know” is a valid argument. If you’ve got strong experience in making something work (and you’ve managed to make it work), and you don’t have a lot of time for science experiments, then there’s a lot to be said for going with what you know.

  This goes beyond simply knowing what to do when things go wrong, of course. You’ll also know how to install and configure it already, how to monitor it, and so on.

  What’s more, in ldirectord’s case I had already proven that it worked in an architecture almost identical to Github’s, and with a similar load profile. At a previous job, I had ldirectord serving a sustained aggregate of 2500 TCP connections per second on a 128MB Xen VM, passing to a large set of backends in a manner almost identical to Github.

• Anchor has a lot of experience with ldirectord. Whilst my experiences are one thing, there’s a lot more to building an infrastructure than just setting it up. I like to take holidays as much as anyone, and so there was no point in using something that nobody else in the company had any experience with, if there was something else that we did all know about.

  Thankfully, ldirectord lined up nicely, since it’s what we use for our other load balancing setups (not setup by me, either — these were already in place before I arrived). This meant that there was already a pile of documentation and knowledge amongst the sysadmin team about ldirectord and it’s quirks. Also, being automation junkies, we already had Puppet dialled in to install and configure ldirectord, and we knew exactly how to monitor it.

• Ldirectord will do the job. With the prior experiences of myself and the rest of the Anchor team, we were confident that ldirectord would do the job, and at the end of the day that’s what really matters.

The Alternatives

It’s all well and good to say “we know it and it works”, but I’m not really expecting anyone to just read that and say “well, OK, I guess we’ll use ldirectord”. In fact, if you apply the above criteria to your own situation, there’s every possibility that you’ll come up with a different answer — and if you’ve never setup a load balancer at all, then you’ve got no experiences to use to guide you.

So, here are the other load balancing options I’ve dealt with, and what I think of them. This might give you a bit of food for thought when choosing your load balancer.

• keepalived. This is the project closest to ldirectord in terms of functionality and operation. It actually uses the same load balancing “core” as ldirectord, IPVS, part of the Linux Virtual Server project. As such, it performs similarly to ldirectord when it comes to actually redirecting requests to backends, and is another excellent choice for load balancing.

  For Github, though, there wasn’t any benefit in using keepalived. Whilst I used keepalived extensively at my last job, nobody else in at Anchor had had much to do with it. Also, keepalived has a built-in failover mechanism, which we didn’t need because we already use Heartbeat/Pacemaker for all our HA/failover requirements. I also feel that keepalived is more complicated when compared directly to ldirectord, largely because of it’s built-in failover capabilities. That’s not to say that combining Pacemaker and ldirectord is dirt simple, but if you’ve already got Pacemaker on hand anyway…

  If all you needed was a HA load balancer, and had no experience with either ldirectord or keepalived, I’d probably recommend keepalived over ldirectord, as it’s one project and one piece of software to do everything you need.

• Load-balancing appliances. Sometimes misleadingly referred to as “hardware” load balancers (they’re still chock full of software, kids — and unlike high-end routers, I don’t know of any true L4 load balancer that has it’s forwarding plane entirely in hardware).

  I loathe these things. They’re expensive, restrictive, slow, and generally cause you a lot more pain and suffering than they’re worth. At my last job, one of my projects was to convert most of one of our existing clusters from a load-balancing appliance to use keepalived. Why would we do this? Because the $100k worth of appliance wasn’t capable of doing the job that $15k worth of commodity hardware and an installation of keepalived were handling with ease — and with capacity to spare. That cluster was our smallest, too, with probably only 2/3 the capacity of the other clusters run by keepalived.

  At the job where I had ldirectord handling 2500 conn/sec, we had also previously used a load-balancing appliance, which was supplied and managed by the hosting provider. It was a management nightmare — we couldn’t get any useful statistics out of it at all, like the conn/sec coming in or going out, and we couldn’t usefully adjust the weightings of each backend (to tune how many connections were going to each different sort of machine) or manage the system in real-time. When we switched to using ldirectord, a small shell script (involving watch and ipvsadm, mostly) was all it took for the CTO to be able to watch exactly how the cluster was performing, in real time, throughout the day. He loved the visibility — and the fact that we were saving several hundred dollars a month didn’t hurt, either.

• haproxy. While we use haproxy extensively within Github, I don’t think haproxy is the right solution as the front-end load balancer for a high volume website. Being a proxy, rather than a simple TCP connection redirector, it has much larger overheads in CPU and memory, and adds more latency to the connections. All of Github’s load balancing is being done out of one small VM, and it barely raises a sweat. The return traffic doesn’t even go back through the load balancer at Github, since we’re using a really neat mode of IPVS that allows the traffic to return to the client directly. While you can throw hardware at the load balancing problem, I still prefer to be efficient where possible.

  Since haproxy makes a second TCP connection, rather than just redirecting an existing one, it mangles the source IP address information — and while you can work around that in HTTP with custom headers, that doesn’t work for other protocols like SSH. I cringe at the thought of trying to defend against a DDoS attack when the most useful piece of diagnostic information (the source IP) can’t be correlated against the actions of an attacker on the site.

  If all you know is haproxy, and you’re running a low-volume site that only has to deal with HTTP(S), then haproxy will probably do the job — it’s certainly handling more connections inside Github than most sites will ever see. However, I’d recommend getting someone who does systems administration full-time (like us!) to install and manage a real load balancer like ldirectord rather than use haproxy, along with keeping your other basic infrastructure on track. Wouldn’t you rather be developing new features rather than dealing with this stuff?

So, there’s one geek’s opinions on load balancing. Questions and comments appreciated, and if you’d like to know more about any part of the Github architecture (or any other aspect of systems administration), please let us know in the comments and I’ll whip up some more blog posts.

The ZDnet article on the Github move gave me a wry chuckle, though. It made it sound like the move signified some sort of rejection of the Church of the Hypervisor — that virtualisation had been tested and found wanting. In actual fact, there’s more virtual machines running in the Github infrastructure now than there were previously, providing a lot of very essential services.

I really don’t think of myself as a virtualisation nay-sayer. I started using virtual machines with User-Mode Linux, back before anyone outside of Cambridge had ever heard of Xen, and I got on board with Xen back in the 2.0 days. I’ve introduced widespread virtualisation at two previous jobs, I was a big supporter of the use of virtualisation at my last job, and I’ve been working on Anchor’s High-Availability VM product recently. Virtualisation hater I ain’t.

Conversely, though, I don’t think VMs are the answer to all the world’s problems. They’re a fantastic opportunity for a lot of sites: everyone can be running on high-quality, server-grade hardware (redundant power, hardware RAID, fast busses, etc) without the need to either purchase or maintain that hardware. Furthermore, each VM, by virtue of it’s isolation, is more easily managed and scaled independently of the other VMs. Need more memory? Allocate it. This box is getting a little overloaded? No problem, just move a VM to another piece of hardware.

The simple fact is that very, very few sites need a whole dedicated server — even an entry-level server is massive overkill for most sites. In this situation, you can either:

• Spend the extra money, assuming that you’ll grow and recoup those costs;
• Buy a cheaper machine, either a basic desktop machine or second-hand server, and take the hit in reliability;
• Use shared hosting, where everyone’s on the same OS installation (which has tradeoffs in control and isolation); or
• Use a virtual machine.

Unsurprisingly, I like the latter option. It saves you money, avoids the reliability headaches of cheaper hardware and the management headaches of shared hosting.

Management is the big on-going cost of most sites. Virtualisation simplifies that by isolating different sites and services from each other, so that when it comes time to scale them, it’s not a big job. Most people who’ve been working as a developer or sysadmin will be able to recall the unpleasant feeling when that big-ball-of-wax that everyone calls “the server” starts to run out of huff, and there’s no better hardware to put it on, and no more software optimisation to be done. The call goes out, “move some services to another server”. Damn.

See, when everything’s on the one machine, they intertwine and become hard to separate. That little hack that Roger The Talented Intern put in to make mail processing run faster? That involved digging into the SMTP server queue and pulling out messages directly; if you separate the web server and the mail server, that’ll break — but I bet you don’t find that out until you move.

I hate doing archaeology on these sorts of machines, because it’s guaranteed that things will break, tempers will run hot, and sadness will result. The cost of doing the move (in IT staff time, downtime, customer and staff dissatisfaction, and so on) can easily equal or exceed that of the hardware itself — and yes, I’m still talking about good-quality, server-grade hardware here. People are expensive, and good people even more so.

Instead, if you run logically separate services in separate VMs, when the time comes to scale something, it really is a piece of cake to migrate a VM — shutdown, copy the disk image, boot it back up. Piece of cake. Sure, there’s some overhead in running those separate services in VMs, and yes, you’ll be looking to buy a second machine sooner than you would otherwise, but again, the savings made by not having to gently tease apart a dozen root-bound systems on a single machine will probably pay for that second machine. Let’s not even consider the costs of another separation in two years time when the services you put onto that other machine need to be separated again…

This use of virtualisation is all well and dandy if you’re one of the vast majority of sites that don’t need to service 125,000 users and 2.5TB of filesystem data. Github, though — they’re one of the (un)lucky few. When you’re using a machine’s worth (or more) of processing power on a single service, there’s no benefit to virtualising that. In Github’s case, there’s four physical machines running just the frontend services — each of which has the same specs as the machines that are running the VMs for the site. Sticking the frontend services into VMs in that case would have been a fruitless move. Similarly for the backend file storage, and the database. They’re all single services consuming a machine’s worth (or more) of resources, so we give them physical machines.

Down the track, as Github grows and individual VMs work harder and need more resources, we’ll first increase the size of those VMs, before making the decision to move a power-hungry VM off onto it’s own physical hardware. That’s an easy move — between the natural isolation provided by virtual machines and the strong configuration management policy we’ve adopted, transitioning from a VM to a physical machine will be painless — and painless systems management is, after all, the aim of the game.

1. GitHub is just as popular as we thought,
2. The migration was worth it, as things are running much faster (just check your twitter feeds, or better yet, check your GitHub source tree for no reason ); and,
3. People are interested in what has gone under the hood of the new GitHub (insert your favorite fast car here; otherwise lets say a roadster).

Taking these three things into account, this installment will discuss why things are so much faster post migration compared to prior.

I said ‘faster’ and not ‘fast’, because GitHub is now as fast as any website should be. So in comparison, yes, GitHub is fast now, however it is akin to riding your bicycle with half inflated tires: when fully inflated, suddenly your old bike is blazing fast. Now this is not to be critical of the former architecture which held its merits when GitHub was founded. GitHub had simply moved to a stage where a infrastructure architecture refresh was logical.

The main thing, in the large, that made this new architecture fast was that we were given a blank slate and large amounts of freedom to make an architecture that would do the job well.  This is an incredibly rare thing, and it no doubt took a lot of courage on Github’s part.  For that, we have to say “thankyou” to the Github team for letting us have that freedom.  I like to think that we’ve repaid that trust with a pretty awesome architecture that will serve them well for some time to come.

SCALE: When looking at the new architecture as a whole, the increased scale is immediately evident. GitHub now consumes far more hardware than ever before:

Old Infrastructure:

• 10 VMs
• 39 VCPUs
• 54GB RAM

New Infrastructure:

• 16 physical machines
• 128 physical cores
• 288GB RAM

Or for those who enjoy visual cues:

It is a credit to the old infrastructure and GitHub’s code that it ran so well on so little (in comparison). The first credit for increased performance is increased scale.

An important note regarding the hardware is that there is nothing special (or industry secretive) regarding it. The solution in its entirety is run from commodity hardware. No special black boxes doing scary things with packets and routes. No appliance servers. The solution architecture developed by Anchor can be used with any hardware vendor (insert: Dell, HP, IBM, SuperMicro, etc). Vendor neutrality provides GitHub with no encumbrance with either scaling up or out, a key issue when considering growth and future flexibility.

Note: The architectures flexibility allows for the user repository storage to be expanded with a mix of vendor hardware (should GitHub ever change hardware vendor). Furthermore, any component can be exchanged for another vendor’s hardware with no change to GitHubs architecture or software.

In a nutshell, the increased scale provides:

• More GitHub front-end servers to service your requests;
• More storage; and
• More I/O bandwidth when working with your repository data

HARDWARE PERFORMANCE: The speed specifications of the underlying components is important, in addition to how that hardware is utilised.

Storage I/O: A common factor in poor performance with any solution is an I/O bottleneck at the storage level.  This pain was GitHub’s. To alleviate this, not only is the storage now distributed across several servers (distributing the I/O), but it is now running on direct-attached 15,000 RPM SAS disks on battery-backed hardware RAID. Therefore, the second credit for increased performance is faster storage.

Direct access to hardware: Virtualisation is great. What isn’t great is when virtualisation is used as a universal solution. At Anchor we believe there is a place for virtualisation, and systems with massive I/O or CPU requirements is not that place. By moving resource heavy systems onto dedicated hardware, any contention for resources between individual VMs is removed. The third credit goes to less overhead.

ARCHITECTURE: Throwing hardware at a scaling problem is an easy solution, but without the right division of resources and the right software to properly use it, it’s not going to run real fast.

For GitHub, this was their innovative Git command proxying systems, which do an excellent job of taking requests from the frontends (where users connect with their web browser, git client, or SSH client) and shipping them to the fileservers.  The database structure, filesystem layout, and code efficiency also contribute to this.

Given that the software isn’t our speciality, there’s not a lot for us to say about this, but Github are planning a series of posts on their blog, and I’m quite sure it’ll be enlightening.

TO REVIEW: The factors involved in GitHub’s faster response on the new infrastructure include (but are not limited to):

• Increased Infrastructure (Scale)
• Faster Hardware ( Storage)
• No resource contention (More resources per server)
• Solid, scalable architecture (Awesomeness)

Keep an eye on this space, as we delve into technology specific posts regards what kinds of 11 herbs and spices Anchor used to realise the new GitHub architecture.

Some readers may protest this point, stating that GitHub is hosted in the USA while Anchor is located in Australia. How then has Anchor architected, implemented and (going forwards) manage GitHub’s infrastructure with such a geographical encumbrance?

All will be revealed in a blog entry in three of many parts.

Part 1: (This Post) Designing for success (Otherwise known as: Making GitHub’s dream a reality and nightmares a thing of the past)

Part 2: Speed matters

Part N: (To be announced)

For obvious reasons, we cannot expose GitHub’s architecture in full, however we are sharing some of the more interesting technologies/architecture we have implemented, and the rationale for doing so. Essentially what we have done to make GitHub’s dreams a reality.

Geographical encumbrance

It is a credit to GitHub’s management that they were willing to look the world over for the right team to support them. While they do not want to be harried by anything outside the GitHub application (i.e. Hardware, O/S, Management, etc), they still needed to ensure that the right company was employed to look after these components.

Why Anchor? Anchor’s flexibility to manage a solution on third-party hosted hardware (anywhere in the world) and versatility in developing an architecture to suit this scenario were part of the rationale. Anchor’s reputation for needing to know how technology works (again, no black boxes) and then working out how to improve it was a major contribution.

Enough fluff, now to the meat;

One can imagine that the architecture required to support GitHub is complex mix. We won’t lie; there are many moving parts. Some of the key criteria for designing the solution included:

Scalability

GitHub states it growth as “400 new users and 1000 new repositories every day”. Post migration GitHub will be running on infrastructure spread across 15+ physical hosts/servers. It is essential that the infrastructure can grow with the user base, from 10’s  to 100’s of servers, without the need to re-architect everything. Without a doubt, growing without the associated pain is a major objective for GitHub as it moves forward.

Interesting Note: GitHub’s new physical infrastructure (at migration) consists of:

• 15+ physical servers
• 10+ virtual servers
• 128 physical processor cores
• Over 288GBs RAM
• 1TB+ of storage

GitHub’s software architecture is modular by nature and scalability friendly. Components outside the core software, however, were not as readably scalable. This has been achieved with the following improvements;

• Distributed Storage Architecture (with real-time slaves). Distribution of GitHub’s source code repos across multiple partitions and multiple nodes (including redundant slaves) provided improvements in performance, scalability and reliability. By removing the limitation of using a single filesystem volume for storage, the issue of dealing with large scale storage has been avoided. New partitions can be rapidly added on demand with little to no fuss.

The graphic below illustrates a simplified request to the distributed file storage repo:

GitHub Distributed Repo Storage

• (Sensible) Virtualisation. Previously, GitHub’s infrastructure was entirely virtualised. While virtualisation has its merits, there are reasons to avoid it. Services that aren’t I/O-heavy can be virtualised, while components with high I/O requirements are run on dedicated (“bare metal”) servers. For GitHub, this means file storage and databases are not virtualised. Otherwise, virtualisation is used to provide a mix of server consolidation, rapid deployment and service redundancy/HA.
• Horizontal scalability (on-demand, via automated build infrastructure). The ability to add additional components to the infrastructure in an automated fashion reduces scale-out time and removes user error from builds/configuration. In addition, this also turns the server build/deployment procedure into a measurable deliverable. Over time this can be review and improved (Thank you W. Edwards Deming).

Reliability

As with most businesses, High Availability (or business continuance) is essential to a success. To achieve this a combination of DRBD, virtualisation, heartbeat and load balancing has been employed.

• Mirroring Data; DRBD is utilised for several purposes.

1. It is used to ensure the redundant (read: slave) storage partitions and nodes are in sync with the active counterparts.
2. DRBD is also key in providing HA functionality across the virtualised environment.

Several Xen hosts are deployed with the following scenario; Server 1 runs VM A(active) B(active) C(offline DRBD mirrored) D(offline DRBD mirrored), and Server 2 runs VM A(offline DRBD mirrored) VM B(offline DRBD mirrored) VM D(active) VM E(active). This provides active failover if either of the virtualisation hosts fail.

The graphic below illustrates the replicated, highly-available storage architecture:

GitHub Storage HA/Replication

• Consistency; via automated builds and configuration management. With any horizontally-scaled solution, consistency amongst similar components is essential. One of the most notable achievements across the entire architecture is the complete integration of automated build infrastructure. A new/additional component of the solution can be rapidly built and added to the overall system regardless of the architecture (physical or virtual).
• Redundancy; A simple way to ensure greater uptime and lower the risk of service interruption is to introduce as much redundancy as possible. GitHub is a great example of this practice. Data links, Ethernet/switching, server and components all have a redundant twin ready to swing into action should the primary fail.

Conclusions

The implementation of any new architecture for an already mature product is never easy. Anchor engineers have been working tirelessly with GitHub staff to ensure the any growing pains are transparent to the users. In the next entry, we will be sharing some of our insights in regard to migrating GitHub from their existing host and infrastructure to the new Anchor developed model. Until then, we hope you enjoy the new faster GitHub, more of the time (well, all/any of the time) than ever before.