SCRIPT IT!

As a big fan of automation, there wasn’t much chance that this whole thing wasn’t going to be scripted up the wazoo. We just need to copy the filesystem data across, dump the database and load it into the new site… and we’re done. Right?

HA! Not likely. To give you an idea of the scale of this thing, it took close to 24 hours just to do an rsync scan of the repository filesystems, without actually copying any data. Then there’s the database — the events table alone contained approximately 81.5 million records, which took a great many hours to dump from the live database during pre-migration work. It doesn’t take a great mathematician to realise that copying all this data over the Internet while the site was down for business wasn’t going to fly.

Initially, we were going to rely on the bandwidth of a station wagon full of tapes (or a couple of USB drives in a FedEx jet, anyway) to do the initial copying of data. However, due to some technical problems at the old facility, the “average transfer rate” wasn’t very high (the copy to disk took several weeks to complete), and we ended up kicking off a network-based initial sync of the repository data that finished less than half an hour after the drives were plugged into the machines at the new data centre. While I’m still a fan of shipping disks around for large-scale transfers, I won’t discount using the Internet to transfer such a large data set around so quickly next time.

Incrementalism

Since a single real-time copy wasn’t practical, we’d have to look to incremental copying, where we pre-sync as much data as possible before the Big Cutover Day, and then only copy the latest changes while the site is down.

Thankfully, Github’s software design has pretty much all the hooks we needed to make this a straightforward task. For example, we didn’t have to dump the entire events table, because once a row is written it’s never changed — so we only need to dump events that were created since the last dump.

The system also keeps track of the last time a repository was changed, which means that we can ask the database for a list of repositories that have changed since the last sync, which makes for a very simple (and quick!) incremental sync. For a smaller data set we would just use rsync directly, but due to the performance limitations of the previous hosting environment, this took far, far too long to do with just rsync.

So, we can script everything, and there’s the ability to do repeated incremental syncs. What do these scripts look like?

Well, first up, there’s a lot of them. It was best to write separate scripts to synchronise each data set — one for the repositories, one for the events table, one for the rest of the database, one for gists, and so on. This meant that it was fairly trivial to develop these scripts in parallel, and they could be tested and run independently of each other.

Also, each task that had to be performed for a given data set was in its own script, so each step could be tested independently. For example, the repo sync job consisted of one script to collect the list of repos that needed resyncing and write that list to disk, another script to sync a single repository, and a third script to loop over all the repos listed by the first script and invoke the second script for each of them.

The other important properties of these scripts were:

• We relied heavily on multitasking to overcome bandwidth limitations from a single TCP stream. When you’re copying data over high capacity links, your available transfer rate is constrained more by the round-trip time between the endpoints than the available bandwidth — the longer it takes for an ACK to get back to the sender, the slower your data will flow. So, since we had eight filesystems to copy data from, we fired off eight parallel rsync processes as child processes of the individual scripts.
• Each script kept track of what it was doing and what it had done, and tried to avoid doing the same work again. The repository syncs kept track of the repositories that had already been copied by means of a timestamp file — when we did a sync, we touched a file and then used the mtime of that file (stat -c %Y ftw!) to determine the start time of the next sync. The events table was straightforward — before each dump, we just ask the destination table where it’s up to, and dump from there. Even the “main” database, which we dumped in it’s entireity each time, was dumped to a file compressed with `gzip –rsyncable` before being rsync’d across, saving a good few minutes of network transfer time on each cycle.
• If something went wrong during the sync, we knew about it immediately. We wired up a small SMS sending script to send us alerts if the script terminated improperly. This saved us a lot of waiting and watching, because we knew that we’d be told when we had to take notice of what’s going on.
• Everything was logged. The stdout and stderr of all processes was captured, and the scripts wrote their own log entries to that stream as well as to a “summary” log, like this: echo $(date) processing repo $repo |tee -a $LOGFILE. Any errors were tagged with a unique string and written in a machine-parseable format, so we could re-run any failed components of the sync to ensure that nobody was missed.
• While there were typically several scripts that had to be run in an appropriate order to make a sync happen, there was always a single script that did everything that needed doing — we never had to run more than one command to get a given sync done.

Once all of these individual scripts had been written, tested, debugged, tested a few more times, and generally fretted over until our nails were chewed to the quick, it was time to assemble the master script. I’m not about to run a dozen scripts to migrate a site when one will suffice. This was particularly important in Github’s case because to minimise downtime we wanted to run several things in parallel, then wait until they’d all finished, then run the syncs that depended on the data we’d synced in the last lot, and so on. Our scripts looked a lot like this:

task1 >logs/task1.log 2>&1 &
task1_pid=${!}

task2 >logs/task2.log 2>&1 &
task2_pid=${!}

wait $task1_pid
wait $task2_pid

task3 >logs/task3.log 2>&1 &
task3_pid=${!}

task4 >logs/task4.log 2>&1 &
task4_pid=${!}

task5 >logs/task5.log 2>&1 &
task5_pid=${!}

wait $task3_pid
wait $task4_pid
wait $task5_pid

There was also a pile of “doing this, now doing this, now doing this” logging (with timestamps) that helped us to get a feel for how long the different parts would take, and where everything was up to.

When we actually performed the cutover, the “main” sync script was running for a total of 27 minutes. Given that we’d given ourselves an hour to get everything across, we were all quite pleased with this outcome.

Putting on the brakes

Whilst all these scripts ran really well, and the background processes made everything run really fast, I must say it was a right pain in the butt to stop things mid-flight when it was necessary. Hitting Ctrl-C only stopped the foreground (controller) script, and all of the children that had been started in the background kept flying along.

Doing this again, I’d make sure all my scripts had traps on SIGINT that killed off all the child processes that they had spawned. In retrospect, this is just a variant of “one script to start everything” — you should only need to do one thing (Ctrl-C) to stop it all, as well.

Also, the timestamp files weren’t handled real well. If you did kill things off mid-run (or, heaven forbid, a script crashed out) then the timestamp files would be wrong, because we just did a straight touch at the beginning of the script. What would have been better would be something like this:

touch stamp.new
do_all_the_work
mv stamp stamp.prev
mv stamp.new stamp

This would make sure that premature death would leave the stamp as-is, while still capturing the true start time of the job (which a simple touch at the end would fail to do).

When Databases Attack

Testing the new site before we let users at it, we found that creating gists wasn’t working right. It turned out that the database dumping script didn’t have the right set of options, and the schemas of the tables weren’t quite right (no autoincrements), and that was giving gist creation conniptions. Thankfully, the bug in the script was quickly spotted and the database dump was re-run. We even managed to get the second dump and load completed before our scheduled maintenance window was finished. If our scripts hadn’t
been broken down by data set, this resyncing process would have been made a whole lot harder because we wouldn’t have been able to easily run just the parts that needed to be redone.

Once we opened the floodgates of the new site, everything ran happily for a minute or two, and then ground to a halt. The whaaaaa? Poke, prod… hmm, the database is running a bit hotter than I’d expect… whoa! 1500 queries active, all against the events table, with the disks working so hard the heads nearly came out the sides of the cases. What’s going on here?

As it turns out, schema insanity had struck again — this time, some of the indexes on the events table had failed to come across. While we know what happened with the main database dump, this one is still a mystery. How did some of the indexes fail to materialise? We’ve gone over the dumps and can’t find how they got lost. We’re putting it down to yet another case of MySQL doing dumb things without telling anyone.

Limiting the impact

As a final small improvement to the migration process, the site was able to into a “read only” mode, so that users could still browse code and pull from repositories while we were migrating. This made the migration a lot less intrusive for users, because a lot of site functions still worked, especially those made by casual users (who would be less likely to know all about the time of the migration).

Lessons Learnt

Here are a few things I’ll definitely do differently next time:

• Anywhere you’re depending on a third party to execute part of your migration, have a backup plan in case they can’t deliver — and know when you’ll have to execute your backup plan. In our case, knowing exactly how long it would have taken to copy all the data over the Internet and then calculating back, we would have known to start copying over the network a few days earlier than we did.
• Make sure that synchronisation scripts are as easy to stop as they are to start.
• Verify the database schemas completely on the destination DB server by manual inspection, as well as dumping them and comparing to what’s on the source DB server.

I wonder when we’ll get our next Github-scale migration…

Why ldirectord rocks

The reasons for Github using ldirectord are fairly straightforward:

• I have a lot of experience with ldirectord. Never underestimate the value of knowing where the bodies are buried. In ldirectord’s case, there aren’t many skeletons, but “better the devil you know” is a valid argument. If you’ve got strong experience in making something work (and you’ve managed to make it work), and you don’t have a lot of time for science experiments, then there’s a lot to be said for going with what you know.

  This goes beyond simply knowing what to do when things go wrong, of course. You’ll also know how to install and configure it already, how to monitor it, and so on.

  What’s more, in ldirectord’s case I had already proven that it worked in an architecture almost identical to Github’s, and with a similar load profile. At a previous job, I had ldirectord serving a sustained aggregate of 2500 TCP connections per second on a 128MB Xen VM, passing to a large set of backends in a manner almost identical to Github.

• Anchor has a lot of experience with ldirectord. Whilst my experiences are one thing, there’s a lot more to building an infrastructure than just setting it up. I like to take holidays as much as anyone, and so there was no point in using something that nobody else in the company had any experience with, if there was something else that we did all know about.

  Thankfully, ldirectord lined up nicely, since it’s what we use for our other load balancing setups (not setup by me, either — these were already in place before I arrived). This meant that there was already a pile of documentation and knowledge amongst the sysadmin team about ldirectord and it’s quirks. Also, being automation junkies, we already had Puppet dialled in to install and configure ldirectord, and we knew exactly how to monitor it.

• Ldirectord will do the job. With the prior experiences of myself and the rest of the Anchor team, we were confident that ldirectord would do the job, and at the end of the day that’s what really matters.

The Alternatives

It’s all well and good to say “we know it and it works”, but I’m not really expecting anyone to just read that and say “well, OK, I guess we’ll use ldirectord”. In fact, if you apply the above criteria to your own situation, there’s every possibility that you’ll come up with a different answer — and if you’ve never setup a load balancer at all, then you’ve got no experiences to use to guide you.

So, here are the other load balancing options I’ve dealt with, and what I think of them. This might give you a bit of food for thought when choosing your load balancer.

• keepalived. This is the project closest to ldirectord in terms of functionality and operation. It actually uses the same load balancing “core” as ldirectord, IPVS, part of the Linux Virtual Server project. As such, it performs similarly to ldirectord when it comes to actually redirecting requests to backends, and is another excellent choice for load balancing.

  For Github, though, there wasn’t any benefit in using keepalived. Whilst I used keepalived extensively at my last job, nobody else in at Anchor had had much to do with it. Also, keepalived has a built-in failover mechanism, which we didn’t need because we already use Heartbeat/Pacemaker for all our HA/failover requirements. I also feel that keepalived is more complicated when compared directly to ldirectord, largely because of it’s built-in failover capabilities. That’s not to say that combining Pacemaker and ldirectord is dirt simple, but if you’ve already got Pacemaker on hand anyway…

  If all you needed was a HA load balancer, and had no experience with either ldirectord or keepalived, I’d probably recommend keepalived over ldirectord, as it’s one project and one piece of software to do everything you need.

• Load-balancing appliances. Sometimes misleadingly referred to as “hardware” load balancers (they’re still chock full of software, kids — and unlike high-end routers, I don’t know of any true L4 load balancer that has it’s forwarding plane entirely in hardware).

  I loathe these things. They’re expensive, restrictive, slow, and generally cause you a lot more pain and suffering than they’re worth. At my last job, one of my projects was to convert most of one of our existing clusters from a load-balancing appliance to use keepalived. Why would we do this? Because the $100k worth of appliance wasn’t capable of doing the job that $15k worth of commodity hardware and an installation of keepalived were handling with ease — and with capacity to spare. That cluster was our smallest, too, with probably only 2/3 the capacity of the other clusters run by keepalived.

  At the job where I had ldirectord handling 2500 conn/sec, we had also previously used a load-balancing appliance, which was supplied and managed by the hosting provider. It was a management nightmare — we couldn’t get any useful statistics out of it at all, like the conn/sec coming in or going out, and we couldn’t usefully adjust the weightings of each backend (to tune how many connections were going to each different sort of machine) or manage the system in real-time. When we switched to using ldirectord, a small shell script (involving watch and ipvsadm, mostly) was all it took for the CTO to be able to watch exactly how the cluster was performing, in real time, throughout the day. He loved the visibility — and the fact that we were saving several hundred dollars a month didn’t hurt, either.

• haproxy. While we use haproxy extensively within Github, I don’t think haproxy is the right solution as the front-end load balancer for a high volume website. Being a proxy, rather than a simple TCP connection redirector, it has much larger overheads in CPU and memory, and adds more latency to the connections. All of Github’s load balancing is being done out of one small VM, and it barely raises a sweat. The return traffic doesn’t even go back through the load balancer at Github, since we’re using a really neat mode of IPVS that allows the traffic to return to the client directly. While you can throw hardware at the load balancing problem, I still prefer to be efficient where possible.

  Since haproxy makes a second TCP connection, rather than just redirecting an existing one, it mangles the source IP address information — and while you can work around that in HTTP with custom headers, that doesn’t work for other protocols like SSH. I cringe at the thought of trying to defend against a DDoS attack when the most useful piece of diagnostic information (the source IP) can’t be correlated against the actions of an attacker on the site.

  If all you know is haproxy, and you’re running a low-volume site that only has to deal with HTTP(S), then haproxy will probably do the job — it’s certainly handling more connections inside Github than most sites will ever see. However, I’d recommend getting someone who does systems administration full-time (like us!) to install and manage a real load balancer like ldirectord rather than use haproxy, along with keeping your other basic infrastructure on track. Wouldn’t you rather be developing new features rather than dealing with this stuff?

So, there’s one geek’s opinions on load balancing. Questions and comments appreciated, and if you’d like to know more about any part of the Github architecture (or any other aspect of systems administration), please let us know in the comments and I’ll whip up some more blog posts.

The ZDnet article on the Github move gave me a wry chuckle, though. It made it sound like the move signified some sort of rejection of the Church of the Hypervisor — that virtualisation had been tested and found wanting. In actual fact, there’s more virtual machines running in the Github infrastructure now than there were previously, providing a lot of very essential services.

I really don’t think of myself as a virtualisation nay-sayer. I started using virtual machines with User-Mode Linux, back before anyone outside of Cambridge had ever heard of Xen, and I got on board with Xen back in the 2.0 days. I’ve introduced widespread virtualisation at two previous jobs, I was a big supporter of the use of virtualisation at my last job, and I’ve been working on Anchor’s High-Availability VM product recently. Virtualisation hater I ain’t.

Conversely, though, I don’t think VMs are the answer to all the world’s problems. They’re a fantastic opportunity for a lot of sites: everyone can be running on high-quality, server-grade hardware (redundant power, hardware RAID, fast busses, etc) without the need to either purchase or maintain that hardware. Furthermore, each VM, by virtue of it’s isolation, is more easily managed and scaled independently of the other VMs. Need more memory? Allocate it. This box is getting a little overloaded? No problem, just move a VM to another piece of hardware.

The simple fact is that very, very few sites need a whole dedicated server — even an entry-level server is massive overkill for most sites. In this situation, you can either:

• Spend the extra money, assuming that you’ll grow and recoup those costs;
• Buy a cheaper machine, either a basic desktop machine or second-hand server, and take the hit in reliability;
• Use shared hosting, where everyone’s on the same OS installation (which has tradeoffs in control and isolation); or
• Use a virtual machine.

Unsurprisingly, I like the latter option. It saves you money, avoids the reliability headaches of cheaper hardware and the management headaches of shared hosting.

Management is the big on-going cost of most sites. Virtualisation simplifies that by isolating different sites and services from each other, so that when it comes time to scale them, it’s not a big job. Most people who’ve been working as a developer or sysadmin will be able to recall the unpleasant feeling when that big-ball-of-wax that everyone calls “the server” starts to run out of huff, and there’s no better hardware to put it on, and no more software optimisation to be done. The call goes out, “move some services to another server”. Damn.

See, when everything’s on the one machine, they intertwine and become hard to separate. That little hack that Roger The Talented Intern put in to make mail processing run faster? That involved digging into the SMTP server queue and pulling out messages directly; if you separate the web server and the mail server, that’ll break — but I bet you don’t find that out until you move.

I hate doing archaeology on these sorts of machines, because it’s guaranteed that things will break, tempers will run hot, and sadness will result. The cost of doing the move (in IT staff time, downtime, customer and staff dissatisfaction, and so on) can easily equal or exceed that of the hardware itself — and yes, I’m still talking about good-quality, server-grade hardware here. People are expensive, and good people even more so.

Instead, if you run logically separate services in separate VMs, when the time comes to scale something, it really is a piece of cake to migrate a VM — shutdown, copy the disk image, boot it back up. Piece of cake. Sure, there’s some overhead in running those separate services in VMs, and yes, you’ll be looking to buy a second machine sooner than you would otherwise, but again, the savings made by not having to gently tease apart a dozen root-bound systems on a single machine will probably pay for that second machine. Let’s not even consider the costs of another separation in two years time when the services you put onto that other machine need to be separated again…

This use of virtualisation is all well and dandy if you’re one of the vast majority of sites that don’t need to service 125,000 users and 2.5TB of filesystem data. Github, though — they’re one of the (un)lucky few. When you’re using a machine’s worth (or more) of processing power on a single service, there’s no benefit to virtualising that. In Github’s case, there’s four physical machines running just the frontend services — each of which has the same specs as the machines that are running the VMs for the site. Sticking the frontend services into VMs in that case would have been a fruitless move. Similarly for the backend file storage, and the database. They’re all single services consuming a machine’s worth (or more) of resources, so we give them physical machines.

Down the track, as Github grows and individual VMs work harder and need more resources, we’ll first increase the size of those VMs, before making the decision to move a power-hungry VM off onto it’s own physical hardware. That’s an easy move — between the natural isolation provided by virtual machines and the strong configuration management policy we’ve adopted, transitioning from a VM to a physical machine will be painless — and painless systems management is, after all, the aim of the game.

1. GitHub is just as popular as we thought,
2. The migration was worth it, as things are running much faster (just check your twitter feeds, or better yet, check your GitHub source tree for no reason ); and,
3. People are interested in what has gone under the hood of the new GitHub (insert your favorite fast car here; otherwise lets say a roadster).

Taking these three things into account, this installment will discuss why things are so much faster post migration compared to prior.

I said ‘faster’ and not ‘fast’, because GitHub is now as fast as any website should be. So in comparison, yes, GitHub is fast now, however it is akin to riding your bicycle with half inflated tires: when fully inflated, suddenly your old bike is blazing fast. Now this is not to be critical of the former architecture which held its merits when GitHub was founded. GitHub had simply moved to a stage where a infrastructure architecture refresh was logical.

The main thing, in the large, that made this new architecture fast was that we were given a blank slate and large amounts of freedom to make an architecture that would do the job well.  This is an incredibly rare thing, and it no doubt took a lot of courage on Github’s part.  For that, we have to say “thankyou” to the Github team for letting us have that freedom.  I like to think that we’ve repaid that trust with a pretty awesome architecture that will serve them well for some time to come.

SCALE: When looking at the new architecture as a whole, the increased scale is immediately evident. GitHub now consumes far more hardware than ever before:

Old Infrastructure:

• 10 VMs
• 39 VCPUs
• 54GB RAM

New Infrastructure:

• 16 physical machines
• 128 physical cores
• 288GB RAM

Or for those who enjoy visual cues:

It is a credit to the old infrastructure and GitHub’s code that it ran so well on so little (in comparison). The first credit for increased performance is increased scale.

An important note regarding the hardware is that there is nothing special (or industry secretive) regarding it. The solution in its entirety is run from commodity hardware. No special black boxes doing scary things with packets and routes. No appliance servers. The solution architecture developed by Anchor can be used with any hardware vendor (insert: Dell, HP, IBM, SuperMicro, etc). Vendor neutrality provides GitHub with no encumbrance with either scaling up or out, a key issue when considering growth and future flexibility.

Note: The architectures flexibility allows for the user repository storage to be expanded with a mix of vendor hardware (should GitHub ever change hardware vendor). Furthermore, any component can be exchanged for another vendor’s hardware with no change to GitHubs architecture or software.

In a nutshell, the increased scale provides:

• More GitHub front-end servers to service your requests;
• More storage; and
• More I/O bandwidth when working with your repository data

HARDWARE PERFORMANCE: The speed specifications of the underlying components is important, in addition to how that hardware is utilised.

Storage I/O: A common factor in poor performance with any solution is an I/O bottleneck at the storage level.  This pain was GitHub’s. To alleviate this, not only is the storage now distributed across several servers (distributing the I/O), but it is now running on direct-attached 15,000 RPM SAS disks on battery-backed hardware RAID. Therefore, the second credit for increased performance is faster storage.

Direct access to hardware: Virtualisation is great. What isn’t great is when virtualisation is used as a universal solution. At Anchor we believe there is a place for virtualisation, and systems with massive I/O or CPU requirements is not that place. By moving resource heavy systems onto dedicated hardware, any contention for resources between individual VMs is removed. The third credit goes to less overhead.

ARCHITECTURE: Throwing hardware at a scaling problem is an easy solution, but without the right division of resources and the right software to properly use it, it’s not going to run real fast.

For GitHub, this was their innovative Git command proxying systems, which do an excellent job of taking requests from the frontends (where users connect with their web browser, git client, or SSH client) and shipping them to the fileservers.  The database structure, filesystem layout, and code efficiency also contribute to this.

Given that the software isn’t our speciality, there’s not a lot for us to say about this, but Github are planning a series of posts on their blog, and I’m quite sure it’ll be enlightening.

TO REVIEW: The factors involved in GitHub’s faster response on the new infrastructure include (but are not limited to):

• Increased Infrastructure (Scale)
• Faster Hardware ( Storage)
• No resource contention (More resources per server)
• Solid, scalable architecture (Awesomeness)

Keep an eye on this space, as we delve into technology specific posts regards what kinds of 11 herbs and spices Anchor used to realise the new GitHub architecture.

Some readers may protest this point, stating that GitHub is hosted in the USA while Anchor is located in Australia. How then has Anchor architected, implemented and (going forwards) manage GitHub’s infrastructure with such a geographical encumbrance?

All will be revealed in a blog entry in three of many parts.

Part 1: (This Post) Designing for success (Otherwise known as: Making GitHub’s dream a reality and nightmares a thing of the past)

Part 2: Speed matters

Part N: (To be announced)

For obvious reasons, we cannot expose GitHub’s architecture in full, however we are sharing some of the more interesting technologies/architecture we have implemented, and the rationale for doing so. Essentially what we have done to make GitHub’s dreams a reality.

Geographical encumbrance

It is a credit to GitHub’s management that they were willing to look the world over for the right team to support them. While they do not want to be harried by anything outside the GitHub application (i.e. Hardware, O/S, Management, etc), they still needed to ensure that the right company was employed to look after these components.

Why Anchor? Anchor’s flexibility to manage a solution on third-party hosted hardware (anywhere in the world) and versatility in developing an architecture to suit this scenario were part of the rationale. Anchor’s reputation for needing to know how technology works (again, no black boxes) and then working out how to improve it was a major contribution.

Enough fluff, now to the meat;

One can imagine that the architecture required to support GitHub is complex mix. We won’t lie; there are many moving parts. Some of the key criteria for designing the solution included:

Scalability

GitHub states it growth as “400 new users and 1000 new repositories every day”. Post migration GitHub will be running on infrastructure spread across 15+ physical hosts/servers. It is essential that the infrastructure can grow with the user base, from 10’s  to 100’s of servers, without the need to re-architect everything. Without a doubt, growing without the associated pain is a major objective for GitHub as it moves forward.

Interesting Note: GitHub’s new physical infrastructure (at migration) consists of:

• 15+ physical servers
• 10+ virtual servers
• 128 physical processor cores
• Over 288GBs RAM
• 1TB+ of storage

GitHub’s software architecture is modular by nature and scalability friendly. Components outside the core software, however, were not as readably scalable. This has been achieved with the following improvements;

• Distributed Storage Architecture (with real-time slaves). Distribution of GitHub’s source code repos across multiple partitions and multiple nodes (including redundant slaves) provided improvements in performance, scalability and reliability. By removing the limitation of using a single filesystem volume for storage, the issue of dealing with large scale storage has been avoided. New partitions can be rapidly added on demand with little to no fuss.

The graphic below illustrates a simplified request to the distributed file storage repo:

GitHub Distributed Repo Storage

• (Sensible) Virtualisation. Previously, GitHub’s infrastructure was entirely virtualised. While virtualisation has its merits, there are reasons to avoid it. Services that aren’t I/O-heavy can be virtualised, while components with high I/O requirements are run on dedicated (“bare metal”) servers. For GitHub, this means file storage and databases are not virtualised. Otherwise, virtualisation is used to provide a mix of server consolidation, rapid deployment and service redundancy/HA.
• Horizontal scalability (on-demand, via automated build infrastructure). The ability to add additional components to the infrastructure in an automated fashion reduces scale-out time and removes user error from builds/configuration. In addition, this also turns the server build/deployment procedure into a measurable deliverable. Over time this can be review and improved (Thank you W. Edwards Deming).

Reliability

As with most businesses, High Availability (or business continuance) is essential to a success. To achieve this a combination of DRBD, virtualisation, heartbeat and load balancing has been employed.

• Mirroring Data; DRBD is utilised for several purposes.

1. It is used to ensure the redundant (read: slave) storage partitions and nodes are in sync with the active counterparts.
2. DRBD is also key in providing HA functionality across the virtualised environment.

Several Xen hosts are deployed with the following scenario; Server 1 runs VM A(active) B(active) C(offline DRBD mirrored) D(offline DRBD mirrored), and Server 2 runs VM A(offline DRBD mirrored) VM B(offline DRBD mirrored) VM D(active) VM E(active). This provides active failover if either of the virtualisation hosts fail.

The graphic below illustrates the replicated, highly-available storage architecture:

GitHub Storage HA/Replication

• Consistency; via automated builds and configuration management. With any horizontally-scaled solution, consistency amongst similar components is essential. One of the most notable achievements across the entire architecture is the complete integration of automated build infrastructure. A new/additional component of the solution can be rapidly built and added to the overall system regardless of the architecture (physical or virtual).
• Redundancy; A simple way to ensure greater uptime and lower the risk of service interruption is to introduce as much redundancy as possible. GitHub is a great example of this practice. Data links, Ethernet/switching, server and components all have a redundant twin ready to swing into action should the primary fail.

Conclusions

The implementation of any new architecture for an already mature product is never easy. Anchor engineers have been working tirelessly with GitHub staff to ensure the any growing pains are transparent to the users. In the next entry, we will be sharing some of our insights in regard to migrating GitHub from their existing host and infrastructure to the new Anchor developed model. Until then, we hope you enjoy the new faster GitHub, more of the time (well, all/any of the time) than ever before.

Remember when you were a kid, and every time you got a new toy you’d just have to play with it all the time? That mentality doesn’t go away as you grow up, it just gets a little more sophisticated. With new technologies, I’m still very much this way. I remember when I first learnt about flex and bison — for the next six months or so, every programming problem I encountered just had to be solved with a minilanguage implemented in flex/bison. I shudder to think that any of that code might still be out there…

Anyway, this week’s shiny new toy has been Heartbeat / Pacemaker. I’ve played with it a fair bit in the past, but just in two-node (Heartbeat v1) clusters. For Project Starbug, though, I’ve been taking it to new heights of awesome (multi-node, easily expandable HA VM clusters, for example). So, of course, anywhere that a bit of high-availability might be good, I’ve laid it on thick. With the Puppet manifests we’ve got for managing Pacemaker, it’s almost harder not to make something HA (seriously, our Pacemaker manifests are awesome).

Unfortunately, in a couple of places I kinda forgot that some services have their own ways of doing HA, and they’re generally superior to tying a service and an IP together and telling Pacemaker to go do it’s thing. The two services that I’ve just converted back away from Heartbeat are NTP and DNS. Yeah, that’s right — I setup pacemaker resources for our NTP server and DNS server, because I suffer from occasional bouts of acute “shiny toy syndrome”. I’ve now recovered, having learnt my lesson (for now).

If you’re really, really lucky, you’ve been spared the painful experience of putting up a wiki somewhere (or, worse, forked out a pile of cash for a “knowledge management system”), sticking some info into it at random, and then… nothing. You planted the seeds of a documentation tree, why isn’t it growing, and flowering, and solving all of your problems for now and forever?

For Project Starbug, we’re creating a whole new infrastructure, more-or-less from scratch. This is the easiest possible environment to make work, because you’re not constrained by what is already in place (and that you can’t afford to get rid of), and the whole thing isn’t in production so there’s no need to get freaked out by the thought of taking a major site off the Internet due to making an ill-advised change — and, most relevantly to this discussion, there’s no giant mass of undocumented… stuff that needs to be picked apart and documented. There’s nothing more deadly to motivation than the idea that when you’ve got this bit documented, there’s only 350,000 other bits to go.

So, if I didn’t want to end up with a shiny, new, incomprehensible and undocumented system, we needed to start focusing on documentation right off the bat and build the documentation alongside the rest of the system. This, in turn, meant that we needed to have something easy to work with, well structured, and above all ready to go before anything else could really kick off.

What to use was a no-brainer. Wikis are straightforward to access and edit, and there’s very little downside to them. We use moin internally for our documentation extensively, so it wasn’t a hard sell to spin up another copy of the wiki software to contain all of the documentation for this project. Most widely-used wiki engines these days are on much the same level, though, and it’s really just a matter of preference which one to use — mostly based around the language you’re most comfortable using (Python == Moin, PHP == MediaWiki, Perl == twiki, Ruby == instiki, Java == something useless and enterprisey), because you really want to be able to write plugins and extensions. One day I’d love to try ikiwiki, because that means I can edit wiki pages without even needing to open my web browser, which will be a particularly special kind of bliss.

Why did we use a separate wiki, though, and not an extension of our existing one? We want to communicate with the customer as well as we possibly can, and the content of the wiki is like a big, persistent communications nexus, and giving the customer (especially this customer, who really knows their stuff) direct access to be able to read all the internal procedures and technical information relating to the management of their infrastructure is a massive boon to communication. Who knows when they might see something we’ve written and say, “Hey, that’s not right!” and fix it? We’re the system administration experts, not the experts in their application, so it makes perfect sense to have them as tightly integrated as possible into the management of the whole infrastructure.

Though we may have made it over “Documentation Hurdle #1″, the race had barely even begun. Plenty of well-intentioned doc projects have gotten something started, and then withered on the vine. The key is to make sure that the documentation stays maintained, and keeping up with the growth of the infrastructure and it’s constant changes. The most important way to do this is to identify the reasons why people don’t keep a reasonably useable documentation repository maintained, and remove those reasons, leaving no possible excuse not to write docs. It needs to be easier to write docs than to not write them, otherwise they’ll get forgotten in the pressure of the moment, and playing catch-up is painful and annoying.

In the next article, we’ll examine why people don’t write docs as often as they know they should, and how to create a “documentation culture” in your team.

I am, of course, talking about… requirements gathering (bom bom bommmmmm).

In the case of project starbug, the requirements gathering work sits at the “fairly straightforward” end of the spectrum — but it’s by no means easy. What makes the job easier than average is that the site is currently operational, and our primary job is making sure that the new server farm that we’re building will (a) match what is currently running (in terms of system setup), and (b) have enough capacity for future growth.

The system configuration to support the customer’s application is fairly easy to achieve for this project — the customer knows their software and what it requires really, really well, and we’ve got the existing setup to examine to try and work out how the pieces fit together if it isn’t immediately obvious. The main requirement here is to make sure that all the requirements are documented thoroughly. Yeah, writing docs isn’t the glamourous end of the job, but it is important, and is something that pays dividends down the line. More on that in another article, though.

The capacity issue is a lot trickier. The new architecture we’re building is completely different to the current architecture (which isn’t scaling well, hence why it’s being left behind), so it’s hard to draw any direct performance metrics by just looking at what hardware is already in use (especially since the current setup uses virtualisation a little too heavily, which makes comparisons based on hardware spec even harder).

Based on a cursory examination of the bottlenecks in the existing system, along with previous knowledge of the system behaviour, I decided that the primary bottleneck of the system is disk I/O. This site isn’t your typical large-scale website; it does a lot more file management than is typical. As a result, the key thing we need to ensure in our new hardware setup is that there is sufficient disk I/O capacity.

Memory constraints (large app servers, mostly) take a close second in the “what is going to kill us here” stakes, as the current infrastructure is using somewhere north of 100GB of RAM (spread across all the various servers that are being used). We want to provision this plus some extra, as moah RAMs == moah disk caching, and moah disk caching == better effective disk I/O. Win all round!

CPU, on the other hand, is practically never an issue. The servers run a lot of separate processes, but they’re almost always waiting on stuff coming from the disk, so with the current state of the art in server CPUs being quad core, we really shouldn’t have a CPU bottleneck.

Although I said earlier that memory and disk I/O were tied for the title of “biggest performance bottleneck”, there was really no competition for which one of these was going to keep me up at night. Solving the memory problem is easy — modern chassis can easily accomodate 32GB (or more) of RAM. There was never any doubt that we’d be using at least a half dozen machines, so stocking them all with 32GB of RAM should be plenty.

No, the worry was always going to be the file I/O capacity, and making sure that we had both the speed we needed, as well as the storage capacity. While the site doesn’t need petabytes of storage, it does need a decent amount of space, and it all needs to be pretty quick. What’s annoying (but understandable) about storage systems is that you can either have a lot of capacity (1.5TB SATA drives are common as MCSEs) or you can have a lot of speed (15k SAS drives max out at 300GB). We could get the storage space we needed with 300GB drives, but will it be quick enough?

To try and make some sort of an apples-to-apples comparison, I needed to have a number that represented how much I/O was being done at present, and which could be compared to what our new hardware infrastructure is capable of.

In the end, what I went with was running the sar tool on a number of the existing machines to try and get an idea of how much disk I/O is being requested by the machines. There are a number of things that might make this comparison inaccurate, but in the end I decided that there wasn’t really any better metric.

The key thing was to try and get the statistics at the same “layer” of the stack in both cases — in this case, when the kernel passes the I/O request off to the disk (or RAID controller, in this case). The benefits of this are that it’s a single statistic to compare, and it’s not ridiculously impossible to synthesise a load at this level for benchmarking purposes (obviously, running the live site on the new infrastructure to benchmark the new hardware isn’t a real winning strategy). When all’s said and done, though, these benchmarks are an estimate, and are unlikely to be completely accurate. That needs to be kept in mind when doing the hardware estimations later on.

All of this information gathering and benchmarking takes a pile of effort, but without it there’s no chance whatsoever that any sizeable infrastructure will be correct for the job it needs to do. I was surprised in this case at how little hardware we ended up needing, however on a previously sized system I worked on the initial guesstimates turned out to be an order of magnitude too low (the system ended up with some thirty-odd servers instead of the five initially ordered). Without a comprehensive analysis of the reality of the situation, you’re either going to end up with a poorly performing site, or else a pile of unused hardware.

Because of the scale of the development, and some of the novel approaches we’re going with, we’ve decided to blog about the experience of setting it all up. In effect, we’ll be doing the development of this infrastructure in public. Over the next couple of months, as everything comes together, I’ll be regularly writing up what we’re doing, how we’re doing it, and the good, the bad, and the ugly. Some details will need to be obscured, for customer confidentiality reasons, but as much information will be made public as we possibly can. If you’ve never been involved in a big infrastructure project, hopefully you’ll be able to get a feel for what goes into something like this.

We’re code naming this sizeable effort “Project Starbug”, and all the blog posts in this series will be tagged with “project starbug”, for ease of identification. Follow along, and watch the adventure unfold…