Tag

project starbug Archives - AWS Managed Services by Anchor

Nov 13 2009
1

Bringing the Mountain to Mohamed

By | Technical | 2 Comments

I have never in my life been asked, “How do porcupines make love?”. However, I know the answer very well: “very carefully”. In the same vein, when migrating the mass of data that makes up Github, you take your time and you work very, very carefully. Since this sort of migration doesn’t happen every day, and it’s not something you want to be learning on the job, I thought I’d write down my experiences for posterity. SCRIPT IT! As a big fan of automation, there wasn’t much chance that this whole thing wasn’t going to be scripted up the wazoo. We just need to copy the filesystem data across, dump the database and load it into the new site… and we’re done. Right? HA! Not likely. To give you an…

Read More
Oct 31 2009
1

Load balancing at Github: Why ldirectord?

By | Technical | 14 Comments

Some comments on Github’s blog post “How We Made Github Fast” have been asking about why ldirectord was chosen as the load balancer for the new site. Since I made most of the architecture decisions for the Github project, it’s probably easiest if I answer that question directly here, rather than in a comment. Why ldirectord rocks The reasons for Github using ldirectord are fairly straightforward: I have a lot of experience with ldirectord. Never underestimate the value of knowing where the bodies are buried. In ldirectord’s case, there aren’t many skeletons, but “better the devil you know” is a valid argument. If you’ve got strong experience in making something work (and you’ve managed to make it work), and you don’t have a lot of time for science experiments, then…

Read More
Sep 30 2009
1

Virtualisation: It’s a Technology, not a Religion

By | Technical | One Comment

It’s been interesting to look at the press coverage, blog posts, and tweets surrounding the move of Github to an Anchor-managed infrastructure — I’ve never worked on something so public before. I think the article about “Vampire Programmers” has been my favourite so far. The ZDnet article on the Github move gave me a wry chuckle, though. It made it sound like the move signified some sort of rejection of the Church of the Hypervisor — that virtualisation had been tested and found wanting. In actual fact, there’s more virtual machines running in the Github infrastructure now than there were previously, providing a lot of very essential services. I really don’t think of myself as a virtualisation nay-sayer. I started using virtual machines with User-Mode Linux, back before anyone outside…

Read More
Sep 29 2009
1

GitHub: Speed matters

By | Technical | 4 Comments

Impressions from the first article (in its first day) and the first 24 hours of the GitHub migration, have caused us at Anchor to believe that; GitHub is just as popular as we thought, The migration was worth it, as things are running much faster (just check your twitter feeds, or better yet, check your GitHub source tree for no reason 😉 ); and, People are interested in what has gone under the hood of the new GitHub (insert your favorite fast car here; otherwise lets say a roadster). Taking these three things into account, this installment will discuss why things are so much faster post migration compared to prior. I said ‘faster’ and not ‘fast’, because GitHub is now as fast as any website should be. So in comparison,…

Read More
Sep 28 2009
1

GitHub: Designing Success

By | Technical | 5 Comments

At Anchor we do not believe in black box solutions.  Sharing is caring and we like to share. In this post we specifically want to share our triumph with Project StarBug, better known to the wider world as GitHub. For the uninitiated, GitHub is ‘Social Networking meets Source Code management’, or in GitHubs own words ‘Git is a fast, efficient, distributed version control system ideal for the collaborative development of software. GitHub is the easiest (and prettiest) way to participate in that collaboration: fork projects, send pull requests, monitor development, all with ease.’. Some readers may protest this point, stating that GitHub is hosted in the USA while Anchor is located in Australia. How then has Anchor architected, implemented and (going forwards) manage GitHub’s infrastructure with such a geographical encumbrance?…

Read More
Sep 25 2009
1

Just because you CAN, Doesn’t mean you SHOULD

By | Technical | No Comments

(Yeah, I’ve been really slack with the blog posts about Project Starbug, but unfortunately when the choice is between doing the cool stuff, and blogging about it, the blogging tends to lose. I am still planning on writing all about things when things die down. In the meantime…) Remember when you were a kid, and every time you got a new toy you’d just have to play with it all the time? That mentality doesn’t go away as you grow up, it just gets a little more sophisticated. With new technologies, I’m still very much this way. I remember when I first learnt about flex and bison — for the next six months or so, every programming problem I encountered just had to be solved with a minilanguage implemented in…

Read More
Aug 06 2009
1

The Zen of Documentation Maintenance

By | Technical | No Comments

Given that you’ve been suddenly and completely convinced of the need for documentation in my previous post, the question still remains: how does one make documentation appear on a consistent and ongoing basis? If you’re really, really lucky, you’ve been spared the painful experience of putting up a wiki somewhere (or, worse, forked out a pile of cash for a “knowledge management system”), sticking some info into it at random, and then… nothing. You planted the seeds of a documentation tree, why isn’t it growing, and flowering, and solving all of your problems for now and forever? For Project Starbug, we’re creating a whole new infrastructure, more-or-less from scratch. This is the easiest possible environment to make work, because you’re not constrained by what is already in place (and that…

Read More
Aug 04 2009
1

Know Thy Enemy

By | Technical | No Comments

Long before any code gets written or any servers deployed, a quiet yet crucial job is being performed. The poor tech who is doing this work won’t get much credit, and almost certainly none of the glory, but if this job isn’t done properly, then none of what gets done later will be of much use. I am, of course, talking about… requirements gathering (bom bom bommmmmm). In the case of project starbug, the requirements gathering work sits at the “fairly straightforward” end of the spectrum — but it’s by no means easy. What makes the job easier than average is that the site is currently operational, and our primary job is making sure that the new server farm that we’re building will (a) match what is currently running (in…

Read More
Aug 03 2009
1

Infrastructure development as performance art

By | Technical | No Comments

Anchor recently signed a new customer. This is not normally news, but then again, this is not a normal customer. They’re fairly sizeable, and need a large scale dedicated infrastructure to handle their request volume. Because of the scale of the development, and some of the novel approaches we’re going with, we’ve decided to blog about the experience of setting it all up. In effect, we’ll be doing the development of this infrastructure in public. Over the next couple of months, as everything comes together, I’ll be regularly writing up what we’re doing, how we’re doing it, and the good, the bad, and the ugly. Some details will need to be obscured, for customer confidentiality reasons, but as much information will be made public as we possibly can. If you’ve…

Read More