Comments on: GitHub: Speed matters http://www.anchor.com.au/blog/2009/09/github-speed-matters/ A view into the Anchor Engineroom Mon, 06 Feb 2012 04:09:41 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Davy Jones http://www.anchor.com.au/blog/2009/09/github-speed-matters/comment-page-1/#comment-660 Davy Jones Sun, 17 Jan 2010 20:45:58 +0000 http://www.anchor.com.au/blog/?p=1161#comment-660 As it happens, your guess would be wrong. <grin> Github doesn't use any network filesystem at all. Instead, "higher level" requests are made to the fileservers (via work queues, SSH, cron jobs, etc) and local processes do all the IO locally. This is a massive efficiency saving, as it is rare to the point of non-existence that the network response is larger than the amount of disk IO would be if the same job were to be done over NFS, and there's no shared locks or other painful NFS artifacts to deal with. As it happens, your guess would be wrong. <grin> Github doesn’t use any network filesystem at all. Instead, “higher level” requests are made to the fileservers (via work queues, SSH, cron jobs, etc) and local processes do all the IO locally. This is a massive efficiency saving, as it is rare to the point of non-existence that the network response is larger than the amount of disk IO would be if the same job were to be done over NFS, and there’s no shared locks or other painful NFS artifacts to deal with.

]]> By: cn0308 http://www.anchor.com.au/blog/2009/09/github-speed-matters/comment-page-1/#comment-656 cn0308 Sat, 16 Jan 2010 16:48:20 +0000 http://www.anchor.com.au/blog/?p=1161#comment-656 Great article. When it comes to storage, what file system is used for sharing? I would guess NFS. Great article. When it comes to storage, what file system is used for sharing? I would guess NFS.

]]> By: matt http://www.anchor.com.au/blog/2009/09/github-speed-matters/comment-page-1/#comment-243 matt Thu, 01 Oct 2009 02:14:16 +0000 http://www.anchor.com.au/blog/?p=1161#comment-243 Hey Ben, You missed ripping on "refresh" as well... yeah, we let the sales guys out of the cage and it does kinda show. They're back at the martini shaker now. Although it doesn't take a genius to know that buying more hardware or making the code more efficient is the way to speed increases -- we write whole articles on how to do that (like this one: http://www.anchor.com.au/hosting/development/HuntingThePerformanceWumpus) -- for our customers and others to work from. On the other hand, you can't get a commodity machine with 128 cores, 288GB of RAM, and several TB of screamingly-fast disk (at least not for anything approaching a sensible price), and you *certainly* can't rely on being able to upgrade the hardware at the speed that Github is growing. The architecture of the infrastructure (heh) is quite important in hanging all the pieces together, and whilst it doesn't take a genius, I'd say that it *does* take a team that knows what they're doing -- but then, yes, I am biased. For the record, as far as I know, the code that was running on the new hardware was identical to that running on the old hardware at cutover time (modulo the changes needed to use the new distributed repo storage). In other words, there was no "grand optimisation" of the code base when the cutover happened -- the speed boost is down to the way we specced and arranged the machines. Hey Ben,

You missed ripping on “refresh” as well… yeah, we let the sales guys out of the cage and it does kinda show. They’re back at the martini shaker now.

Although it doesn’t take a genius to know that buying more hardware or making the code more efficient is the way to speed increases — we write whole articles on how to do that (like this one: http://www.anchor.com.au/hosting/development/HuntingThePerformanceWumpus) — for our customers and others to work from.

On the other hand, you can’t get a commodity machine with 128 cores, 288GB of RAM, and several TB of screamingly-fast disk (at least not for anything approaching a sensible price), and you *certainly* can’t rely on being able to upgrade the hardware at the speed that Github is growing. The architecture of the infrastructure (heh) is quite important in hanging all the pieces together, and whilst it doesn’t take a genius, I’d say that it *does* take a team that knows what they’re doing — but then, yes, I am biased.

For the record, as far as I know, the code that was running on the new hardware was identical to that running on the old hardware at cutover time (modulo the changes needed to use the new distributed repo storage). In other words, there was no “grand optimisation” of the code base when the cutover happened — the speed boost is down to the way we specced and arranged the machines.

]]> By: Ben http://www.anchor.com.au/blog/2009/09/github-speed-matters/comment-page-1/#comment-227 Ben Wed, 30 Sep 2009 02:31:13 +0000 http://www.anchor.com.au/blog/?p=1161#comment-227 Buzzword alert! Infrastructure architecture? try saying that three times fast. The speedup doesn't require a genius though: move away from VMs, add faster disks, add more RAM & CPU, and optimize the application. Buzzword alert! Infrastructure architecture? try saying that three times fast. The speedup doesn’t require a genius though: move away from VMs, add faster disks, add more RAM & CPU, and optimize the application.

]]>