Anchor speaking at LCA2012, come listen!

Published January 16th, 2012 by Barney Desmond

I think the title sums it up nicely. If you needed further incentive to come along, I would proudly inform you that my esteemed colleagues Messrs David Basden and Chris Collins will be discussing the finer points of the automated production of heterogeneous server systems. Activities will commence tomorrow (Tuesday) at half-past-ten in room C001, following the completion of elevenses.

In all seriousness, we do hope you’ll come along if you’re attending Linuxconf and this tickles your fancy:
Any monkey can build the same server over and over again reliably.
But what if you need reliable server builds, and every single one is a little different?

If you’d like a little more in-depth detail, the LCA website has a copy of the abstract for the talk.

In addition to presenting, Anchor is sponsoring LCA and has sent a crack team of DevOperatives to look after things. Come and say Hi if you spot us. :)

0
Comments

Your Magento store + Anchor = ?

Published January 13th, 2012 by Barney Desmond

A little bit of horn-blowing, the correct answer is of course “a winning combination”. :)

We often find ourselves bothered by PHP instead of being hot-and-bothered, but Magento is a pretty well-engineered app. It’s got solid documentation (a godsend), and while it’s very resource intensive if you’re a $5-a-month hosting customer, it’s clear they’ve given a lot of thought to scalability for running a serious online shop.

Scalability? Yes please! If you’re interested in that sort of thing, we recently published a little case study about our friends at Games Paradise, and how we helped them gear up for the Christmas season.

Feel free to get in touch if you have any questions, or want to know more about what we do.

0
Comments

Github forks their sysadmins!

Published December 12th, 2011 by Barney Desmond

With a proud tear in the eye but a heart full of excitement, Anchor announces a parting of ways with Github as their hosting management provider.

Since we first got our hands into their systems about two years ago, Github has grown. A lot. For our sysadmins it’s been a process of gradual but continuous change, and it’s an eye opener to step back and take in the sheer scale of it all. We’d like to share that with you, and also talk about what the changes mean at a higher level.

Taking stock

Late in 2009, our project lead Matt Palmer penned a few technical posts about the size of the new architecture. Comprising some 17 physical servers and a dozen or so VMs, this was a huge upgrade for the whole architecture at the time.

Today Github has 48 pieces of hardware, and about twice as many VMs, with more coming online all the time. There’s about five times as much repo data being hosted, too. And that’s after they figured out how to dedupe all those copies of XBMC that people keep forking.

We’d like to think that this is a testament to the solidarity of our design. It’s not flawless by any means, but it’s proven to be very robust, manageable, and scalable enough to keep pace with Github’s growth.

Self sufficiency

So where to now? A recent count says Github has over a million users, about an eight-fold increase, and their team has grown similarly from five to 47 54 55 56! (for the sake of comparison, Anchor has gone from 18 to 30 staff – our sysadmins are denser).

Github now employs three full-time sysadmins in addition to their customer-facing technical support team, which means around-the-clock coverage. Anchor’s management services let customers leverage our size to get a high level of service at an affordable cost, but as a massively technical company working on a large scale, it makes even more sense for Github to bring it all in-house.

This is something that’s been in the pipeline for a while, and it’s been a pleasantly painless process of handing over the reins. Github’s sysadmins now handle the regular stream of diskspace alerts and other blips on the radar, and new VM services that we know nothing about appear on a regular basis (seriously guys, what’s a “cheddar”?), all thanks to the fully-automated build systems that we developed when the project started.

We wish the Github crew good luck and smooth sailing for the future. They’ve been a great bunch to work with and we look forward to more Octocat adventures.

0
Comments

Signups open for beta test of Anchor’s US presence

Published November 23rd, 2011 by Barney Desmond

Good news, everyone!

Anchor’s US hosting infrastructure is ready for business. We’re not so brazen/naïve as to think that it’s perfect, but we’re pretty damned confident that it’s ready to go.

This is where you come in. Starting in the first week of December, we want to give you a free VPS for three months and see how you like it. We want you to use it, a lot, and we’re not charging for anything – not for the server, not for the bandwidth, nothing. It’s the legendary Anchor products and service, for free.

The server comes sans-management, so you’ve got root and are welcome to run whatever you like on them. If this sounds like your idea of fun, APPLY ONLINE NOW to let us know you’d like to be part of the trial.

Get in quick if you’re interested – we don’t overprovision these things, so we only have a limited number to give away. Applications are subject to acceptance of our beta agreement, and the offer is good for CentOS or Debian Linux only. Have at it!

Tags: , ,
Posted in FTW, Newsletter

 Leave a comment

0
Comments

Dual-stack IPv6/IPv4 as standard on new US deployments

Published November 22nd, 2011 by Barney Desmond

The focus for this post is obviously about our IPv6 deployment plans, but I’d like to take a small detour through our US presence on the way there.

Anchor’s networking and automation gurus have been hard at work preparing our new kit over in the US, and the day we go live is fast approaching. In the process we’ve had literally zero personal presence over there, not one plane ticket was bought. That we can get away with this is mostly thanks to two things: Equinix and DRACs (Dell’s remote-management interface).

Equinix

One of the reasons we went with Equinix is their high level of support, which goes nicely with Anchor’s approach to business.

The servers were dropshipped direct to the LA3 datacentre where staff unpacked them and racked them up according to our instructions. Once the DRACs were plugged in it was our turn to get excited.

DRACs

DRACs let you do stuff remotely, pretty much anything short of implementing a big red self destruct button. Wanna boot an OS install image from 12,000km away? You can do that.

With a little bit of shuffling, we’ve bootstrapped a new environment over there, ready to build high-availability VMs and servers.

The core of Anchor's LAX1 presence, all gigabit and redundant up the wazoo

IPv6

Which brings us to the fun stuff. The question of offering IPv6 has been on the cards for four or five years now, but to date there hadn’t been enough of a business case for us to commit to. The LAX1 POP changes this.

Anchor’s initial offerings will be focused on our VPS product, which will have full dual-stack connectivity from day 0. For those customers with IPv6 aspirations, we’re ready for you. For everyone else, we hope its enabled-by-default status will drive some interest.

As a small mark of this commitment, we’ve given our DRACs IPv6 addresses only. The practical upshot of this is that it forces us to ensure that all our infrastructure supports IPv6 – routing, switching, DNS, the works. This is popularly called “eating your own dogfood”, we wouldn’t sell something that we don’t use ourselves.

Why IPv6?

For those who don’t follow the IPv4 to IPv6 transition, the rest of this post will summarise Anchor’s perspective specifically.

It’s generally agreed that we’re going to run out of the IPv4 addresses that we know and love, and quite soon. Estimates vary, but unless there’s a revolutionary change in our use of addresses it looks like we’ll hit the wall within about three years.

IPv6 support in the backbone of the internet is already well established, but the real challenge is pushing it all the way to end-users. IPv6 requires big changes through the entire technology stack: routers, switches, firewalls, DNS, servers, operating systems, application configurations and more, it’s all there.

This makes it extremely costly to retrofit existing systems. ISPs have a nasty chicken-and-egg problem because of that – given that demand will only be driven by availability, it’s much easier to do nothing instead of committing to big spending with uncertain returns.

Even assuming that an ISP wanted to sell IPv6 to their consumers, there’s very few content providers doing IPv6. Due to aforementioned lack of demand, content providers would be crazy to present IPv6-only content and lose the whole IPv4 market. Given this, it ultimately looks like the shift to IPv6 will be driven by “external” forces.

As an example, big companies like Google are in a position to push for adoption (try visiting http://ipv6.google.com/), and in theory even offer IPv6-only content that enough users would hassle their ISP for. On the other end of the hypothetical stick, if hosting providers increase the cost (technical and monetary) of IPv4 hosting, it could push enough content to IPv6 to grind through a transition.

These are just two examples, but hopefully they illustrate the difficulty and significance of the problem.

0
Comments

Supermicro hardware up for grabs

Published August 19th, 2011 by Barney Desmond

In the last year or so Anchor has been making a strong push towards energy-efficient Dell hardware and VPSes. As a result, we find ourselves with a collection of older Supermicro servers on our hands that still perform well, but aren’t viable to sell to customers any more.

Rather than scrapping them and foisting more e-waste on the environment, we’d prefer to send them to a good home where a geek or hacker can make use of them. If this sounds like you, we’d love you to have a gander at our ebay listings.

To whet your appetite, here’s a sample of what you can expect.

Supermicro Nyanserver 6015P-8R

Supermicro Highroller 6014H-T

Feel free to get in touch if you have any questions about the hardware.

N.B. Nyanserver may not be as nyan as shown, photograph is for demonstrative purposes only.

0
Comments

Resonance Cachecade

Published August 15th, 2011 by Barney Desmond

We don’t normally post about hardware wankery, but this little piece of shininess appeared for free in some of the newer Dell servers we’ve been ordering, and it actually sounds like it’s not an awful hack.

Cachecade is an LSI technology (Dell PERC cards are rebranded gear) that adds a read-cache tier to the RAID logic, in the form of solid-state disks. While SSDs are still too expensive for mass-scale primary storage, they’re cheap enough that you can burn a few hundred bucks and get 50gb worth of faster reads.

The real benefit of this style of read-cache should be for random block reads, where SSDs proverbially drop excrement over rotational media from a great height. The jury is still out for us – we’ve just started using cachecade on a couple of VM hypervisors and a customer DB server, but we’re hoping to see some noticeable impact even on a qualitative basis.

In truth, the performance improvements will be difficult for us to quantify on our own workloads. You can apparently get this functionality if you purchase the new LSI® MegaRAID® CacheCade™ Pro 2.0, but I’d bet that it’s not exposed through something sane (like SNMP) and you’ll be forced to use the perennially-awful MegaCLI tool to get at the data.

0
Comments

“dmesg is like a standup comedy routine”

Published August 3rd, 2011 by Barney Desmond

It happens to the best of us. Like the coyote running headlong off the cliff into open space, you blink a few times, feeling around for a handhold. You’ve just run what should’ve been a pretty safe command, but something’s just not quite right…

And so it was the case for me the other day. In prep for migration of an old desktop-chassis server to a KVM guest, I was zeroing out some blocks for the filesystem on the destination.

dd bs=1M count=1 if=/dev/zero of=/dev/nemo/root

I should note that this really isn’t a good idea when your big shiny VM server is named “nemo”. Barely seconds later, the ever-vigilant nagios was reporting that nemo’s disk was full, and that there were only “-” inodes left. nemo was more sure of itself, with all 64Z of the diskspace used. At least there was apparently 8.4 millibytes left, room to swing a lolcat.

I panicked a bit and had to consciously unclench my jaw muscles. Then I panicked some more. Linux, the tough bastard that it is, was running fine for the most part and colleagues wasted no time SSHing in.

“omg wtf”

“dude what did you do??”

“lol! dmesg is like a standup comedy routine”

The VM guests were ticking over just fine, but nemo was rather unimpressed with this incursion and wouldn’t stand for it one bit.

EXT3-fs error (device dm-0) in ext3_reserve_inode_write: IO failure; Aborting journal on device dm-0.
EXT3-fs error (device dm-0) in ext3_dirty_inode: IO failure; ext3_abort called.
EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal; Remounting filesystem read-only
EXT3-fs error (device dm-0): ext3_readdir: bad entry in directory #2457831: inode out of bounds – offset=0, inode=2457831, rec_len=12, name_len=1
EXT3-fs error (device dm-0): ext3_readdir: bad entry in directory #2457635: inode out of bounds – offset=0, inode=2457635, rec_len=12, name_len=1

It was undeniably bad, but not beyond repair. Just how bad could it be..? The guests are okay, but we need this fixed soon. We thought quickly. Solutions came, but they were all long shots. It’d be S-rank ninjawork if you could pull it off, but it’d be ugly.

  • Attempt to create an identical filesystem, with the same parameters, and steal the first meg? You’ll miss some data, but a fsck might get it consistent again?
    No, too unlikely to work and dirty as all foo.
  • Ooh, mkfs.ext3 is deterministic. You can calculate where the backup superblocks are on the disk and copy one of them out. You’ll probably lose a block group but it’s something work with, then fsck to tidy up?
    That sounds pretty awful, and will fail if the volume has been resized.
  • If you could just dig the superblock out of memory…
    No!

We eventually settled on a restore from backup of the root filesystem, while everything was live. It goes a little something like this:

  1. The machine’s got 64gb of RAM, might as well use it. Luckily our tools are okay and bits of the filesystem structure are cached. Start by making /tmp pretty huge. We could’ve tried to carve out a new LV, but the LVM tools were crapping out because the filesystem was now read-only.
    mount -o remount,size=12000m /tmp
  2. We need a filesystem to restore to, there’s only a couple gig of data. A loopback-mounted file will do the job. Have to use dd to create it though >_<
    dd bs=1G count=0 seek=10 if=/dev/null of=/tmp/recovery_fs
    mkfs.ext3 /tmp/recovery_fs  # this bit is super fast :)
    mkdir /tmp/recovery_target
    mount /tmp/recovery_fs /tmp/recovery_target
  3. Now we go ahead with the restore. You do take backups of all your data, don’t you?
    One point to note here is that we skip mountpoints like /proc, /sys, /dev, /tmp, etc. from our backups. After the restore, you have to create them on the target, otherwise you might have a rough time booting up again.
  4. We’re just about done, but the filesystem is too big for the destination volume. We’re going to shrink it down, which necessitates a little more work.
    umount /tmp/recovery_target
    e2fsck -C0 -f /tmp/recovery_fs  # the filesystem *must* be fsck'd before shrinking
    resize2fs /tmp/recovery_fs 5G   # the destination is 6gb, so we're being very generous on sizes
  5. And now the master-stroke, splat this new, known-good filesystem straight over the one I touched inappropriately earlier.
    dd bs=1G if=/tmp/recovery_fs of=/dev/nemo/root
    5+0 records in
    5+0 records out
    5368709120 bytes (5.4 GB) copied, 3.97025 seconds, 1.4 GB/s
  6. I wish we could get disk performance like that all the time. Finally, remount the filesystem and see what happens…
    mount -o remount,ro /dev/nemo/root
  7. Now the system was well and truly confused, which we’d anticipated. Not content to unceremoniously damage the root filesystem, we’d just completely pulled the rug out from underneath it (and replaced it with a fine Persian carpet!)

Well, the VM guests weren’t quite in production yet… Really, a reboot was going to be the cleanest way of getting all this sorted, and it only takes like five minutes.

With a well-timed kick in the still-blissfully-unaware VMs my colleagues brought them to a quiescent state as I frobbed the virtual power switch to trigger a reboot (remote server management cards are the greatest thing since Netscape’s fishcam). As awesomely as that turned out, I’m not looking forward to doing it again.

0
Comments

SElinux Hates you yeah!

Published November 3rd, 2010 by Barney Desmond
macrossfrontier-08-carrot

Yeah, it's kinda like that...

SElinux is something I haven’t seen much discussion on, so I thought it was worth a mention. For a little context, we used to use SElinux on a number of our servers for about three years, but decided last year that it was impractical to continue doing so. Okay, SElinux doesn’t really hate you, and leaving it at “impractical” is a bit disingenuous, so I’ll elaborate on our experiences and decisions.

As environments go, shared hosting is a particularly complex one. Having hundreds of users on a server with no mutual trust is a high stakes environment and is ripe for any mechanism that promises greater security. Our first experiment was to attempt to retrofit an existing server, but we couldn’t get it to play nice. Undeterred, we decided to start on a fresh system and develop things alongside it.

This worked well, by and large, and we could deal with the challenges it posed, but still found that it caused a lot of headaches for customers. SElinux works great when things are setup as it expects, but most shared hosting systems are anything but.

That’s really what this post boils down to, expectations. “Why doesn’t my webapp Just Work?

It’s a perfectly reasonable question. SElinux works behind the scenes, so it’s not at all obvious to a customer when the mandatory access controls kick in. We can work around these problems, but it breaks expectations badly – not just in terms of function, but our customer experience.

We take pride in looking after our customers, and our frontline support staff can deal with most issues very quickly and effectively. With SElinux in the mix you need to grab the ear of a resident guru and get them to take a look. This all adds up to time that could have been spent helping more customers, for something that needn’t have been a problem in the first place. We don’t mind fixing the problem, that’s part of the tradeoff for security, but why should the customer have to waste their time when the same app would Just Work pretty much anywhere else?

We did, for a time, deploy SElinux on our dedicated servers as well, thinking it’d be manageable. It worked a lot more smoothly once you got used to it, but the conclusion was ultimately the same. Anchor supports a very wide range of customers, and no two server builds are the same. Most customers are migrating an existing setup to us, and this means they’ve got a set of expectations. We cater to this, and unfortunately SElinux doesn’t work well with those expectations.

Ultimately, we found that the tradeoff for deploying SElinux wasn’t justifiable. You have to be objective: it wasn’t helping customers and was wasting their time. Meanwhile, it doesn’t guard against the types of attacks you’re actually likely to see in the wild, which tend to exploit frontend application code. Hands up everyone who uses an open-source CMS? Yeah, I thought so. A well-oiled patching regimen is going to be far more effective. Did you get 0-day’d? Good thing you’ve got those backups, eh?

Anyone successfully deploying SElinux on complex hosting boxes? Step up and claim your cookie :)

1
Comment

Stay-at-home servers

Published November 2nd, 2010 by Barney Desmond

So this is a bit old, but it’s been kicking around my bag of links for a little while. Who said Microsoft didn’t know how to have a little fun?

You can even get a deadtree copy from Amazon. If that doesn’t do it for you, there’s apparently PDFs to be had as well.

0
Comments