All Posts By

Barney Desmond

Apr 26 2013
2

The btrfs backup experiment

By | Technical | 4 Comments

Today we’re talking about our experience with btrfs, the next-gen Linux filesystem. btrfs has been maturing rapidly over the last few years and offers many compelling features for modern systems, so we’ve been putting it through its paces on some of our backup servers. How does it stack up? Read on! We chose to test btrfs on backup servers because they can make good use of the features on offer, and the threat-level of data loss is low. For our backups, the biggest benefit comes from copy-on-write and atomic snapshots. At Anchor we use a modified version of Dirvish with support for btrfs: instead of hardlinking directories to provide historical snapshots we just use btrfs’ snapshot facility, which is very quick. Expiring old snapshots is similarly quick – it’s a…

Read More
Apr 18 2013
1

Peering into the private lives of our network admins

By | Company News, Technical | No Comments

In a departure from our usual technical posts, we thought we’d share with you a little of what it’s like to be a network admin here at Anchor. A properly functioning network is critical for a internet services company, and our net-ops team does a fantastic job of ensuring that things keep running smoothly. When they’re not dealing with problems (which are rare) they’re hard at work improving and upgrading our network. Just last week they extended our VoIP PABX to the US and enabled QoS support from end to end. A bit of late-night work was needed to avoid disrupting the support team, but it came off without a hitch. Hang on a moment. What is that, is that… a pony? Zoom in and enhance it.

Read More
Apr 11 2013
1

Nagios checks for iSCSI targets with a blind initiator

By | Technical | No Comments

We’ve recently found ourselves in the situation where we’re managing a highly-available storage service for a customer without actually having direct access to the data on the server. HA storage is commonplace for us now, but not having access to the data is unusual, particularly as a full-stack hosting provider. The reason for this is that the server is presenting iSCSI targets, effectively networked block devices for the clients (“initiators” in iSCSI parlance). We don’t have access to the clients so we can’t find out what they’re doing. In short, there’s no easy access to the data, and it would probably be dangerous for us to try – we’d have to join their OCFS2 cluster to avoid corruption. That doesn’t mean that we can’t monitor them though. As long as…

Read More
Apr 04 2013
2

Redis Rethought: Exciting Extremes with Larger-Than-Memory Datasets

By | Technical | 2 Comments

In a recent post we talked about Redeye, a way to manage a cluster of Redis shards. Sharding is important for pure storage capacity, but also to manage availability and performance, seeing as Redis can be CPU bound in a suffuciently large environment. This time we’re tackling the other side of the equation, simply being able to hold all your data without dropping it. You’ll recall that Redis is purely an in-memory datastore. This makes it amazingly fast, but volatile. To get around this you can have persistence across restarts using RDB and AOF files. The catch is that you’re ultimately limited by the amount of RAM you have available – if you have 500gb of data to store then you need 500gb of RAM, either in a single server…

Read More
Mar 28 2013
1

Limitations of BDB under concurrent access

By | Technical | No Comments

Berkeley DB (BDB) isn’t exactly the most glamorous database engine around, but it’s surprisingly widely deployed and feature rich. It’s designed for embedded use, so you’ll tend to find it integrated into a lot of applications if you go looking. One of our internal development efforts is evaluating various key-value databases as a secondary store to Redis. Because it’s accessed via in-code API calls instead of the network protocols that everyone’s familiar with, access semantics are quite different. BDB supports multiple simultaneous readers and writers, but the circumstances under which this is safe weren’t immediately clear.

Read More
Mar 21 2013
1

Deploying Python webapps with uWSGI

By | Technical | No Comments

If you’ve deployed a number of WSGI apps like Moin and Django in your time, you’re probably familiar with Gunicorn. Like Unicorn for Rack/Rails apps, from which it borrowed the overall design philosophy, it’s straightforward, lightweight and performant. Make no mistaken, Gunicorn does a great job, but there’s a new kid on the block: uWSGI. We’re starting to roll out uWSGI in favour of Gunicorn, so we thought we’d share some of the thinking behind this and what the benefits are. Let’s briefly cover what Gunicorn and uWSGI have in common: They’re both containers that’ll serve up a WSGI app, providing an HTTP connector that a frontend webserver can proxy requests to. This is similar to using mongrel/thin/unicorn for Rails apps, and is very flexible because any frontend webserver (we…

Read More
Mar 15 2013
1

Updated RSpec Cheatsheet

By | Technical | 3 Comments

If you’re a regular reader you’ll know that we’ve been writing a lot of API code recently for internal development. This is a team effort with lots of coding happening in parallel in separate modules, so making sure they all work together correctly is critical. The way to do this is with comprehensive unit-testing at the lower levels, and conformance testing between APIs themselves. We’ve been coding in Ruby, so RSpec was a natural choice. RSpec is nice because it’s practically provided by default (“batteries included!”), really easy to pick up and most importantly, really easy to read. The importance of readability can’t be overstated when you’re working in a team where different people are writing, implementing and validating specifications. Readability is good, but you have to be able to…

Read More
Mar 07 2013
1

Flying with Redeye

By | Technical | No Comments

Several months ago we talked about extending the functionality of Redis, the popular noSQL key-value datastore. Since then we’ve taken things a bit further and we think it’s worth sharing. The Customer needs to store a lot of data in Redis, a few hundred GB at last check, and growing. Redis’ single-threaded nature means it doesn’t scale vertically without bounds though, you can only process so many thousands of transactions per second before you burn up a whole CPU. The solution is to shard your data and scale out horizontally across more CPUs and more RAM, so those hundreds of GB now live on a cluster of about a dozen independent Redis instances. Shard boundaries can’t be easily adjusted, so we created a large number of shards initially and let…

Read More
Feb 28 2013
1

Let’s Play: JRuby

By | Technical | No Comments

One of the cool things to arise out of the last decade or so of programming language development is reimplementations. Plenty of people have said to themselves “why can’t I run $myFavouriteLanguage on the JVM?”. So they go and make it happen! Anchor has historically been a strong Python house, but we’re pragmatic about getting things done. Nagios checks? We’ll write Perl and Ruby if it gets the job done. Automation tasks? Plenty of shell and Perl there. Recently we’ve picked up JRuby. Put simply, JRuby ports the Ruby interpreter to the JVM – you still write pretty much the same Ruby code, but it runs on the JVM and the backtraces are four times longer. Why would you do this? It turns out there’s some really good reasons. Here’s…

Read More