All Posts By

Barney Desmond

Mar 02 2015
1

A Year of OpenStack

By | General | One Comment

In late 2013 we looked around the company and asked ourselves the questions any management team has to ask: what are we doing, where do we need to be, and what’s holding us back from getting there? Because Anchor had grown organically across many years, our internal procedures for infrastructure management were spread across a number of tools that weren’t scalable and efficient enough to keep up with the demands of new sales. Furthermore, they weren’t compatible with a product-based self-serve future. There are a number of pieces to addressing that, but certainly it would be tremendously useful to have API-driven software defined infrastructure. The recommendation, universally, was that we needed to consider OpenStack. We were astonished at the degree of rigour and testing that had gone into vetting changes (since the beginning, there have been no un-gated…

Read More
Jun 27 2014
1

UMAD: Knowledge, Now

By | General | No Comments

One of the big problems we face here at Anchor is the weight of knowledge that we carry around. We’ve been in business for almost 15 years, and the amount of information that we’ve acquired in that time is immense, largely due to the fact that we’ve dealt with many different technologies for many unique customers. How things work, how to fix things, how stuff gets done – these things tend to get encoded into people’s brains, and Anchor is no different. Even when it’s written down it can be hard to find that information. This is a challenge because it holds you back. Your ace employees with all the knowledge and experience are often the ones least available to teach new staff, which limits your ability to scale up…

Read More
Oct 28 2013
1

Anchor Movin’ On Out!

By | Company News | No Comments

It’s an exciting time at Anchor as another big move is in the works. We moved to the current premises at 230 Clarence about 3 years ago now. At the time we thought it was huge and had boundless plains to share, but strong business and steady recruitment has seen us grow to about fifty staff now! The exact date for the move isn’t fixed yet, but we’re prepping for the convoy to set sail sometime in late November. The destination? A few blocks east, just across the road from Hyde Park, and a substantial jump in elevation. In the meantime we’ve been doing a bit of spring cleaning, we’re getting rid of anything that we don’t need to take with us. The amount of stuff we’ve got lying around…

Read More
Sep 30 2013
1

Extending PostgreSQL with high level languages (and cats)

By | Technical | No Comments

In a recent post we extolled the virtues of creating your own brand new operators in PostgreSQL. SELECT =^_^= FROM happycats; That’s well and good, but the output was a little lacklustre, returning “meow” for every tuple. We’d like to make it more interesting, and one way to add interesting functionality to Postgres is to embed a procedural language. This lets you juggle data with a little more finesse when it comes to certain operations, compared to the usual relational algebra. We’re going to use Perl because it’s easy to integrate with Postgres, and is generally a quick and dirty way to Get Stuff Done. When embedded in Postgres it’s referred to as PL/Perl. Let’s get started. We begin by “installing” the language into the database in which we wish…

Read More
Sep 23 2013
1

A gentle intro to bughunting

By | Technical | One Comment

A lot of the bughunting that we talk about here is pretty involved, and requires in-depth knowledge of the systems and conventions in play. It’s not exactly conducive to learning if you’re just trying to get started, so we thought we’d take the opportunity to walk through a small bug that we found the other day. It’s really basic, is limited to userspace, and only needs a couple of common tools. We’ll assume that you know a little bit of C and have used gdb to poke around your own code before, but are stuck when it comes to real-world problems. Something that we deploy heavily at Anchor is daemontools, written by Dan Bernstein (aka. DJB). It keeps services running and works in conjunction with a few other small utilities…

Read More
Sep 18 2013
1

Extending PostgreSQL for fun: with cats

By | Technical | 2 Comments

Perhaps you’ve thought I wish I had more cats in my Postgres database before. We certainly have. Just the other day we were lamenting some of the differences between MySQL and PostgreSQL, particularly the way that MySQL has case-insensitive matching using the LIKE operator, while Postgres has LIKE and ILIKE. This got us thinking, it’d be amusing to have more vague (and hilariously unwieldy) operators, such as: SELECT * FROM foo WHERE a VAGUELY RESEMBLES b; Something like this isn’t too implausible. It’s similar to what a full-text search entails, but it’s far from trivial. This got us thinking: if we can’t easily do that, can we at least have some amusing query syntax? Yes we can!

Read More
Aug 28 2013
1

Out-Tridging Tridge

By | Technical | 2 Comments

Andrew Tridgell’s rsync utility is widely used for pushing files around between servers. Anyone can copy files across the network, what makes rsync special is that it compares the files on each end and only transfers the differences, instead of pushing the whole file across the wire when only a few bytes need to be updated. We use rsync to backup servers every single day, though we recently found a few big files where rsync was going to take an eternity to perform what should’ve been several hours of work. So, we busted out the butterfly nets and went to catch us some bugs. rsync in a nutshell Most servers don’t change a great deal on a day to day basis, so transferring just the differences is a very smart…

Read More
Jul 26 2013
1

A most unusual temporal issue

By | Technical | No Comments

Most of us have suffered time-related woes, dealing with servers that just can’t keep their clock straight even with NTP. Well we’ve just come across a new one, and NTP isn’t going to save the day – what happens when you have multiple misbehaving clocks? We first noticed the problem when reloading the firewall on the host, a KVM server that had been virtualised from hardware. We use filtergen for ruleset generation, and one of its postreload scripts restarts fail2ban, an anti-bruteforce tool. One part of fail2ban’s initscript makes a call to sleep 1 to give things time to settle. We noticed that the invocation of sleep was taking more than a second, much more. Something wasn’t right and the automatic 15sec rollback was kicking in, having not received explicit…

Read More
Jul 18 2013
1

Anchor’s new Sydney datacentre presence coming online

By | Technical | No Comments

Our network admin and ops manager were loading hardware into boxes last night before leaving the office. This is somewhat unusual, so I grabbed them to find out what was happening before they escaped. “We’re off to install SYD3” It’s finally here! We’ve been talking about it for a while, and now it’s imminent. We’ve picked up some space in Equinix’s shiny new SY3 datacentre, which is going to be a beachhead for Anchor’s expansion plans. Opening the SYD3 point of presence (POP) is a strategic maneuver. Connectivity will initially be a dark fibre to SYD1, essentially making SYD3 a remote territory that, at the network layer, will be fully internal to SYD1. Once things are settled in, SYD3 will light up with local transit providers and start doing fully…

Read More
Jul 11 2013
1

Slick log aggregation with Kibana

By | Technical | No Comments

Log aggregation is awesome-cool, you already know this. There’s no shortage of solutions for doing this, and you’ve probably heard of a few like Splunk, Graylog, Logstash, Scribe. There’s plenty more that integrate and gather data in various ways, and you can always can roll your own. Having all your logs in one place makes it much easier to spot patterns and perform analysis. This is particularly useful when you have a cluster of machines doing roughly the same thing, as it makes anomalies more obvious and less prone to noise from false positives. One of our devs was itching for a frontend tie into the service-oriented architecture stuff that we’re building at the moment, particularly one that could be easily hacked up and modified, so he went looking and…

Read More