All Posts By

Matt Palmer

Test Flight gains new wings

By | Company News | No Comments

While they’re not live on the new infrastructure we’re building for them quite yet, we just wanted to put out a quick congratulations to TestFlight for their launch of TestFlight Live. Even though we’re not professional iOS devs, we have a keen eye for cool tech, and TestFlight’s approach to helping devs make better apps has appealed to us from the moment they first approached us to build them a new, super-scalable and fully-managed infrastructure in our new US facility. We’ll have a post or two in the future about what we’re doing with Test Flight, when it all goes live. Stay tuned!

Read More

Getting a start in IT: Part 2

By | Technical | No Comments

(This is the second part in a two-part series on getting your first job in IT. Part 1 of the series covers getting a job in the first place; this article is about the sort of job you can expect when you’re starting out) Working in IT, like anything in life, is a learning curve. You start out knowing nothing, and you slowly gain experience and knowledge that allows you to work at a higher level, on more difficult (and, hopefully, interesting) problems. A difficult fact for many people to accept is that there is no substitute for real experience. TAFE (community college), University, and industry certifications are not equivalents. These academic credentials have their value, but they’re not an equivalent for experience. They’re like power mirrors or window tinting…

Read More

Getting a start in IT: Part 1

By | Technical | No Comments

Nobody in the IT industry was born with into it. At some point in the (distant?) past, every one of us was in the position of looking for (or falling into) our first job in the industry. I think us old hands forget how daunting that process was, the memory having been dimmed by the passage of time. So I’ve written a couple of posts (Part 2 will be coming out in a few days) to try and demystify the process a little, for people who might be looking to get their start. (Obligatory pimpage: Anchor is often looking for good people; keep an eye on our jobs page for new positions as they come up) Competition for good entry-level jobs in IT is always going to be tough. Whether…

Read More

Shining some sun on the cloud

By | Company News | No Comments

Being the cynical, hard-bitten sysadmins that we are, we’re a bit skeptical about some of the more grandiose claims about cloud computing: 100% uptime, never having to worry about scalability, and all those other things that people who don’t understand reality seem to get terribly excited about. It’s good to see every now and then that someone else has an experience that matches our own, such as Mixpanel’s decision to move off Rackspace’s cloud and onto dedicated servers. I’d love to know how to negotiate 50%-75% off a vendor’s list price, though…

Read More

Why you should use LVM: part 1319755409 in an infinite series

By | Technical | One Comment

At anchor, we loves us some LVM. It makes managing storage a breeze. Now, we’ve got one more reason to love it. Because now we have lvmsync. Like most modern hosting companies, we run a lot of VPSes, and sometimes the chunk of storage they’re on isn’t where it needs to be, so we have to transfer it around. Ordinarily, this would involve a lengthy period of downtime while dd did it’s business and sent the whole LV across a network. NOT ANY MORE! Now, with the magic of lvmsync we can transfer the bulk of the data while the VM is running, and then do a quick transfer of just the changes after we shut the VM down. We think our customers will appreciate the reduction in downtime, and…

Read More

SSH ControlMaster: The Good, The Bad, The Ugly

By | Technical | 6 Comments

Do you love SSH for the good it has done for mankind, but get annoyed by how long it takes to establish a connection over a high-latency connection? Perhaps you have a process that needs to make thousands of SSH connections, and you’d like a little extra speed from the whole thing. Either way, ControlMaster is your new best friend. The concept is very simple — rather than each new SSH connection to a particular server opening up a new TCP connection, you instead multiplex all of your SSH connections down one TCP connection. The authentication only happens once, when the TCP connection is opened, and thereafter all your extra SSH sessions are sent down that connection. If you’re SSHing between machines on the same LAN, or otherwise a short…

Read More

ERROR: SSH agent has too many keys

By | Technical | No Comments

Unfortunately, SSH doesn’t produce this error, although it darn well should… I just had a Github customer report that they couldn’t access their repos via SSH, despite it all working properly yesterday, and “not having changed anything”. A bit of debug logging and an inspired leap of intuition on the part of another sysadmin in the office, and the answer was quickly found. First off, the symptoms: Debug logging showed that the user was connecting successfully, presenting six SSH keys (none of which were the key of interest) before disconnecting; The SSH key was in the user’s SSH agent (you can verify this with a quick ssh-add -l); There were more than six keys in the SSH agent This last symptom is the key point. As an anti-brute-force measure (I…

Read More

Monitor your servers like it’s 1996

By | Technical | No Comments

Whilst I’m a fan of using percentages for my disk space checks, sometimes an explicit size is more appropriate. So, you’d expect the following to work nicely: $USER1$/check_disk -w 5G -c 1G -p /data/foo If you don’t actually test that this works (by artificially filling your disk and seeing what happens), you may be dismayed to find that you only get alerted when the disk has 5MB of free disk space. Why is this? Because Nagios, despite the fact that nobody has sweated the megabytes for about a gazillion years, doesn’t support ‘G’ as a suffix for thresholds. Oh, it’ll make a good show of pretending — after all, the output formatting options have ‘GB’ as an option — but nope, for your thresholds it’s “5000M” all the way. ROCK…

Read More

Industry Analysts: Putting the “arse” in Analyst

By | Company News | No Comments

I’ve never been a real fan of the output of big “industry analysis” firms, since their reports never seemed to really tell the whole story, and didn’t match up with my experiences “in the trenches”. Now I know why. A representative sample: “I see. So, the companies in your magic quadrant, are they all paying clients of yours?” “Well, yes they are,” He said, proudly. “Well, if they are all paying clients, then what’s so ‘magic’ about being in the quadrant?” “The companies are not all rated at the same level, some are rated much higher than others.” “And should I be surprised to hear that the companies that pay you more so you can afford to have entire teams cover them full-time; you tend to know a lot about,…

Read More