US POP: Bandwidth Vendor selection process complete

Published December 22nd, 2011 by Keiran Holloway

So on the back of our last blog post on this topic it is with some excitement to announce that we’ve signed up with a total of three network providers to provide our network connectivity in our US Point of Presence.

As mentioned in previous posts, there are essentially two networks that we will be provisioning.

1) Public network which is configured for high-availability and performance in mind. For this link we’ve provisioned two independent network suppliers:

- 100Mbps Fixed-cost link with Level 3
- 100Mbps Fixed-cost link with Hurricane Electric

Across our public network we will be doing all our own BGP routing using the Anchor AS18020 which allows us to completely control how our traffic is routed. Both of these connections are provided as a fibre hand-off which will give us the ability to rapidly increase this all the way up to 1Gbps on each link as necessary.

2) Out-of-bound network which will be used for the remote management of the infrastructure

- 10Mbps fixed cost link with Internap.

This link should provide the reliability necessary for us to continue to comprehensively manage the network from the other side of the world.

conveniently, both of these connections have already been provisioned and deployed which means we’re already operating all our Beta clients on a redundant network with plenty of excess capacity.

0
Comments

Ya gotta admire the chutzpah…

Published December 17th, 2011 by Davy Jones

It’s no secret that here at Anchor, we’re not huge fans of the level of support you get from most commercial software vendors. But a recent incident with a certain vendor of crappy hosting management control panels really took the cake…

It all began, as these things do, on a sunny spring morn. The ticket came in, saying “the control panel says our licence is invalid or expired, even though we paid for a new licence a couple of months ago”. As this tends to cause customer-facing outages, it was a fairly important problem that needed fixing.

(Sidenote: Is it really such a clever idea to run a piece of software that has a feature that is deliberately designed and intended to stop the software from working at the deranged whim of the monkeys who sold it to you? I think not)

Digging into the problem, we could find no obvious cause of the fault — firewall open, packets flowing, manual renewal of the licence via the little button in the web UI seemed to work… all very strange.

As the problem adversely effected the customer’s ability to continue to provide money to the vendor, we thought the vendor might be somewhat keen to help rectify the problem, so as to ensure the ongoing supply of said money. So, we contacted their support department.

“You don’t have an extended super-dooper-bend-over-and-take-it support plan; please pay us $90″, replied the support department, with ‘nary a “how do you do” to soften the blow.

“But wait, we’re trying to ensure the customer can continue to pay you money!”, we replied, on the assumption that the support drone on the other end of the e-mail program was just functionally illiterate (isn’t it great the level of service you get for your money)

“We know. We don’t care. Pay up.” was the curt reply.

Well, doesn’t that just obtain the baked goods. In order to get assistance with paying them money, the vendor wants us to pay yet more money. The logic defies all attempts at analysis or explanation.

Posted in WTF

 Leave a comment

0
Comments

Preparing for the holiday season

Published December 13th, 2011 by Keiran Holloway

For online retailers, Christmas is a worrying time for customers, who will want to know when you’re able to ship to them and whether they can speak to a customer service rep if something goes awry.

Nothing will drive them away sooner than hard-to-find contact information. Bring it to the front of your site for the holiday period, and include the hours your customer service team will be available each day.

Be sure you’re clear with your customers about cut off dates for orders to be purchased, shipped, and delivered well in time for the holidays. Make it easier to find by using direct language like “Shipping Deadlines” or “Shipping Cutoff” rather than polite euphemisms like “Shipping Info” and “Shipping Details”.

Freeze your code

The holiday season is not the time to be making big changes to your code base. Consider freezing your code in the next few days while the full development team are still available and monitor the environment closely to ensure likely problems are identified and resolved before the team takes off for a well-earned break.

Triage and simplify

If you’re a retailer, social sites or sports media publisher and you’re expecting a higher than usual workload over the holidays, consider simplifying to reduce load, by temporarily switching off non-essential resource-intensive code, data or content.

For example, adjusting a retailer catalog for a few peak days could mean a smaller product catalogue, which would be served more quickly and might reduce server load. Removing or disabling resource intensive features/functions, can ensure system resources are dedicated to core functionality.

Check and double-check your monitoring

If a server fails in the forest and nobody hears about it, did it ever really fail? Most definitely yes, but if you don’t hear about it, you can’t take action and while it’s down it’s doing harm to your business. Its safe to assume your team will be making fewer manual checks of the system over the holidays, so now is the time to test and re-test your automated monitoring to make sure that it trips for the right events, and that notifications will be delivered to team members who understand they’ll be responsible for responding over the holidays.

It may also be worth asking us to investigate whether the monitoring tools you are using are negatively impacting site performance.


Backup or be sleighed

We wouldn’t let you get by without any backup strategy at all, but now is a good time to ensure that your backup routine will be appropriate for the holidays, that key staff know where and how to begin rollback and recovery, and that you’re confident you can get it done quickly with your available holiday team members. If in doubt, give the Anchor team a call.

Test or hit Santa’s naughty list

Test early and often to make sure there’s enough capacity available while you’re holidaying, and that loads are balanced correctly. 
Make sure predicted traffic levels are reviewed by all your internal stakeholders and confirmed as realistic.


Check everything twice, and check from front to back — connectivity, to firewall, to application and web servers. Check more than twice to catch that Murphy’s Law instance if you can.

Wishing you a safe, happy and 100% uptime this holidays!

0
Comments

Github forks their sysadmins!

Published December 12th, 2011 by Barney Desmond

With a proud tear in the eye but a heart full of excitement, Anchor announces a parting of ways with Github as their hosting management provider.

Since we first got our hands into their systems about two years ago, Github has grown. A lot. For our sysadmins it’s been a process of gradual but continuous change, and it’s an eye opener to step back and take in the sheer scale of it all. We’d like to share that with you, and also talk about what the changes mean at a higher level.

Taking stock

Late in 2009, our project lead Matt Palmer penned a few technical posts about the size of the new architecture. Comprising some 17 physical servers and a dozen or so VMs, this was a huge upgrade for the whole architecture at the time.

Today Github has 48 pieces of hardware, and about twice as many VMs, with more coming online all the time. There’s about five times as much repo data being hosted, too. And that’s after they figured out how to dedupe all those copies of XBMC that people keep forking.

We’d like to think that this is a testament to the solidarity of our design. It’s not flawless by any means, but it’s proven to be very robust, manageable, and scalable enough to keep pace with Github’s growth.

Self sufficiency

So where to now? A recent count says Github has over a million users, about an eight-fold increase, and their team has grown similarly from five to 47 54 55 56! (for the sake of comparison, Anchor has gone from 18 to 30 staff – our sysadmins are denser).

Github now employs three full-time sysadmins in addition to their customer-facing technical support team, which means around-the-clock coverage. Anchor’s management services let customers leverage our size to get a high level of service at an affordable cost, but as a massively technical company working on a large scale, it makes even more sense for Github to bring it all in-house.

This is something that’s been in the pipeline for a while, and it’s been a pleasantly painless process of handing over the reins. Github’s sysadmins now handle the regular stream of diskspace alerts and other blips on the radar, and new VM services that we know nothing about appear on a regular basis (seriously guys, what’s a “cheddar”?), all thanks to the fully-automated build systems that we developed when the project started.

We wish the Github crew good luck and smooth sailing for the future. They’ve been a great bunch to work with and we look forward to more Octocat adventures.

0
Comments

Getting a start in IT: Part 2

Published December 6th, 2011 by matt

(This is the second part in a two-part series on getting your first job in IT. Part 1 of the series covers getting a job in the first place; this article is about the sort of job you can expect when you’re starting out)

Working in IT, like anything in life, is a learning curve. You start out knowing nothing, and you slowly gain experience and knowledge that allows you to work at a higher level, on more difficult (and, hopefully, interesting) problems.

A difficult fact for many people to accept is that there is no substitute for real experience. TAFE (community college), University, and industry certifications are not equivalents. These academic credentials have their value, but they’re not an equivalent for experience. They’re like power mirrors or window tinting on a car — they add value, but power mirrors don’t get you anywhere by themselves — you need four wheels and an engine.

This means that everyone will “do their time” working at the entry-level jobs, one way or another. Whether you have any sort of relevant academic credentials or not, you will spend some time learning the ropes, at a job which you will probably feel isn’t what you signed up for (very little IT bears any resemblance to scenes from “The Social Network”).

Now, you might luck out, and your “entry level” job might be that you’re the entire IT staff for a small company (that’s what I did at while I was at Uni), in which case you’re both the printer monkey and the CTO, but it’s not an easy road to travel. You’ll be working your ring off just trying to learn enough to be able to do the practical basics of your job, and you will learn a huge pile of bad habits as there’s nobody with experience to tell you “no, don’t do that”.

This isn’t to say that every single person who has a job in IT spent 18 months as a gofer, and there’s no other option. I only said that you needed practical, hands-on experience, not necessarily a pay-check. There are a few ways of getting that experience that doesn’t involve a regular job as someone’s cable monkey, but they do require a certain amount of effort on your part. For development work, it’s fairly easy to get some experience by contributing to open source projects. You learn a lot of really essential skills: the most important of which is to write real code. No course I know of is even a half-decent substitute for writing real code. You also get plenty of exposure to the real technologies and processes used in software development, like revision control, testing, collaboration, communication, and compromise.

Sysadmins have a bit of a harder time because there’s not many public systems that you can administer as a hobby, but running your own server on a cheap VM teaches you an awful lot about the basics of systems administration (whether you learn good things or bad things is actually far less important).

These sorts of “hobbyist” things you can do often blur into actual work experience, too — running a server on the Internet for yourself and a few friends might get you a job through word of mouth setting up a static website and blog for a friend’s dad who’s running his own small business. He tells a friend of his who runs a bigger business, which leads to something else, and so on and so on. All of that’s good experience, and helps to demonstrate that you’ve got a bit more to offer than everyone else.

Even then, though, don’t get your hopes up. SAGE‘s Core Job Descriptions (PDF) defines a Junior Sysadmin as someone with one to three years of full-time experience as a systems administrator.

0
Comments

Getting a start in IT: Part 1

Published December 5th, 2011 by matt

Nobody in the IT industry was born with into it. At some point in the (distant?) past, every one of us was in the position of looking for (or falling into) our first job in the industry. I think us old hands forget how daunting that process was, the memory having been dimmed by the passage of time. So I’ve written a couple of posts (Part 2 will be coming out in a few days) to try and demystify the process a little, for people who might be looking to get their start.

(Obligatory pimpage: Anchor is often looking for good people; keep an eye on our jobs page for new positions as they come up)

Competition for good entry-level jobs in IT is always going to be tough. Whether it’s an employee-friendly market (where there are lots of jobs and not many people who want them), or an employer-friendly market (where there are few jobs and plenty of applicants), you will always be competing against other people who also want the job.

This isn’t a resume article — although I’m going to do one one day soon, because life’s too short to spend it filtering crap resumes. Instead, I’m going to talk about what you can actively do to make yourself a more attractive candidate, especially early in your career. Later on, your work history and job performance count for practically everything, but you can’t get a good work history without landing that first job, so let’s talk about that first.

There are two ways to “get ahead” in the IT industry when you’re starting out. By this, I mean get the “better” entry-level jobs, with a bit better pay and conditions and hopefully a better career progression path. The two methods have different target audiences amongst employers, so you can tune your approach somewhat depending on the sort of job you might like to land:

  • Go to University (College, for some). Consistently get astonishingly high marks, academic awards, scholarships, extra-curricular activities (“President of the University Computing Club”), etc. This will take you to the top of the list primarily at large corporate employers. It shows you’re a very hard worker and very smart (you can’t consistently get top marks without some of both), and it shows you are capable of focusing on tasks that you don’t necessarily find interesting.

    Not all of working life is doing the things you love; being able to stick with something that is important but boring is an important trait at any job, but it is particularly valuable in the corporate world, and hence is highly sought after (and well paid).

  • The other way to get noticed is to live la vida nerda: Basically, being a geek. That’s a very strong hiring point here at Anchor, and at many companies at the less corporate end of the industry.

    “Being a geek” means things like learning programming languages (especially non-mainstream ones) in your own time, because you find it interesting. Having a 19″ rack at home filled with cast-off network and server hardware, running a variety of OSes, which you can’t help but compulsively fiddle with because you just heard about a new OS that runs on anything and uses whalesong-over-IP as it’s native networking protocol. It might also mean explaining in an interview that the reason you didn’t get a good mark in some irrelevant subject is because you were working so hard on making your wyse-60 talk to the BeOS installation you had running on the spare Juniper router that you forgot that you were supposed to be doing the final exam (not that that ever happened to me, oh no)

Of course, the two aren’t mutually exclusive, and in many ways they go together to some degree. If you’re a hardcore geek, the chances are most IT subjects will be much easier, and you’ll get good marks easily. You’ll almost certainly still need to work hard at those subjects that don’t interest you, though, if you want to get top marks across the board — which is why employers look for consistently excellent marks, to make sure you’re not just a geek savant.

A side note on industry certifications: at best, all they will do for you is to increase the amount of theoretical trivia you know. While IT is mostly theoretical trivia, and having more of it can make you more effective, it’s the practical application of that trivia that is the job we do, and quite frankly no degree or industry certification teaches you the ability to get real things done in the real world.

This leads me nicely into the second half of this post, which will appear in a few days’ time, which is all about what your first job in IT will actually involve.

So, there’s the skinny. To get a decent corporate IT job, you’ll be best served by going to Uni and studying like crazy. To get a job at Anchor, good marks don’t hurt, but you get more points for being genuinely interested in technology. if you fit into neither of these categories, you will blend into the dozens, if not hundreds, of other applicants for every job vacancy that comes up on the big job boards, and you will have a hard time landing a good job in IT.

0
Comments