US POP: Bandwidth Vendor selection process complete

Published December 22nd, 2011 by Keiran Holloway

So on the back of our last blog post on this topic it is with some excitement to announce that we’ve signed up with a total of three network providers to provide our network connectivity in our US Point of Presence.

As mentioned in previous posts, there are essentially two networks that we will be provisioning.

1) Public network which is configured for high-availability and performance in mind. For this link we’ve provisioned two independent network suppliers:

- 100Mbps Fixed-cost link with Level 3
- 100Mbps Fixed-cost link with Hurricane Electric

Across our public network we will be doing all our own BGP routing using the Anchor AS18020 which allows us to completely control how our traffic is routed. Both of these connections are provided as a fibre hand-off which will give us the ability to rapidly increase this all the way up to 1Gbps on each link as necessary.

2) Out-of-bound network which will be used for the remote management of the infrastructure

- 10Mbps fixed cost link with Internap.

This link should provide the reliability necessary for us to continue to comprehensively manage the network from the other side of the world.

conveniently, both of these connections have already been provisioned and deployed which means we’re already operating all our Beta clients on a redundant network with plenty of excess capacity.

0
Comments

Preparing for the holiday season

Published December 13th, 2011 by Keiran Holloway

For online retailers, Christmas is a worrying time for customers, who will want to know when you’re able to ship to them and whether they can speak to a customer service rep if something goes awry.

Nothing will drive them away sooner than hard-to-find contact information. Bring it to the front of your site for the holiday period, and include the hours your customer service team will be available each day.

Be sure you’re clear with your customers about cut off dates for orders to be purchased, shipped, and delivered well in time for the holidays. Make it easier to find by using direct language like “Shipping Deadlines” or “Shipping Cutoff” rather than polite euphemisms like “Shipping Info” and “Shipping Details”.

Freeze your code

The holiday season is not the time to be making big changes to your code base. Consider freezing your code in the next few days while the full development team are still available and monitor the environment closely to ensure likely problems are identified and resolved before the team takes off for a well-earned break.

Triage and simplify

If you’re a retailer, social sites or sports media publisher and you’re expecting a higher than usual workload over the holidays, consider simplifying to reduce load, by temporarily switching off non-essential resource-intensive code, data or content.

For example, adjusting a retailer catalog for a few peak days could mean a smaller product catalogue, which would be served more quickly and might reduce server load. Removing or disabling resource intensive features/functions, can ensure system resources are dedicated to core functionality.

Check and double-check your monitoring

If a server fails in the forest and nobody hears about it, did it ever really fail? Most definitely yes, but if you don’t hear about it, you can’t take action and while it’s down it’s doing harm to your business. Its safe to assume your team will be making fewer manual checks of the system over the holidays, so now is the time to test and re-test your automated monitoring to make sure that it trips for the right events, and that notifications will be delivered to team members who understand they’ll be responsible for responding over the holidays.

It may also be worth asking us to investigate whether the monitoring tools you are using are negatively impacting site performance.


Backup or be sleighed

We wouldn’t let you get by without any backup strategy at all, but now is a good time to ensure that your backup routine will be appropriate for the holidays, that key staff know where and how to begin rollback and recovery, and that you’re confident you can get it done quickly with your available holiday team members. If in doubt, give the Anchor team a call.

Test or hit Santa’s naughty list

Test early and often to make sure there’s enough capacity available while you’re holidaying, and that loads are balanced correctly. 
Make sure predicted traffic levels are reviewed by all your internal stakeholders and confirmed as realistic.


Check everything twice, and check from front to back — connectivity, to firewall, to application and web servers. Check more than twice to catch that Murphy’s Law instance if you can.

Wishing you a safe, happy and 100% uptime this holidays!

0
Comments

US POP: Data Centre Facility Selection Process Complete

Published November 14th, 2011 by Keiran Holloway

After our usual negotiation process, it is with great pleasure to announce the Anchor has decided to partner with Equinix for data centre services in North America.

With over 98 data centres world-wide, Equinix are probably one of the most mature data centre vendors on the planet.

Given the extra advantage of having a presence here in Sydney where they’ve recently opened up their new facility – SY3 it was an easy selection as it also gave us direct access to Australian sales saff, based on local time zones.

The specific facility which we will be operating out of is their LA3 facility which is based within 3 miles of LAX, which makes it very accessible in the event that we need to physically attend the site.

Initially we will be commencing with a single rack in the data centre and be fitting this out with our core infrastructure and a number of hosts which we will be able to offer our initial hosting services.

At this point in time it is expected that we will be able to commence our public beta services towards the end of November. For a limited time period, a small number of clients will be able to have services hosts in the US on Anchor managed infrastructure — in return, we ask that you use the bandwidth/hardware extensively and provide feedback on various aspects of the service.

If this sounds like something that you would be interested in, please do not hesitate to email beta@anchor.com.au

1
Comment

US POP: Vendor Selection

Published November 8th, 2011 by Keiran Holloway

So now we’ve made the decision to deploy hardware in the US we need to start making some of the practical solutions, such as:

1. Which facility provider should we be using? and;
2. Where should the data centre be physically located?

To make this decision we had a number of important requirements for each of the services we’d need to procure.

Data Centre Providers

  • The data centre must be carrier netural facility and rated as a tier3 or greater data centre as per uptime institute guidelines
  • Given we do not have any staff on that ground at this point we require good smart-hands which includes a team that will complete all of the initial deployment:
  • 1) Receiving the servers and network devices from the hardware vendor and verify received as ordered
    2) Install kit into racks and record location for our internal documentation
    3) Cable up the machine to both power and networking. Carefully following cabling diagrams prepared by Anchor and supplied to the technician.
    4) Take care of the rubbish removal from the facility and disposal
    5) Be available 24×7 for emergency response to failed servers/hardware
    6) Provide a realistic service level agreement for these services so we can reliability predict mean time to repair after hardware failure.
    7) Be capable enough to get the initial equipment to the point where we could access them remotely to bootstrap the environment.

  • Facility Location was important to us as well. Do we want somewhere on the West coast which is closest to Australia, resulting in the lowest level of latency and is much easier to visit in the event we wish to go to the facility in person? Or somewhere on the East coast, which positions us better on a Global scale but has longer latency and is less accessible? How much would the price vary from location to location. There’s an awful lot of competition on the West coast of America — so perhaps that would mean prices would be more competitive?
  • Network Suppliers

    The beauty of doing this entire “Internet thing” for a while is that we already have reasonable amount of experience when it comes to negotiating bandwidth contracts with telcos and other IP transit suppliers. We also have a pretty good idea on how we want to structure our connectivitity.

    We also essentially need to deploy two networks:

    1. Our public facing network connectivity which would be using need:

  • To be fully multi-homed. Ie, we never allow ourselves to purchase bandwidth from one single supplier or companies which share common network components upstream. The is absolutely necessary to avoid any single point of failure.
  • Allow us to receive a full BGP feed and allow us to dictate how our traffic is routed. We don’t want to be relying on third parties to make changes to our network traffic.
  • Have a primary data link which was fast and had way more capacity than we would need from day 1. (At least 100Mbps)
  • Have a secondary link which has the ability to be rapidly increased (talking minutes versus hours for the upgrade).
  • 2. An out-of-band, management network. This network was going to be used to build up our infrastructure from day zero. When we say build up, we mean install operating systems, configure routers and get our primary, public facing network running. Once the environment has been bootstrapped we would be using this network for day to day management services and in the unlikely event that our primary, redundant network becomes unavailable give us a way in and diagnosing what specifically is going on. Some of the requirements for this link are totally opposite to the publiuc facing network:

  • The link only needs to have limited capacity. 10Mbps will be sufficient enough for our purposes.
  • This connection should be as simple as possible. No BGP routing, go through as few network devices as possible (no routers, just switches).
  • Must be totally independant of the Primary/Backup links. Geographic diversity from the other connections is a must.
  • Must be reliable
  • Hardware Vendors

    Historically we’ve used supermicro servers here at Anchor for all our dedicated server and virtual private server solutions. In more recent times we’ve been deploying Dell Hardware for various reasons. Some of these include improved performance, greater power efficiently but one of the biggest gains has actually been as result of the included DRACs (Dell Remote Access Cards), with these units we can get access to the machine consoles as if we are sitting in front of the physical machine. This means we’re able to do more and more work remotely without actually needing to be at the data centre in person. Obviously, when we’re deploying hardware on the other side of the globe this inclusion is absolutely imperative. With Dell’s Global presence it makes this decision very much a ‘no brainer’

    The power rails which we use in Australia are APC devices which come with remote reboot capabilities. This allows for machines to be powered off and on remotely. We have done a fairly considerable amount of development using the devices both to track power usage as well as integration in provisioning systems. On this basis, we would be continuing with these units.

    The final question is the switching infrastructure and misc items such as cables and rack cage nuts. For here the important thing was to find a supplier who was local, could delivery everything to the data centre and be vendors for HP (who we use for the our switching infrastructure) as well as the APC remote reboot devices.

    The hunt begins!

    0
    Comments

    Anchor to build a US Point of Presence

    Published November 2nd, 2011 by Keiran Holloway

    It’s not really any big secret that we’ve gradually been doing more and more work throughout North America. Nor should it really come as any surprise that we’ve been around doing this web hosting thing for quite a while now and are always looking at ways that we can improve our offerings to our valued customers.

    On this basis, we’ve recently made the decision to commence a build-out of a US-based facility to offer an alternative point of presence based on the west coast of America.

    What was the motivation behind this?

    Over the past 2 years we’ve had the absolute privilege and pleasure of building and delivering various complex hosting environments in facilities other than our primary point of presence in Sydney, Australia. There obvious ones include Github, Stocktwits, GroupMe, Huggies, Leadformance and AffinityLive. We’ve had an awful lot of enjoyment building and successfully delivering custom solutions under our
    remote management product. That said, however, there has been some challenges that we’ve come across and we are 100% confident that we can significantly improve our product offering by being in control of all aspects of service delivery.

    Some of the problems we encountered whilst using other hosting providers included:

    - Network outages during business hours whilst routing upgrades were being
    completed,
    - RAM upgrades and receiving RAM with errors,
    - Receiving replacement drives with previous client data still resident on the
    drives,
    - receiving machines which had been built many months previously, left with default passwords and then subsequently compromised
    - rapid deployment systems which actually don’t do either and;
    - best of all, intermittent networking problems which couldn’t be solved by the network provider and we could only work around through changing the MAC addresses on the physical devices. Seriously? WHHAA?

    For these reasons and various others, we’ve come to the conclusion:

    This isn’t good enough for Anchor – We’re building in the US.

    In this industry, reputation is everything and when you have high levels of dependence on suppliers then it can be difficult to be entirely in control of your own destiny.

    Another factor in the decision making process was that, to my knowledge, there is no other managed hosting provider in the Australian marketplace which is providing, high-quality, premium style managed services using data centers in America. With up to 20% of Australian websites hosted in places other than locally, there seems to be a really unique opportunity to be providing premium support to our Australian customer-base whilst leveraging some of the benefits of having the servers physical located in the states — for example, much larger data bandwidth allocations and closer proximity to the American audience.

    Over the new few weeks, we will be taking all our knowledge and skills that we’ve honed the past 10 years running a robust, reliable and premium hosting service. We’re then going to apply it to clean slate in a data centre facility on the other side of the globe.

    Sound like a challenge? You betcha!

    This is an very exciting time and upon completion we will be able offer high-volume bandwidth services with a US presence both to our existing to our Australian clients as well as extending our reach into the International market in the longer term. During this build up phase I am committed to providing a continous updates as to our detailed processes to give transparency into what goes into building up such an environment.

    0
    Comments

    Would you like my job?

    Published November 2nd, 2011 by Keiran Holloway

    No, really. We need a technical writer.

    Someone who can take our thousands of pages of documentation and improve on it. Someone who will be posting content, right here, right now.

    Interested? See http://www.anchor.com.au/about-us/jobs for a full job description.

    0
    Comments

    cPanel University?! You’re doing it all wrong!

    Published October 21st, 2011 by Keiran Holloway

    Today I was minding my own business at my desk when I stumbled upon university.cpanel.net, a site which allows you to obtain “industry certification” for the cPanel Web Hosting Manager.

    The first thing I did was check the date; It’s not April 1st.. So I sat there stunned for a minute or two, wondering if I should laugh or cry.

    Upon further inspection, it actually seemed to be true. You can now go and do an online course and become a certifed cPanel technician!

    For anyone who has done business with us in the past, we don’t make too much of a secret that we don’t think too much of control panels such as cpanel or plesk. In fact, we’ve quite openly published our thoughts on this in the past.

    That said, trying to think about this a little, I’ve got to ask myself the question — “If you’re building a web-based interface which is designed to allow end-users to control their web-hosting service, then surely expecting certification is doing it all wrong?

    Whilst digging further, the actual value of this certification is admittedly some what questionable:

    - The first level testing consists of a total of 18 questions, takes 15 minutes and you need to get 15 of the questions right.
    - You can continue to re-take exams if you fail
    - They can’t actually supply any technical theory or text books
    - The advanced levels of training require you to be proficient in perl — surely if you need to use a programming language to configure your “easy-to-use” control panel, you’ve pretty much missed the point.

    As we’ve discussed in the past — cpanel significantly and drastically reduces the barrier of entry to becoming a hosting provider. It allows people who would otherwise not be capable nor qualified to run a fully fledged hosting company and hide behind the pretty exterior of the cpanel user interface. This is scary. Why? some of the approaches and methods which are used by cpanel are considerably questionable.

    Some of the observations which we’ve made include:

    - Installing cpanel is like a unix security evolutionary throw back. A newly built machine had an extra 12 processes running as the root user.
    - The security history is so poor that it has a “Scan for Trojan Horses” dialog page.
    - There is no inbuilt firewall management utility, yet it is quite keen to change handcrafted firewall rules added by hand
    - MySQL is compiled without SSL support
    - The update dialog page has people have to chose between 4 different update sources — instead of just one which works.
    - http is run as the nobody user
    - It entirely ignores the Filesystem Hierarchy Standard and stores most files under /usr/local/x
    - If you want to add an SSL certificate for a subdomain that isn’t configured, when you paste the certificate file in, cpanel will successfully parse the cert, extract the correct CN, and map it to the correct user. But when you then paste the key and submit, it’ll bomb saying the CN doesn’t exist. If it doesn’t exist, how did you manage to find a user???
    - It actually comes with /scripts/fix_common_problems

    Having courses which explicitly train people up to this level and little further is, to my mind, a grave misgiving. It suggests that anyone can spend some coin on an online test and become sufficiently proficient enough to comprehensively run a entire web hosting company. Speaking as someone who has had 7 years experience in this industry, providing web hosting services is more complex than simply doing a handful of online tests and installing some random piece of software; doing it well requires the backing of a intelligent, experienced and knowledgeable team of system administrators. Thinking that any piece of software can replace this is not only naive, but a school of thought which potentially leaves the web-hosting industry, as a whole, to be brought into disrepute.

    Tags: ,
    Posted in WTF

     Leave a comment

    1
    Comment

    Happy International SUIT UP Day!

    Published October 13th, 2011 by Keiran Holloway

    For one day of the year, all the Anchorites put away the board shorts and flip-flops, to celebrate in style with Cheap Suits and Expensive Scotch!

    In honour of Barney Stinson from How I Met Your Mother, a whole bunch of sysadmins at Anchor went to great lengths to wear a suit to work today! (A certain employee was caught off-guard yesterday and had to purchase a new one)

    We know it’s real, because we saw it on the internet

    The Anchor team all schmicked up!

    Tags:
    Posted in FTW, WTF

     Leave a comment

    0
    Comments

    Anchor Presenting at DevOps Meetup – Thursday September 15th

    Published September 8th, 2011 by Keiran Holloway

    Just a short post to let you all know that Anchor’s David Basden and Chris Collins will be presenting their software which is being used in our on-going projects to automate everything relating to Anchor dedicated server builds.

    The software, which has been released as open source on github as was discussed in brief in the previously two blog posts The automation waltz and Automate all the things.

    The event will be held at 7pm, next Thursday, 15th September at the Orient in Sydney and will give you the fantastic opportunity to have a beer and chat with the authors of the software as well as like minded people with both developer and systemadmin backgrounds.

    Comprehensive details as well as registration for the meetup can be found at September DevOps Meetup

    Look forward to seeing you then!

    0
    Comments

    Devs and SysAdmins – It is possible to live harmoniously!

    Published August 17th, 2011 by Keiran Holloway

    Historically, the battle lines were drawn and everyone was bracing themselves, ready for combat..

    On one side, you have the Devs wanting to make a change to a production server immediately to get their new shiny feature working and the SysOps on the other side, desperate not to give an inch for fear of upsetting the uptime gods.

    Each party thinking to themselves: “does it really need to be this hard?” .. “Why doesn’t the other get it?”

    The Times They Are a-Changin’ and with it, a new fan-dangled word:

    DEVOPS

    wait! … wtf??! Devops??! Who?! What?

    Well.. Depending on who you ask, it means any number of things, my view is it comes down to a combination of attitude, cultural and process changes which need to be applied to both system administrators and developers alike. Forming the super-creature…. a “DevOp”. Through this combination it will allow people working on either aspect to operate in a smarter, more collaborated fashion and achieve outcomes which are best for business.

    At Anchor, obviously being a group of professional sysadmins who manage public-facing websites factors like reputation and uptime are everything; meaning that we feel these pressures essentially on a day to day basis. That said, we do have the pleasure of working with some of the best developers around and we’re awfully keen to embrace any changes which allow us to support these people better.

    With this in mind we routinely are actively involved in local DevOps meetups here in Sydney — coincidently there’s a meetup on tomorrow night, held at the orient in the historic rocks area of Sydney. With this in mind we would encourage anyone who is a dev, an op or a devop to drop in to one of these meetups and chat with like minded folk.

    On a slightly different, but still related note, Anchor’s Benjamin Smith will be presenting at the annual PyCon conference, being held in Sydney this weekend at 10:20am Sunday. The topic of the talk is: “Sysadmins vs Developers, a take from the other side of the fence”, whilst registration has closed for this conference, this talk will be recorded and presented on Blip.TV (URL will be provided once available) with the slides from the talk being released via our website.

    0
    Comments