2011 November | Anchor Web Hosting Blog

Dual-stack IPv6/IPv4 as standard on new US deployments

Published November 22nd, 2011 by Barney Desmond

The focus for this post is obviously about our IPv6 deployment plans, but I’d like to take a small detour through our US presence on the way there.

Anchor’s networking and automation gurus have been hard at work preparing our new kit over in the US, and the day we go live is fast approaching. In the process we’ve had literally zero personal presence over there, not one plane ticket was bought. That we can get away with this is mostly thanks to two things: Equinix and DRACs (Dell’s remote-management interface).

Equinix

One of the reasons we went with Equinix is their high level of support, which goes nicely with Anchor’s approach to business.

The servers were dropshipped direct to the LA3 datacentre where staff unpacked them and racked them up according to our instructions. Once the DRACs were plugged in it was our turn to get excited.

DRACs

DRACs let you do stuff remotely, pretty much anything short of implementing a big red self destruct button. Wanna boot an OS install image from 12,000km away? You can do that.

With a little bit of shuffling, we’ve bootstrapped a new environment over there, ready to build high-availability VMs and servers.

The core of Anchor's LAX1 presence, all gigabit and redundant up the wazoo

IPv6

Which brings us to the fun stuff. The question of offering IPv6 has been on the cards for four or five years now, but to date there hadn’t been enough of a business case for us to commit to. The LAX1 POP changes this.

Anchor’s initial offerings will be focused on our VPS product, which will have full dual-stack connectivity from day 0. For those customers with IPv6 aspirations, we’re ready for you. For everyone else, we hope its enabled-by-default status will drive some interest.

As a small mark of this commitment, we’ve given our DRACs IPv6 addresses only. The practical upshot of this is that it forces us to ensure that all our infrastructure supports IPv6 – routing, switching, DNS, the works. This is popularly called “eating your own dogfood”, we wouldn’t sell something that we don’t use ourselves.

Why IPv6?

For those who don’t follow the IPv4 to IPv6 transition, the rest of this post will summarise Anchor’s perspective specifically.

It’s generally agreed that we’re going to run out of the IPv4 addresses that we know and love, and quite soon. Estimates vary, but unless there’s a revolutionary change in our use of addresses it looks like we’ll hit the wall within about three years.

IPv6 support in the backbone of the internet is already well established, but the real challenge is pushing it all the way to end-users. IPv6 requires big changes through the entire technology stack: routers, switches, firewalls, DNS, servers, operating systems, application configurations and more, it’s all there.

This makes it extremely costly to retrofit existing systems. ISPs have a nasty chicken-and-egg problem because of that – given that demand will only be driven by availability, it’s much easier to do nothing instead of committing to big spending with uncertain returns.

Even assuming that an ISP wanted to sell IPv6 to their consumers, there’s very few content providers doing IPv6. Due to aforementioned lack of demand, content providers would be crazy to present IPv6-only content and lose the whole IPv4 market. Given this, it ultimately looks like the shift to IPv6 will be driven by “external” forces.

As an example, big companies like Google are in a position to push for adoption (try visiting http://ipv6.google.com/), and in theory even offer IPv6-only content that enough users would hassle their ISP for. On the other end of the hypothetical stick, if hosting providers increase the cost (technical and monetary) of IPv4 hosting, it could push enough content to IPv6 to grind through a transition.

These are just two examples, but hopefully they illustrate the difficulty and significance of the problem.

Tags: connectivity, drac, equinix, ipv6, networking, us pop
Posted in FTW

Leave a comment

Comments

The Internet is trying to kill you

Published November 21st, 2011 by David Basden

Here are my notes from a 5 minute lightning talk that I gave at the Open Source Developers Conference in Canberra last week. It went down pretty well, and it was a lot of fun giving it. It was targeted at web developers, and titled “The Internet is trying to kill you”.

Hi, my name is David. I work for a web hosting company called Anchor Systems as a sysadmin, and like all sysadmins I’ve been avoiding actual sysadmin work as much as possible, and have been writing some web apps.

So people writing web apps, Hey!

The internet hates you.

Now I’m just trying to get across one simple point, and that is that there are people on the internet who want to set your shiny new web app on fire just to see it burn. Maybe they want to do it for the lolz. Maybe they want attention. Maybe they want your data. Maybe they’re just plain bored. They want it all to end in fire.

And you’re next.

You don’t have to be big. You don’t have to be popular. Earlier this afternoon I spun up a clean VM on a free IP address that had never been used before, ever. Nothing even pointing to it in DNS. People were hitting port 80 and 443 in less than a minute. This is a fact of life: This is background noise.

I don’t want you to feel safe and comfortable when you’re writing web applications. I want you to be thinking like a card-carrying member of the tin-foil hat brigade. Because this is the way that you don’t end up on the front page of The Australian’s IT section.

Here are some things that you are hopefully already thinking about:

Forms

The data’s fine. I check it all in javascript!

It’s okay. I pass the metadata in an <input type=”hidden”> where it can’t be seen

Okay, if you’re thinking this, put down the laptop now.
Talk to the person next to you. Read a couple of books.

NEVER. EVER. trust the browser.

Don’t even assume that your HTTP requests are even coming from
web browsers. We learned that the hard way 20 years ago,

DoS attacks

Hahaha. Why do I have to care? Ping floods aren’t my problem

That SELECT you’re doing on your front page that takes half a second?
Try 10,000 connections a second. Try 100,000.

Don’t just think about your front page. Think all your pages. Think
pages that process form submissions. Think stuff that doesn’t cache
well. Start thinking about what happens when someone is trying to bring down your site just by looking at it or posting data to it.

How about SQL injection attacks? Noone’s going to bother, right?

Yes. Yes they are. They’ve automated it. They’re doing that stuff literally in their sleep. Then they’ve packaged it up and given it to a thousand of their friends that don’t even know what SQL is.

Check your data. Check it again. Get into the mindset that anything that comes from the outside world is malicious, and the malicious person is smarter than you are.

Okay, just quickly some other things:

Never assume that no-one is sniffing your network
Never assume the system you’re running on hasn’t been broken into
Never assume that people using your site are rational.In the time you can say “Why would they DO that?!”, they’re already doing it.

Always assume attackers are smarter than you are, have more resources, and much more experience than you do.

Be paranoid. Wear a tinfoil hat. Have fun.

Posted in FTW

Leave a comment

Comments

The Internet Show – Give Away Congratulations

Published November 9th, 2011 by bsmith

As some of you may have noticed, either from our previous posts or attending the event, Anchor was at The Internet Show . While it wasn’t to the same scale as the expo prior to this, the enthusiasm and calibre of the attendees was fantastic.

While at The Internet Show Anchor presented a talk regarding ‘Why hosting management is important to your online business‘.

For illustrative purposes only, we were dressed better.

It was a fantastic opportunity, for US, to present the facts and benefits face to face with the next crop of online success stories. For everyone who made the effort to sit and watch (and ask questions especially!), thank you. Opportunities to interact with the public in such a candid and personal manner are few and far between. So if you were ever racked by indecision regarding attending a Anchor talk, we not only encourage you to attend, but ask some hard questions while you’re there.

Besides the talk, we unintentionally provided much amusement to attendees via these two trouble makers:

Floating fish that swim, who'd a thunk it?

It was however, with deep regret that Nemo, easily the favourite at the event, decided the ceiling was a far nicer location than the expo floor. At present we are unsure as to his fate, but we have his remote and his left fin to keep us company..

For those of you out there still confused about these things click here.

As a thank you for everyone who stopped by our booth, we also ran a prize draw. This time round, rather than an iPad2, we went for the latest craze. Not only did we update the prize option, we made it an option at the same time!

First prize: iPhone 4S

Shiny.

Second prize: A Harbour Cruise for two including dinner!

First place winner was Adam Stead from Reach Local! The timing was impeccable for Adam, as his trusty iPhone 3GS had been wounded in the course of duty (i.e. the screen was cracked).

The second place, Harbour Cruise, was won by Ben Davey from Mobile Nation! (We were assured that he isn’t prone to sea sickness).

A big thank you to everyone who stopped by and had a chat. We hope to see you all at the next ‘The Internet Show’ event.

Tags: apple, cruise, internet, internet show, iphone, iphone 4s, winning
Posted in FTW

Leave a comment

Comment

US POP: Vendor Selection

Published November 8th, 2011 by Keiran Holloway

So now we’ve made the decision to deploy hardware in the US we need to start making some of the practical solutions, such as:

1. Which facility provider should we be using? and;
2. Where should the data centre be physically located?

To make this decision we had a number of important requirements for each of the services we’d need to procure.

Data Centre Providers

The data centre must be carrier netural facility and rated as a tier3 or greater data centre as per uptime institute guidelines

Given we do not have any staff on that ground at this point we require good smart-hands which includes a team that will complete all of the initial deployment:

1) Receiving the servers and network devices from the hardware vendor and verify received as ordered
2) Install kit into racks and record location for our internal documentation
3) Cable up the machine to both power and networking. Carefully following cabling diagrams prepared by Anchor and supplied to the technician.
4) Take care of the rubbish removal from the facility and disposal
5) Be available 24×7 for emergency response to failed servers/hardware
6) Provide a realistic service level agreement for these services so we can reliability predict mean time to repair after hardware failure.
7) Be capable enough to get the initial equipment to the point where we could access them remotely to bootstrap the environment.

Facility Location was important to us as well. Do we want somewhere on the West coast which is closest to Australia, resulting in the lowest level of latency and is much easier to visit in the event we wish to go to the facility in person? Or somewhere on the East coast, which positions us better on a Global scale but has longer latency and is less accessible? How much would the price vary from location to location. There’s an awful lot of competition on the West coast of America — so perhaps that would mean prices would be more competitive?

Network Suppliers

The beauty of doing this entire “Internet thing” for a while is that we already have reasonable amount of experience when it comes to negotiating bandwidth contracts with telcos and other IP transit suppliers. We also have a pretty good idea on how we want to structure our connectivitity.

We also essentially need to deploy two networks:

1. Our public facing network connectivity which would be using need:

To be fully multi-homed. Ie, we never allow ourselves to purchase bandwidth from one single supplier or companies which share common network components upstream. The is absolutely necessary to avoid any single point of failure.

Allow us to receive a full BGP feed and allow us to dictate how our traffic is routed. We don’t want to be relying on third parties to make changes to our network traffic.

Have a primary data link which was fast and had way more capacity than we would need from day 1. (At least 100Mbps)

Have a secondary link which has the ability to be rapidly increased (talking minutes versus hours for the upgrade).

2. An out-of-band, management network. This network was going to be used to build up our infrastructure from day zero. When we say build up, we mean install operating systems, configure routers and get our primary, public facing network running. Once the environment has been bootstrapped we would be using this network for day to day management services and in the unlikely event that our primary, redundant network becomes unavailable give us a way in and diagnosing what specifically is going on. Some of the requirements for this link are totally opposite to the publiuc facing network:

The link only needs to have limited capacity. 10Mbps will be sufficient enough for our purposes.

This connection should be as simple as possible. No BGP routing, go through as few network devices as possible (no routers, just switches).

Must be totally independant of the Primary/Backup links. Geographic diversity from the other connections is a must.

Must be reliable

Hardware Vendors

Historically we’ve used supermicro servers here at Anchor for all our dedicated server and virtual private server solutions. In more recent times we’ve been deploying Dell Hardware for various reasons. Some of these include improved performance, greater power efficiently but one of the biggest gains has actually been as result of the included DRACs (Dell Remote Access Cards), with these units we can get access to the machine consoles as if we are sitting in front of the physical machine. This means we’re able to do more and more work remotely without actually needing to be at the data centre in person. Obviously, when we’re deploying hardware on the other side of the globe this inclusion is absolutely imperative. With Dell’s Global presence it makes this decision very much a ‘no brainer’

The power rails which we use in Australia are APC devices which come with remote reboot capabilities. This allows for machines to be powered off and on remotely. We have done a fairly considerable amount of development using the devices both to track power usage as well as integration in provisioning systems. On this basis, we would be continuing with these units.

The final question is the switching infrastructure and misc items such as cables and rack cage nuts. For here the important thing was to find a supplier who was local, could delivery everything to the data centre and be vendors for HP (who we use for the our switching infrastructure) as well as the APC remote reboot devices.

The hunt begins!

Tags: dell, hardware vendors, HP, network, project USPOP
Posted in FTW, Newsletter

Leave a comment

Comments

Shining some sun on the cloud

Published November 5th, 2011 by matt

Being the cynical, hard-bitten sysadmins that we are, we’re a bit skeptical about some of the more grandiose claims about cloud computing: 100% uptime, never having to worry about scalability, and all those other things that people who don’t understand reality seem to get terribly excited about.

It’s good to see every now and then that someone else has an experience that matches our own, such as Mixpanel’s decision to move off Rackspace’s cloud and onto dedicated servers. I’d love to know how to negotiate 50%-75% off a vendor’s list price, though…

Posted in WTF

Leave a comment

Comments

Anchor to build a US Point of Presence

Published November 2nd, 2011 by Keiran Holloway

It’s not really any big secret that we’ve gradually been doing more and more work throughout North America. Nor should it really come as any surprise that we’ve been around doing this web hosting thing for quite a while now and are always looking at ways that we can improve our offerings to our valued customers.

On this basis, we’ve recently made the decision to commence a build-out of a US-based facility to offer an alternative point of presence based on the west coast of America.

What was the motivation behind this?

Over the past 2 years we’ve had the absolute privilege and pleasure of building and delivering various complex hosting environments in facilities other than our primary point of presence in Sydney, Australia. There obvious ones include Github, Stocktwits, GroupMe, Huggies, Leadformance and AffinityLive. We’ve had an awful lot of enjoyment building and successfully delivering custom solutions under our
remote management product. That said, however, there has been some challenges that we’ve come across and we are 100% confident that we can significantly improve our product offering by being in control of all aspects of service delivery.

Some of the problems we encountered whilst using other hosting providers included:

- Network outages during business hours whilst routing upgrades were being
completed,
- RAM upgrades and receiving RAM with errors,
- Receiving replacement drives with previous client data still resident on the
drives,
- receiving machines which had been built many months previously, left with default passwords and then subsequently compromised
- rapid deployment systems which actually don’t do either and;
- best of all, intermittent networking problems which couldn’t be solved by the network provider and we could only work around through changing the MAC addresses on the physical devices. Seriously? WHHAA?

For these reasons and various others, we’ve come to the conclusion:

This isn’t good enough for Anchor – We’re building in the US.

In this industry, reputation is everything and when you have high levels of dependence on suppliers then it can be difficult to be entirely in control of your own destiny.

Another factor in the decision making process was that, to my knowledge, there is no other managed hosting provider in the Australian marketplace which is providing, high-quality, premium style managed services using data centers in America. With up to 20% of Australian websites hosted in places other than locally, there seems to be a really unique opportunity to be providing premium support to our Australian customer-base whilst leveraging some of the benefits of having the servers physical located in the states — for example, much larger data bandwidth allocations and closer proximity to the American audience.

Over the new few weeks, we will be taking all our knowledge and skills that we’ve honed the past 10 years running a robust, reliable and premium hosting service. We’re then going to apply it to clean slate in a data centre facility on the other side of the globe.

Sound like a challenge? You betcha!

This is an very exciting time and upon completion we will be able offer high-volume bandwidth services with a US presence both to our existing to our Australian clients as well as extending our reach into the International market in the longer term. During this build up phase I am committed to providing a continous updates as to our detailed processes to give transparency into what goes into building up such an environment.

Tags: dedicated servers, high-bandwidth, project USPOP, reliability, virtual private server, vps
Posted in FTW, Newsletter

Leave a comment

Comments

Blog Posts from November, 2011

Dual-stack IPv6/IPv4 as standard on new US deployments

Equinix

DRACs