Anchor Web Hosting Blog

The Internet is trying to kill you

Published November 21st, 2011 by David Basden

Here are my notes from a 5 minute lightning talk that I gave at the Open Source Developers Conference in Canberra last week. It went down pretty well, and it was a lot of fun giving it. It was targeted at web developers, and titled “The Internet is trying to kill you”.

Hi, my name is David. I work for a web hosting company called Anchor Systems as a sysadmin, and like all sysadmins I’ve been avoiding actual sysadmin work as much as possible, and have been writing some web apps.

So people writing web apps, Hey!

The internet hates you.

Now I’m just trying to get across one simple point, and that is that there are people on the internet who want to set your shiny new web app on fire just to see it burn. Maybe they want to do it for the lolz. Maybe they want attention. Maybe they want your data. Maybe they’re just plain bored. They want it all to end in fire.

And you’re next.

You don’t have to be big. You don’t have to be popular. Earlier this afternoon I spun up a clean VM on a free IP address that had never been used before, ever. Nothing even pointing to it in DNS. People were hitting port 80 and 443 in less than a minute. This is a fact of life: This is background noise.

I don’t want you to feel safe and comfortable when you’re writing web applications. I want you to be thinking like a card-carrying member of the tin-foil hat brigade. Because this is the way that you don’t end up on the front page of The Australian’s IT section.

Here are some things that you are hopefully already thinking about:

Forms

The data’s fine. I check it all in javascript!

It’s okay. I pass the metadata in an <input type=”hidden”> where it can’t be seen

Okay, if you’re thinking this, put down the laptop now.
Talk to the person next to you. Read a couple of books.

NEVER. EVER. trust the browser.

Don’t even assume that your HTTP requests are even coming from
web browsers. We learned that the hard way 20 years ago,

DoS attacks

Hahaha. Why do I have to care? Ping floods aren’t my problem

That SELECT you’re doing on your front page that takes half a second?
Try 10,000 connections a second. Try 100,000.

Don’t just think about your front page. Think all your pages. Think
pages that process form submissions. Think stuff that doesn’t cache
well. Start thinking about what happens when someone is trying to bring down your site just by looking at it or posting data to it.

How about SQL injection attacks? Noone’s going to bother, right?

Yes. Yes they are. They’ve automated it. They’re doing that stuff literally in their sleep. Then they’ve packaged it up and given it to a thousand of their friends that don’t even know what SQL is.

Check your data. Check it again. Get into the mindset that anything that comes from the outside world is malicious, and the malicious person is smarter than you are.

Okay, just quickly some other things:

Never assume that no-one is sniffing your network
Never assume the system you’re running on hasn’t been broken into
Never assume that people using your site are rational.In the time you can say “Why would they DO that?!”, they’re already doing it.

Always assume attackers are smarter than you are, have more resources, and much more experience than you do.

Be paranoid. Wear a tinfoil hat. Have fun.

Posted in FTW

Leave a comment

Comments

The Internet Show – Give Away Congratulations

Published November 9th, 2011 by bsmith

As some of you may have noticed, either from our previous posts or attending the event, Anchor was at The Internet Show . While it wasn’t to the same scale as the expo prior to this, the enthusiasm and calibre of the attendees was fantastic.

While at The Internet Show Anchor presented a talk regarding ‘Why hosting management is important to your online business‘.

For illustrative purposes only, we were dressed better.

It was a fantastic opportunity, for US, to present the facts and benefits face to face with the next crop of online success stories. For everyone who made the effort to sit and watch (and ask questions especially!), thank you. Opportunities to interact with the public in such a candid and personal manner are few and far between. So if you were ever racked by indecision regarding attending a Anchor talk, we not only encourage you to attend, but ask some hard questions while you’re there.

Besides the talk, we unintentionally provided much amusement to attendees via these two trouble makers:

Floating fish that swim, who'd a thunk it?

It was however, with deep regret that Nemo, easily the favourite at the event, decided the ceiling was a far nicer location than the expo floor. At present we are unsure as to his fate, but we have his remote and his left fin to keep us company..

For those of you out there still confused about these things click here.

As a thank you for everyone who stopped by our booth, we also ran a prize draw. This time round, rather than an iPad2, we went for the latest craze. Not only did we update the prize option, we made it an option at the same time!

First prize: iPhone 4S

Shiny.

Second prize: A Harbour Cruise for two including dinner!

First place winner was Adam Stead from Reach Local! The timing was impeccable for Adam, as his trusty iPhone 3GS had been wounded in the course of duty (i.e. the screen was cracked).

The second place, Harbour Cruise, was won by Ben Davey from Mobile Nation! (We were assured that he isn’t prone to sea sickness).

A big thank you to everyone who stopped by and had a chat. We hope to see you all at the next ‘The Internet Show’ event.

Tags: apple, cruise, internet, internet show, iphone, iphone 4s, winning
Posted in FTW

Leave a comment

Comment

US POP: Vendor Selection

Published November 8th, 2011 by Keiran Holloway

So now we’ve made the decision to deploy hardware in the US we need to start making some of the practical solutions, such as:

1. Which facility provider should we be using? and;
2. Where should the data centre be physically located?

To make this decision we had a number of important requirements for each of the services we’d need to procure.

Data Centre Providers

The data centre must be carrier netural facility and rated as a tier3 or greater data centre as per uptime institute guidelines

Given we do not have any staff on that ground at this point we require good smart-hands which includes a team that will complete all of the initial deployment:

1) Receiving the servers and network devices from the hardware vendor and verify received as ordered
2) Install kit into racks and record location for our internal documentation
3) Cable up the machine to both power and networking. Carefully following cabling diagrams prepared by Anchor and supplied to the technician.
4) Take care of the rubbish removal from the facility and disposal
5) Be available 24×7 for emergency response to failed servers/hardware
6) Provide a realistic service level agreement for these services so we can reliability predict mean time to repair after hardware failure.
7) Be capable enough to get the initial equipment to the point where we could access them remotely to bootstrap the environment.

Facility Location was important to us as well. Do we want somewhere on the West coast which is closest to Australia, resulting in the lowest level of latency and is much easier to visit in the event we wish to go to the facility in person? Or somewhere on the East coast, which positions us better on a Global scale but has longer latency and is less accessible? How much would the price vary from location to location. There’s an awful lot of competition on the West coast of America — so perhaps that would mean prices would be more competitive?

Network Suppliers

The beauty of doing this entire “Internet thing” for a while is that we already have reasonable amount of experience when it comes to negotiating bandwidth contracts with telcos and other IP transit suppliers. We also have a pretty good idea on how we want to structure our connectivitity.

We also essentially need to deploy two networks:

1. Our public facing network connectivity which would be using need:

To be fully multi-homed. Ie, we never allow ourselves to purchase bandwidth from one single supplier or companies which share common network components upstream. The is absolutely necessary to avoid any single point of failure.

Allow us to receive a full BGP feed and allow us to dictate how our traffic is routed. We don’t want to be relying on third parties to make changes to our network traffic.

Have a primary data link which was fast and had way more capacity than we would need from day 1. (At least 100Mbps)

Have a secondary link which has the ability to be rapidly increased (talking minutes versus hours for the upgrade).

2. An out-of-band, management network. This network was going to be used to build up our infrastructure from day zero. When we say build up, we mean install operating systems, configure routers and get our primary, public facing network running. Once the environment has been bootstrapped we would be using this network for day to day management services and in the unlikely event that our primary, redundant network becomes unavailable give us a way in and diagnosing what specifically is going on. Some of the requirements for this link are totally opposite to the publiuc facing network:

The link only needs to have limited capacity. 10Mbps will be sufficient enough for our purposes.

This connection should be as simple as possible. No BGP routing, go through as few network devices as possible (no routers, just switches).

Must be totally independant of the Primary/Backup links. Geographic diversity from the other connections is a must.

Must be reliable

Hardware Vendors

Historically we’ve used supermicro servers here at Anchor for all our dedicated server and virtual private server solutions. In more recent times we’ve been deploying Dell Hardware for various reasons. Some of these include improved performance, greater power efficiently but one of the biggest gains has actually been as result of the included DRACs (Dell Remote Access Cards), with these units we can get access to the machine consoles as if we are sitting in front of the physical machine. This means we’re able to do more and more work remotely without actually needing to be at the data centre in person. Obviously, when we’re deploying hardware on the other side of the globe this inclusion is absolutely imperative. With Dell’s Global presence it makes this decision very much a ‘no brainer’

The power rails which we use in Australia are APC devices which come with remote reboot capabilities. This allows for machines to be powered off and on remotely. We have done a fairly considerable amount of development using the devices both to track power usage as well as integration in provisioning systems. On this basis, we would be continuing with these units.

The final question is the switching infrastructure and misc items such as cables and rack cage nuts. For here the important thing was to find a supplier who was local, could delivery everything to the data centre and be vendors for HP (who we use for the our switching infrastructure) as well as the APC remote reboot devices.

The hunt begins!

Tags: dell, hardware vendors, HP, network, project USPOP
Posted in FTW, Newsletter

Leave a comment

Comments

Shining some sun on the cloud

Published November 5th, 2011 by matt

Being the cynical, hard-bitten sysadmins that we are, we’re a bit skeptical about some of the more grandiose claims about cloud computing: 100% uptime, never having to worry about scalability, and all those other things that people who don’t understand reality seem to get terribly excited about.

It’s good to see every now and then that someone else has an experience that matches our own, such as Mixpanel’s decision to move off Rackspace’s cloud and onto dedicated servers. I’d love to know how to negotiate 50%-75% off a vendor’s list price, though…

Posted in WTF

Leave a comment

Comments

Anchor to build a US Point of Presence

Published November 2nd, 2011 by Keiran Holloway

It’s not really any big secret that we’ve gradually been doing more and more work throughout North America. Nor should it really come as any surprise that we’ve been around doing this web hosting thing for quite a while now and are always looking at ways that we can improve our offerings to our valued customers.

On this basis, we’ve recently made the decision to commence a build-out of a US-based facility to offer an alternative point of presence based on the west coast of America.

What was the motivation behind this?

Over the past 2 years we’ve had the absolute privilege and pleasure of building and delivering various complex hosting environments in facilities other than our primary point of presence in Sydney, Australia. There obvious ones include Github, Stocktwits, GroupMe, Huggies, Leadformance and AffinityLive. We’ve had an awful lot of enjoyment building and successfully delivering custom solutions under our
remote management product. That said, however, there has been some challenges that we’ve come across and we are 100% confident that we can significantly improve our product offering by being in control of all aspects of service delivery.

Some of the problems we encountered whilst using other hosting providers included:

- Network outages during business hours whilst routing upgrades were being
completed,
- RAM upgrades and receiving RAM with errors,
- Receiving replacement drives with previous client data still resident on the
drives,
- receiving machines which had been built many months previously, left with default passwords and then subsequently compromised
- rapid deployment systems which actually don’t do either and;
- best of all, intermittent networking problems which couldn’t be solved by the network provider and we could only work around through changing the MAC addresses on the physical devices. Seriously? WHHAA?

For these reasons and various others, we’ve come to the conclusion:

This isn’t good enough for Anchor – We’re building in the US.

In this industry, reputation is everything and when you have high levels of dependence on suppliers then it can be difficult to be entirely in control of your own destiny.

Another factor in the decision making process was that, to my knowledge, there is no other managed hosting provider in the Australian marketplace which is providing, high-quality, premium style managed services using data centers in America. With up to 20% of Australian websites hosted in places other than locally, there seems to be a really unique opportunity to be providing premium support to our Australian customer-base whilst leveraging some of the benefits of having the servers physical located in the states — for example, much larger data bandwidth allocations and closer proximity to the American audience.

Over the new few weeks, we will be taking all our knowledge and skills that we’ve honed the past 10 years running a robust, reliable and premium hosting service. We’re then going to apply it to clean slate in a data centre facility on the other side of the globe.

Sound like a challenge? You betcha!

This is an very exciting time and upon completion we will be able offer high-volume bandwidth services with a US presence both to our existing to our Australian clients as well as extending our reach into the International market in the longer term. During this build up phase I am committed to providing a continous updates as to our detailed processes to give transparency into what goes into building up such an environment.

Tags: dedicated servers, high-bandwidth, project USPOP, reliability, virtual private server, vps
Posted in FTW, Newsletter

Leave a comment

Comments

Why you should use LVM: part 1319755409 in an infinite series

Published October 28th, 2011 by matt

At anchor, we loves us some LVM. It makes managing storage a breeze.

Now, we’ve got one more reason to love it. Because now we have lvmsync.

Like most modern hosting companies, we run a lot of VPSes, and sometimes the chunk of storage they’re on isn’t where it needs to be, so we have to transfer it around. Ordinarily, this would involve a lengthy period of downtime while dd did it’s business and sent the whole LV across a network.

NOT ANY MORE! Now, with the magic of lvmsync we can transfer the bulk of the data while the VM is running, and then do a quick transfer of just the changes after we shut the VM down.

We think our customers will appreciate the reduction in downtime, and I know we’ll appreciate the wonders of LVM just that little bit more.

Tags: lvm, lvmsync, vps
Posted in FTW

Leave a comment

Comments

The Internet Show – Why hosting management is important to your online business

Published October 27th, 2011 by Phillip Pace

The Internet Show is upon us this Thursday/Friday (27-28 October) and as Anchor has a presence, we are busy preparing for it. I’m happy to announce that we will be presenting a talk at the event about “why hosting management is imperative to the success of your website”. Some of the questions we’ll be asking (and answering) are:

What is a system administrator?
Do the marketing promises surrounding the cloud make sense?
Why are system administrators forgotten?
What specifically can a team of specialist systems administrators add to your hosting?

The topic of conversation will center around the fundamentals that make up Anchor, and the approaches that we apply every single day to our hosting business. We will cover the need to have some of the most technically adept systems administratators in the country working to keep your online presence online and available.

We are also exhibiting at the event, so the usual prize giveaways will be on offer at our booth. Drop by and enter the draw to have a chance of winning a new iPhone 4s or romantic dinner cruise for two!

If you have any trouble finding us at the event, simply look up, and you may see a remote controlled clown fish swimming around our booth. Failing that, if you happen to bump into a wandering pirate, he will be able to point you in the right direction.

See you there!

Posted in FTW

Leave a comment

Comment

cPanel University?! You’re doing it all wrong!

Published October 21st, 2011 by Keiran Holloway

Today I was minding my own business at my desk when I stumbled upon university.cpanel.net, a site which allows you to obtain “industry certification” for the cPanel Web Hosting Manager.

The first thing I did was check the date; It’s not April 1st.. So I sat there stunned for a minute or two, wondering if I should laugh or cry.

Upon further inspection, it actually seemed to be true. You can now go and do an online course and become a certifed cPanel technician!

For anyone who has done business with us in the past, we don’t make too much of a secret that we don’t think too much of control panels such as cpanel or plesk. In fact, we’ve quite openly published our thoughts on this in the past.

That said, trying to think about this a little, I’ve got to ask myself the question — “If you’re building a web-based interface which is designed to allow end-users to control their web-hosting service, then surely expecting certification is doing it all wrong?”

Whilst digging further, the actual value of this certification is admittedly some what questionable:

- The first level testing consists of a total of 18 questions, takes 15 minutes and you need to get 15 of the questions right.
- You can continue to re-take exams if you fail
- They can’t actually supply any technical theory or text books
- The advanced levels of training require you to be proficient in perl — surely if you need to use a programming language to configure your “easy-to-use” control panel, you’ve pretty much missed the point.

As we’ve discussed in the past — cpanel significantly and drastically reduces the barrier of entry to becoming a hosting provider. It allows people who would otherwise not be capable nor qualified to run a fully fledged hosting company and hide behind the pretty exterior of the cpanel user interface. This is scary. Why? some of the approaches and methods which are used by cpanel are considerably questionable.

Some of the observations which we’ve made include:

- Installing cpanel is like a unix security evolutionary throw back. A newly built machine had an extra 12 processes running as the root user.
- The security history is so poor that it has a “Scan for Trojan Horses” dialog page.
- There is no inbuilt firewall management utility, yet it is quite keen to change handcrafted firewall rules added by hand
- MySQL is compiled without SSL support
- The update dialog page has people have to chose between 4 different update sources — instead of just one which works.
- http is run as the nobody user
- It entirely ignores the Filesystem Hierarchy Standard and stores most files under /usr/local/x
- If you want to add an SSL certificate for a subdomain that isn’t configured, when you paste the certificate file in, cpanel will successfully parse the cert, extract the correct CN, and map it to the correct user. But when you then paste the key and submit, it’ll bomb saying the CN doesn’t exist. If it doesn’t exist, how did you manage to find a user???
- It actually comes with /scripts/fix_common_problems

Having courses which explicitly train people up to this level and little further is, to my mind, a grave misgiving. It suggests that anyone can spend some coin on an online test and become sufficiently proficient enough to comprehensively run a entire web hosting company. Speaking as someone who has had 7 years experience in this industry, providing web hosting services is more complex than simply doing a handful of online tests and installing some random piece of software; doing it well requires the backing of a intelligent, experienced and knowledgeable team of system administrators. Thinking that any piece of software can replace this is not only naive, but a school of thought which potentially leaves the web-hosting industry, as a whole, to be brought into disrepute.

Tags: cpanel, web hosting
Posted in WTF

Leave a comment

Comment

Blog

The Internet is trying to kill you

Forms

DoS attacks

The Internet Show – Give Away Congratulations

Shining some sun on the cloud

Why you should use LVM: part 1319755409 in an infinite series

The Internet Show – Why hosting management is important to your online business

cPanel University?! You’re doing it all wrong!

Archives

Blogroll

Recent Posts

Meta