US POP: Vendor Selection

Published November 8th, 2011 by Keiran Holloway

So now we’ve made the decision to deploy hardware in the US we need to start making some of the practical solutions, such as:

1. Which facility provider should we be using? and;
2. Where should the data centre be physically located?

To make this decision we had a number of important requirements for each of the services we’d need to procure.

Data Centre Providers

  • The data centre must be carrier netural facility and rated as a tier3 or greater data centre as per uptime institute guidelines
  • Given we do not have any staff on that ground at this point we require good smart-hands which includes a team that will complete all of the initial deployment:
  • 1) Receiving the servers and network devices from the hardware vendor and verify received as ordered
    2) Install kit into racks and record location for our internal documentation
    3) Cable up the machine to both power and networking. Carefully following cabling diagrams prepared by Anchor and supplied to the technician.
    4) Take care of the rubbish removal from the facility and disposal
    5) Be available 24×7 for emergency response to failed servers/hardware
    6) Provide a realistic service level agreement for these services so we can reliability predict mean time to repair after hardware failure.
    7) Be capable enough to get the initial equipment to the point where we could access them remotely to bootstrap the environment.

  • Facility Location was important to us as well. Do we want somewhere on the West coast which is closest to Australia, resulting in the lowest level of latency and is much easier to visit in the event we wish to go to the facility in person? Or somewhere on the East coast, which positions us better on a Global scale but has longer latency and is less accessible? How much would the price vary from location to location. There’s an awful lot of competition on the West coast of America — so perhaps that would mean prices would be more competitive?
  • Network Suppliers

    The beauty of doing this entire “Internet thing” for a while is that we already have reasonable amount of experience when it comes to negotiating bandwidth contracts with telcos and other IP transit suppliers. We also have a pretty good idea on how we want to structure our connectivitity.

    We also essentially need to deploy two networks:

    1. Our public facing network connectivity which would be using need:

  • To be fully multi-homed. Ie, we never allow ourselves to purchase bandwidth from one single supplier or companies which share common network components upstream. The is absolutely necessary to avoid any single point of failure.
  • Allow us to receive a full BGP feed and allow us to dictate how our traffic is routed. We don’t want to be relying on third parties to make changes to our network traffic.
  • Have a primary data link which was fast and had way more capacity than we would need from day 1. (At least 100Mbps)
  • Have a secondary link which has the ability to be rapidly increased (talking minutes versus hours for the upgrade).
  • 2. An out-of-band, management network. This network was going to be used to build up our infrastructure from day zero. When we say build up, we mean install operating systems, configure routers and get our primary, public facing network running. Once the environment has been bootstrapped we would be using this network for day to day management services and in the unlikely event that our primary, redundant network becomes unavailable give us a way in and diagnosing what specifically is going on. Some of the requirements for this link are totally opposite to the publiuc facing network:

  • The link only needs to have limited capacity. 10Mbps will be sufficient enough for our purposes.
  • This connection should be as simple as possible. No BGP routing, go through as few network devices as possible (no routers, just switches).
  • Must be totally independant of the Primary/Backup links. Geographic diversity from the other connections is a must.
  • Must be reliable
  • Hardware Vendors

    Historically we’ve used supermicro servers here at Anchor for all our dedicated server and virtual private server solutions. In more recent times we’ve been deploying Dell Hardware for various reasons. Some of these include improved performance, greater power efficiently but one of the biggest gains has actually been as result of the included DRACs (Dell Remote Access Cards), with these units we can get access to the machine consoles as if we are sitting in front of the physical machine. This means we’re able to do more and more work remotely without actually needing to be at the data centre in person. Obviously, when we’re deploying hardware on the other side of the globe this inclusion is absolutely imperative. With Dell’s Global presence it makes this decision very much a ‘no brainer’

    The power rails which we use in Australia are APC devices which come with remote reboot capabilities. This allows for machines to be powered off and on remotely. We have done a fairly considerable amount of development using the devices both to track power usage as well as integration in provisioning systems. On this basis, we would be continuing with these units.

    The final question is the switching infrastructure and misc items such as cables and rack cage nuts. For here the important thing was to find a supplier who was local, could delivery everything to the data centre and be vendors for HP (who we use for the our switching infrastructure) as well as the APC remote reboot devices.

    The hunt begins!

    0
    Comments

    Evil hack to make arrow and SysReq keys work with a Dell iDRAC KVM and Linux desktop

    Published March 18th, 2011 by egalanos

    So you’re trying to use the arrow keys in a remote server console with the iDRAC KVM. The keys not working and it’s driving you mad? Or perhaps that mission critical server is experiencing pain and you need to make use of the magic sysrq key to debug it?

    Unfortunately, for a long long time, this has not worked if you are using Linux on your desktop.

    Making it work

    Just follow the steps below and you should be good to go.

    1. Download idrac.tar and unpack it:
    2. wget http://www.anchor.com.au/blog/wp-content/uploads/2011/03/idrac.tar
      tar xvf idrac.tar
    3. Build and install it:
    4. cd idrac
      make
      make install

    All done! It should take effect the next time you launch a virtual console.

    If you want to make use of magic SysReq, make sure that it is disabled on your local workstation! (otherwise you may get a bit of a surprise…)

    cat /proc/sys/kernel/sysrq

    If the output is not zero, then edit /etc/sysctl.conf and make sure that it contains

    kernel.sysrq = 0

    then (as root) run sysctl -p

    Under the hood

    When you are logged into an iDRAC and you click on Launch Virtual Console, your browser downloads a dynamically generated JNLP file for a Java Web Start based KVM application. The Java Web Start tool (typically /usr/bin/javaws) processes the .jnlp file and proceeds to download the KVM software from the iDRAC and run it. The KVM software makes a connection back to the iDRAC on the standard VNC port (5900) (with the single use credentials that were provided to it in the .jnlp file).

    At this point, you could easily be mistaken into thinking, Ah, VNC, that’s got to work well right. Such a simple thing and all“. Unfortunately you would be mightily wrong :( .

    Whilst the iDRAC is using the standard VNC port, it appears that the implementation has been somewhat customised. Normally under the VNC protocol, a client will send key events to a remote server using standard X Window System keysym values to represent the key. It appears that, for some unknown reason, that the iDRAC client instead sends raw keycodes which it is able to read through the use of native code (this is why the problem isn’t seen on other platforms). Raw keycodes being constant can not be depended on by an application.

    As you can guess by now, the keycodes in Linux have changed thus causing the assumptions to break. In the past, Linux distributions used the kbd keyboard driver. Around 2008, most Linux distributions switched to the much nicer evdev input driver system which unifies events from different device types, handles hot plug, and generally makes things sane. A consequence of this change however was  that some keyboard keycodes ended up changing. The end result is the broken arrow keys (plus other less frequently used keys).

    The work around above adds more evil to the equation by making a small pre load library to change the behaviour of an X11 function which gets events to silently rewrite the keycode. It currently only remaps the arrow keys and print screen (used for SysReq). When using it, the arrow keys will cease to work in the KVM application menus but that isn’t such a worry. Anyway, have a read of the source code if you need to change anything. Good luck!

    1
    Comment