Getting a start in IT: Part 2

Published December 6th, 2011 by matt

(This is the second part in a two-part series on getting your first job in IT. Part 1 of the series covers getting a job in the first place; this article is about the sort of job you can expect when you’re starting out)

Working in IT, like anything in life, is a learning curve. You start out knowing nothing, and you slowly gain experience and knowledge that allows you to work at a higher level, on more difficult (and, hopefully, interesting) problems.

A difficult fact for many people to accept is that there is no substitute for real experience. TAFE (community college), University, and industry certifications are not equivalents. These academic credentials have their value, but they’re not an equivalent for experience. They’re like power mirrors or window tinting on a car — they add value, but power mirrors don’t get you anywhere by themselves — you need four wheels and an engine.

This means that everyone will “do their time” working at the entry-level jobs, one way or another. Whether you have any sort of relevant academic credentials or not, you will spend some time learning the ropes, at a job which you will probably feel isn’t what you signed up for (very little IT bears any resemblance to scenes from “The Social Network”).

Now, you might luck out, and your “entry level” job might be that you’re the entire IT staff for a small company (that’s what I did at while I was at Uni), in which case you’re both the printer monkey and the CTO, but it’s not an easy road to travel. You’ll be working your ring off just trying to learn enough to be able to do the practical basics of your job, and you will learn a huge pile of bad habits as there’s nobody with experience to tell you “no, don’t do that”.

This isn’t to say that every single person who has a job in IT spent 18 months as a gofer, and there’s no other option. I only said that you needed practical, hands-on experience, not necessarily a pay-check. There are a few ways of getting that experience that doesn’t involve a regular job as someone’s cable monkey, but they do require a certain amount of effort on your part. For development work, it’s fairly easy to get some experience by contributing to open source projects. You learn a lot of really essential skills: the most important of which is to write real code. No course I know of is even a half-decent substitute for writing real code. You also get plenty of exposure to the real technologies and processes used in software development, like revision control, testing, collaboration, communication, and compromise.

Sysadmins have a bit of a harder time because there’s not many public systems that you can administer as a hobby, but running your own server on a cheap VM teaches you an awful lot about the basics of systems administration (whether you learn good things or bad things is actually far less important).

These sorts of “hobbyist” things you can do often blur into actual work experience, too — running a server on the Internet for yourself and a few friends might get you a job through word of mouth setting up a static website and blog for a friend’s dad who’s running his own small business. He tells a friend of his who runs a bigger business, which leads to something else, and so on and so on. All of that’s good experience, and helps to demonstrate that you’ve got a bit more to offer than everyone else.

Even then, though, don’t get your hopes up. SAGE‘s Core Job Descriptions (PDF) defines a Junior Sysadmin as someone with one to three years of full-time experience as a systems administrator.

0
Comments

Getting a start in IT: Part 1

Published December 5th, 2011 by matt

Nobody in the IT industry was born with into it. At some point in the (distant?) past, every one of us was in the position of looking for (or falling into) our first job in the industry. I think us old hands forget how daunting that process was, the memory having been dimmed by the passage of time. So I’ve written a couple of posts (Part 2 will be coming out in a few days) to try and demystify the process a little, for people who might be looking to get their start.

(Obligatory pimpage: Anchor is often looking for good people; keep an eye on our jobs page for new positions as they come up)

Competition for good entry-level jobs in IT is always going to be tough. Whether it’s an employee-friendly market (where there are lots of jobs and not many people who want them), or an employer-friendly market (where there are few jobs and plenty of applicants), you will always be competing against other people who also want the job.

This isn’t a resume article — although I’m going to do one one day soon, because life’s too short to spend it filtering crap resumes. Instead, I’m going to talk about what you can actively do to make yourself a more attractive candidate, especially early in your career. Later on, your work history and job performance count for practically everything, but you can’t get a good work history without landing that first job, so let’s talk about that first.

There are two ways to “get ahead” in the IT industry when you’re starting out. By this, I mean get the “better” entry-level jobs, with a bit better pay and conditions and hopefully a better career progression path. The two methods have different target audiences amongst employers, so you can tune your approach somewhat depending on the sort of job you might like to land:

  • Go to University (College, for some). Consistently get astonishingly high marks, academic awards, scholarships, extra-curricular activities (“President of the University Computing Club”), etc. This will take you to the top of the list primarily at large corporate employers. It shows you’re a very hard worker and very smart (you can’t consistently get top marks without some of both), and it shows you are capable of focusing on tasks that you don’t necessarily find interesting.

    Not all of working life is doing the things you love; being able to stick with something that is important but boring is an important trait at any job, but it is particularly valuable in the corporate world, and hence is highly sought after (and well paid).

  • The other way to get noticed is to live la vida nerda: Basically, being a geek. That’s a very strong hiring point here at Anchor, and at many companies at the less corporate end of the industry.

    “Being a geek” means things like learning programming languages (especially non-mainstream ones) in your own time, because you find it interesting. Having a 19″ rack at home filled with cast-off network and server hardware, running a variety of OSes, which you can’t help but compulsively fiddle with because you just heard about a new OS that runs on anything and uses whalesong-over-IP as it’s native networking protocol. It might also mean explaining in an interview that the reason you didn’t get a good mark in some irrelevant subject is because you were working so hard on making your wyse-60 talk to the BeOS installation you had running on the spare Juniper router that you forgot that you were supposed to be doing the final exam (not that that ever happened to me, oh no)

Of course, the two aren’t mutually exclusive, and in many ways they go together to some degree. If you’re a hardcore geek, the chances are most IT subjects will be much easier, and you’ll get good marks easily. You’ll almost certainly still need to work hard at those subjects that don’t interest you, though, if you want to get top marks across the board — which is why employers look for consistently excellent marks, to make sure you’re not just a geek savant.

A side note on industry certifications: at best, all they will do for you is to increase the amount of theoretical trivia you know. While IT is mostly theoretical trivia, and having more of it can make you more effective, it’s the practical application of that trivia that is the job we do, and quite frankly no degree or industry certification teaches you the ability to get real things done in the real world.

This leads me nicely into the second half of this post, which will appear in a few days’ time, which is all about what your first job in IT will actually involve.

So, there’s the skinny. To get a decent corporate IT job, you’ll be best served by going to Uni and studying like crazy. To get a job at Anchor, good marks don’t hurt, but you get more points for being genuinely interested in technology. if you fit into neither of these categories, you will blend into the dozens, if not hundreds, of other applicants for every job vacancy that comes up on the big job boards, and you will have a hard time landing a good job in IT.

0
Comments

Shining some sun on the cloud

Published November 5th, 2011 by matt

Being the cynical, hard-bitten sysadmins that we are, we’re a bit skeptical about some of the more grandiose claims about cloud computing: 100% uptime, never having to worry about scalability, and all those other things that people who don’t understand reality seem to get terribly excited about.

It’s good to see every now and then that someone else has an experience that matches our own, such as Mixpanel’s decision to move off Rackspace’s cloud and onto dedicated servers. I’d love to know how to negotiate 50%-75% off a vendor’s list price, though…

Posted in WTF

 Leave a comment

0
Comments

Why you should use LVM: part 1319755409 in an infinite series

Published October 28th, 2011 by matt

At anchor, we loves us some LVM. It makes managing storage a breeze.

Now, we’ve got one more reason to love it. Because now we have lvmsync.

Like most modern hosting companies, we run a lot of VPSes, and sometimes the chunk of storage they’re on isn’t where it needs to be, so we have to transfer it around. Ordinarily, this would involve a lengthy period of downtime while dd did it’s business and sent the whole LV across a network.

NOT ANY MORE! Now, with the magic of lvmsync we can transfer the bulk of the data while the VM is running, and then do a quick transfer of just the changes after we shut the VM down.

We think our customers will appreciate the reduction in downtime, and I know we’ll appreciate the wonders of LVM just that little bit more.

Tags: , ,
Posted in FTW

 Leave a comment

0
Comments

Why PostgreSQL?

Published October 2nd, 2011 by matt

In our never-ending pursuit to bring the Joy of PostgreSQL to the entire world, I’d like to recommend this blog post which summarises a lot of PostgreSQL’s useful features. Consider it some nice, light Sunday reading.

Posted in WTF

 Leave a comment

0
Comments

SSH ControlMaster: The Good, The Bad, The Ugly

Published February 24th, 2010 by matt

Do you love SSH for the good it has done for mankind, but get annoyed by how long it takes to establish a connection over a high-latency connection? Perhaps you have a process that needs to make thousands of SSH connections, and you’d like a little extra speed from the whole thing. Either way, ControlMaster is your new best friend.

The concept is very simple — rather than each new SSH connection to a particular server opening up a new TCP connection, you instead multiplex all of your SSH connections down one TCP connection. The authentication only happens once, when the TCP connection is opened, and thereafter all your extra SSH sessions are sent down that connection.

If you’re SSHing between machines on the same LAN, or otherwise a short ping away, you probably wouldn’t notice the difference — the round-trip times are negligible. However, when you’re doing transcontinental SSHing (which we do often, when we’re managing customer machines in the US), it’s a godsend. On some trivial benchmarking I did when validating ControlMaster for our use, I found that we were saving nearly 2.5 seconds per connection — a drop from 3.3 seconds to 0.8. Mighty convenient.

It’s simple to use, too. If you just want to enable “opportunistic” multiplexing, you can do something as simple as this in your SSH config:

Host *
ControlMaster auto
ControlPath ~/.ssh/cm_socket/%r@%h:%p

Then mkdir ~/.ssh/cm_socket, and you’re away. Any time a connection to a remote server exists, it’ll be used as the master for any other connections. Perusal of the ssh_config(5) manpage should give you the necessary hints to setup more restrictive configurations. If you need to disable control master for a given connection (the reasons why this might be necessary will be covered shortly), you can pass -S none to ssh (or set ControlPath none).


Whilst this basic setup is undeniable, pure, distilled awesome, there are some limitations and caveats to beware of. The first, and most important, is that SSH session multiplexing isn’t particularly stable when you try to put a lot of data down it from a lot of connections at once. This came to light fairly early on in my testing, when I stress-tested things by doing about 25 concurrent rsync runs all at once. The result was a large number of rsync sessions going “aiee!” and falling over. So, don’t do that.

The second, semi-related problem, is a simple bandwidth issue. For a given connection latency and TCP configuration, there is a hard limit to how fast you can send data, due to the time it takes to acknowledge the packets being received. When you’re multiplexing multiple file transfers down the one TCP connection, therefore, your total transfer speed will be limited by this TCP speed limit. Once again, it’s unlikely that this will cause you problems on a LAN (where round-trip delays are negligible), but in the high-latency world where connection sharing does the most good from a connection setup perspective, the speed limits will cause much wailing and gnashing of teeth. So, the take home message is: if you’re doing a lot of heavy data transfer over SSH, ControlMaster probably isn’t the solution for your problems. Instead, run multiple concurrent SSH connections, as the TCP speed limits are per-connection, so you can still fill your high-latency gigabit pipe — you just need lots of concurrent connections to do it (see also: BitTorrent).

Finally, there is something of an annoyance with ControlMaster, and it’ll probably confuse you mightily when you first come across it. Because all of your SSH sessions are multiplexed down a single TCP connection initiated by the first SSH session, that first session must stay alive until all of the other sessions are complete. This problem will manifest itself as an apparent “hang” when you log out of the remote session that is acting as the master — instead of getting your local prompt back, SSH will just sit there. If you Ctrl-C or otherwise kill this session, all of the other sessions you’ve got setup to that server will drop, so don’t do that. Instead, when you logout of all the other sessions, the master will then return to the local prompt.

If you’re doing a high volume of SSH connections to a particular remote endpoint, consider setting up a dedicated master connection — that way it’ll always be available (and you don’t have to worry about master logout hangs). I use a simple daemontools service, that runs ssh -MNn user@server. Works an absolute treat.

2
Comments

ERROR: SSH agent has too many keys

Published December 23rd, 2009 by matt

Unfortunately, SSH doesn’t produce this error, although it darn well should…

I just had a Github customer report that they couldn’t access their repos via SSH, despite it all working properly yesterday, and “not having changed anything”. A bit of debug logging and an inspired leap of intuition on the part of another sysadmin in the office, and the answer was quickly found.

First off, the symptoms:

  • Debug logging showed that the user was connecting successfully, presenting six SSH keys (none of which were the key of interest) before disconnecting;
  • The SSH key was in the user’s SSH agent (you can verify this with a quick ssh-add -l);
  • There were more than six keys in the SSH agent

This last symptom is the key point. As an anti-brute-force measure (I assume), SSH won’t allow a user to connect and present more than MaxAuthTries credentials (whether they be passwords or keys) before being forcibly disconnected. The default value for this parameter (if you haven’t realised already) is six.

Whilst this makes a lot of sense for passwords (and a lesser, but still valid, measure for keys) it does mean that you effectively have a hard limit of six keys in your agent simultaneously (at least without using SSH configs to specify a single key to present to the server). Any more than six keys, and you run the very real risk that the key you need to give to a particular server will be number seven in your agent, and all your authentications will fail miserably.

Bumping the value of MaxAuthTries to a much larger value works fine for Github — password auth is disabled, and if you can manage to brute force a key you’re welcome to what you can get — but you certainly can’t rely on inflating MaxAuthTries everywhere to get you out of trouble, so: keep those SSH agents lean, or at least specify IdentityFile for all your servers.

Tags: , ,
Posted in WTF

Comments closed

Comments Off
Comments Off

Monitor your servers like it’s 1996

Published December 3rd, 2009 by matt

Whilst I’m a fan of using percentages for my disk space checks, sometimes an explicit size is more appropriate. So, you’d expect the following to work nicely:

$USER1$/check_disk -w 5G -c 1G -p /data/foo

If you don’t actually test that this works (by artificially filling your disk and seeing what happens), you may be dismayed to find that you only get alerted when the disk has 5MB of free disk space. Why is this?

Because Nagios, despite the fact that nobody has sweated the megabytes for about a gazillion years, doesn’t support ‘G’ as a suffix for thresholds. Oh, it’ll make a good show of pretending — after all, the output formatting options have ‘GB’ as an option — but nope, for your thresholds it’s “5000M” all the way.

ROCK ON!

0
Comments

Industry Analysts: Putting the “arse” in Analyst

Published November 24th, 2009 by matt

I’ve never been a real fan of the output of big “industry analysis” firms, since their reports never seemed to really tell the whole story, and didn’t match up with my experiences “in the trenches”. Now I know why. A representative sample:

“I see. So, the companies in your magic quadrant, are they all paying clients of yours?”

“Well, yes they are,” He said, proudly.

“Well, if they are all paying clients, then what’s so ‘magic’ about being in the quadrant?”

“The companies are not all rated at the same level, some are rated much higher than others.”

“And should I be surprised to hear that the companies that pay you more so you can afford to have entire teams cover them full-time; you tend to know a lot about, and they tend to get better ratings?”

No answer.

“Maybe you should stop calling it the ‘Magic Quadrant’ and call it what it really is; perhaps ‘The Quadrant of Companies That Can Afford To Be In It’.

Go read the whole article, though, it’s pure gold.

Posted in WTF

 Leave a comment

0
Comments

Securing your codez from the wily exploit injectors

Published November 23rd, 2009 by matt

Remember the good old days, when Melissa and ILOVEYOU were the major virus threats, spreading via e-mail and causing all sorts of embarrassing conversations at work? Or maybe even earlier than that, when the only way you could get a virus was by engaging in risky sex? (I mean Software EXchange, of course… get your mind out of the gutter)

These days, anti-virus protection for e-mail is fairly thorough, and nobody’s really swapping floppies full of 16 colour games at recess. Malware authors have moved on to new and more fertile ground — embedding their junk in web pages, and relying on browser exploits to gain access to computers. Of course, with this method, you can only get infected if you actually visit a page that has an infestation, so the malware authors have two options: either entice you to visit their sites, or modify existing websites that users will visit in the course of their day — legitimate sites that people know and trust, but with a little added infection.

Enticing people to a whole dodgy site is usually just a matter of providing something people love to look at and sticking it in search engines. Since the attacker has to have a stable, identifiable presence for the search engines to direct users to, that can also be used by anti-malware lists like stopbadware.org to protect web users, so this isn’t a particularly effective means of attack, and is waning somewhat in popularity. Far more effective is infecting a legitimate website with some form of malware. How does it happen, though? In our experience, there are four vectors for infection:

  1. Brute-force password guessing, where the attacker has a botnet they control to just repeatedly try lots and lots of usernames and passwords. They’re bound to get lucky sooner or later.
  2. Some sort of web-based exploit, typically a vulnerability in the web application that allows the attacker to run code of their choosing; this is then either used directly to edit files, or bootstrapped into sufficient access to edit files via another method.
  3. Password “scraping”, where the attacker gets direct access to the FTP password for your site. This can either be some sort of malware on the workstation of the web developer (or someone else related to the management of the website) that gets the password off the local machine (in a saved password file, or via a keylogger), or else via the “lost password” functionality provided by the hosting provider. Once the attacker has the FTP password for the site, they are free to login to the live site and make whatever changes they like.
  4. Direct modification of the website code on the client-side computer, relying on the developer not to notice it and then upload the compromised content to the live site. We recently had our first “confirmed” case of this (where the web developer found the malicious modifications in their local copy), and they swear blind they didn’t download the HTML from the live site (which would bring the “infection” onto the local machine from the infected live site — which we’ve seen before, and categorise under vectors 1 and/or 2).

The countermeasures required to combat all these vectors boil down to a few simple precautions.

  • Use strong passwords. (Protects against vector 1) Yes, they’re a pain to manage, but a weak password is just an open invitation to getting repeatedly and painfully owned. Of course, the strongest password is a keypair, which leads us to…
  • Don’t use FTP. (Protects against vectors 1 and 3) The list of reasons for this is long, but for securing your website, FTP is a pain because you can only use passwords[1]. Switch to using SFTP (the file transfer component of the SSH protocol) and you can use public keys, which are, for all practical purposes, unguessable. You should also encrypt your keys with a passphrase, which means that even if the attacker does get access to your workstation and copies the key, it’ll be useless to them — unless they keylog your passphrase, which brings us to…
  • Keep your workstation secure. (Vectors 3 and 4) It seems that attackers have realised that the weakest link in the website security chain is still the Windows desktop, and they’re increasingly hitting it as the first step in taking over websites (if you get the right workstation, you can get the credentials to hundreds or thousands of websites, because one web developer often works on many different sites). So, on any machine you connect to webservers from, you need to be doubly, triply sure that it’s rock solid — and that’s just a matter of following all the good advice out on the web. Antivirus, antispyware and firewall software, constantly running, well-configured, and kept up to date; keep up to date with your application patches, especially for your web browser, e-mail client, and core OS; don’t visit dodgy web sites; and so on.
  • Protect your e-mail. (Vector 3) If someone can get access to your e-mail, they can also get access to your website, by using the password recovery feature (or impersonating you to your hosting company). If they delete the e-mails that are coming in before you notice them, you’ll never know what’s going on, and all the password changes and workstation security in the world won’t help you.
  • Don’t use shared hosting. (Vectors 1 and 3). This might seem an odd thing to say, given that we sell shared hosting, but it is a legitimate way to reduce your vulnerability. If you use a dedicated server (including a VPS), you (or we) can configure it to only allow logins from certain IP addresses, rather than the entire Internet. This means that even if an attacker does get your password (or SSH key), via brute force or sniffing it off your workstation, they can’t login from their own machine because it won’t have an authorized IP address. On shared hosting, this configuration is impractical, because hundreds of people have legitimate access to the server, from a great many different IP addresses.
  • Code responsibly. (Vector 2) It is said that “PHP is great, because its ease of use means that any idiot can produce a security hole — and most of them do”. Whilst this is a little derogatory to the many (several? few? one? PLEASE?) good PHP programmers out there, it is certainly fair to say that the capabilities of many people who write dynamic code aren’t up to the challenge of writing code that is exposed to the extremely hostile conditions that are the public Internet.

    Thus, if you are not familiar with the common security practices and problems with the language or environment that you are developing for, stop right now and go learn a little. There’s plenty of good information out there on the Internet from people who have learnt the lessons the hard way. Celebrate the benefits of literacy by learning from their mistakes rather than having to educate yourself by cleaning up an infected website. If you feel that isn’t something you can commit to, then please, for the sake of the Internet, find someone else to write the code.

  • Keep your web applications patched. (Vector 2) Whilst some sites do use custom-built web applications, many sites choose to use a standard CMS or other application to manage their website. This is great, because hopefully someone else is taking a bit of responsibility for the security of the software, but that doesn’t do you any good if you don’t keep it up-to-date. Far, far too many people install a CMS once, then forget about it. Almost all of these applications have a vulnerability at some point, and not keeping them up to date is absolutely fatal, because once a vulnerability is found in a piece of software, an attacker can typically use Google to find all of the publicly-available instances of the vulnerable software, and quickly attack them all.

    This means that you need to keep yourself well-informed of any security updates for your off-the-shelf web applications. Subscribe to a relevant security announcements mailing list, or ensure that your vendor sends them to you. (If your commercial CMS vendor doesn’t have this ability, find a new CMS vendor).

Websites get compromised all the time, by a variety of methods. You should reinforce your defences, lest you’re the next target.


1. The first person to mention Kerberos or other unused-in-practice authentication schemes in a comment gets a free laughing at. If you think SFTP and SCP aren’t supported in widely used web development programs, try finding something that supports GSSAPI…

1
Comment