Good news, everyone! PostgreSQL 8.4 is out, as of several weeks ago. Okay, this isn’t “new” news, but it’s not at all ungood, delay or otherwise. You can catch the feature list here, there’s some good stuff:
http://www.postgresql.org/about/press/features84.html

Of course the real trick is getting more of our customers to use the superior alternative, *le sigh*

As if you *needed* any further convincing to choose Postgres over Mysql, we’ve put together a little comparison for you that highlights some of the most obvious differences for you.
http://www.anchor.com.au/hosting/dedicated/mysql_vs_postgres

Oh yeah, using a Redhat OS? You’re short outta luck, you don’t get anything newer than 8.1.11 in the stock release – go use Debian, it’s better anyway. What are you missing out on? Plenty.

Part A

Use a fresh answer book for this part. Show all working-out with your answers. Approved graphing calculators may be used.

It is 4am on a Friday morning and Eirin has been woken by an SMS from the emergency support line. A colocation customer’s server is offline, and she needs to visit the datacentre. Due to sleep-deprivation and/or ethanol intoxication, Eirin is unable to find the lightswitch, but needs to throw on some clothes and get there ASAP.

Her wardrobe contains three skirts {black,black,red}, four tops {purple,blue,black,black} and two pairs of comfortable shoes {black heels,black sneakers}. Each piece of clothing is chosen at random.

1. How many distinct outfits does Eirin have available?
2. What is the probability that Eirin will have a coordinated outfit?
3. How much easier is it to just pick up whatever’s lying on the floor from last night?
4. Will it be faster for her to put her hair in a single french braid, or twintails?
5. What is the probability that the colocation customer has done something totally silly, like changing their firewall configuration and subsequently looking themselves out in the process?
6. How much will Anchor charge the customer for this out-of-hours callout?

As I mentioned in my last documentation-related article, having a wiki or other documentation repository is really only a small part of the system documentation battle. What really counts is actually getting it chock full of all the knowledge in your organisation, and — possibly even more importantly — keeping it up-to-date forever and ever, Amen.

Just telling people to document their work really doesn’t work. While a quick “hey everyone, write some docs!” might get a desultory page or two written, it’s not really enough. Writing things down needs to be as natural as breathing, and about as easy. In fact, you should really aim for it to be easier to write than to not write, so that the “natural” choice is to do it.

Finding and eliminating the things that prevent people from writing documentation is the key. Every barrier to writing a good, solid document that you can remove means that much more documentation can be written. While I’m sure there are a million possible reasons “why not”, here are a few that I’ve noticed (usually in myself), and how I go about dealing with them.

• They don’t feel they have to. Sure, there aren’t many people who would answer a confident “no” to the question “Do you need to document any of your work?”, but that’s different to it feeling wrong not to document things. Encouraging a culture of documenting things, so that any lack of documentation feels wrong at a gut level, is a crucial part of getting documentation written.Please note that when say “feels wrong”, I don’t mean that it feels illegal or naughty — that way lies guilt and recriminations and every other kind of organisational unpleasantness. No, what I mean is that you feel uneasy about something until it’s been documented — like leaving the milk out on the bench. You just don’t feel right until it’s back in the fridge. (Well, OK, maybe that’s just me, but you get the point).
• “I’ll remember it“: Yeah, sure you will. Whilst there are people who have an eidetic memory, they’re rare enough that it’s unlikely any of them work on your team. What’s more, even if you have someone like this, not everyone is so gifted, and an eidetic memory still doesn’t confer telepathic ability, so communication is still required.People who argue against the need to document just need to be talked down from their suicide pact. Pair them with someone who likes documenting, build a documentation culture, and let them experience how much better it is when things are written down.
• They think they suck at it. Very few people like to do things they’re not good at, and writing docs is no exception. It’s true that some people really do suck at writing, and those people need to be trained to improve their skills, but far more people can do it, they just don’t think they can. Gentle encouragement, pairing with someone who does know they’re good at writing, along with removing the other blockers (so that people start trying it and realising they’re good at it) will solve much of this problem.
• I don’t know where to put this where people will find it. I can strongly relate to this sentiment, as it’s one of the most common reasons I give to myself for why I don’t document things sometimes. Ultimately, what I think any rapidly changing wiki needs is a librarian — someone who makes it their responsibility to keep all the information organised. I can imagine that a large enough organisation might have a literal librarian (someone trained in information collection and management), but almost everyone doesn’t have that much stuff. Instead, there just needs to be someone who knows that they’re responsible for keeping an eye on the clutter, and everyone else needs to know who the librarian is, too, so they know who to talk to about their categorisation woes.I would say that a project manager with a strong technical background is the best person to act as librarian (non-technical project managers, whilst being a bad idea in general unless there’s a strong technical lead, are especially bad in this case because they lack the experience to know how most of the documentation will be used, and hence can’t arrange it properly). They should have the “high level” view necessary to keep classification and arrangement logically sensible, without getting bogged down in the minutiae of one little corner of the world.

  Rotating the job of librarian around the team doesn’t work, as what’s really happening when someone is arranging everything is that the content is being organised in a way that is most logical to the librarian, and then everyone else just starts to learn to think like that person when it comes to finding the information again. Changing who does the job regularly doesn’t allow any of that to happen, and everything falls apart.

• I don’t know what to write. We’ve all been there: the horror of writer’s block. The techniques for overcoming this are many, varied, and are learnt over time as you write more. Pairing, good team support, and experience are the keys to overcoming this problem.
• Not knowing who to write for. This comes down to knowing how to write, but it’s worth making as a separate point. Before you start writing anything, decide who should be reading this and more importantly why they’re reading it. This will help ensure the correct tone, level of detail, and document structure. Describing the architecture of the system is a whole different style of writing from giving a procedure to resuscitate the app server in a real big hurry when it fails, and it’s important to get the tone right, otherwise your document is going to be pretty un-useful.
• Nobody will ever read it anyway. In my pre-document-everything days, this was my go-to excuse. I couldn’t count how many times I used it. It can even be true, sometimes — if someone’s come up empty every time they looked for docs, pretty soon they stop searching. The best way to avoid this feeling is to not let it grow, but if you’re already in this situation, you need to work at reversing the impression.

  The only way to get people back into the habit of looking for and reading documentation is to make them explicitly aware that it exists, and that reading it will solve their problem quicker than interrupting someone else and asking them. This means that your documentation must be easily discoverable (good searching, good indexing), the docs in there must be high quality (constant QA), and it must be very likely that the answer is actually in there (keeping documentation up to date).

  On a closely related point, if you’ve got any external systems that tell you to do things, ensure that they provide easy pointers to relevant documentation. For example, your monitoring system should provide links to wiki pages describing how to diagnose and fix each individual problem (so e-mail alerts for the “Is the primary website up?” check should link to a procedure titled “how to restart the primary website when it fails”).

  As a more concrete example, we’ve got an automated event handler that resizes disk partitions for customer backups when they start to fill up. When that handler fires, it creates a ticket describing who was resized, and provides a direct link to the procedure for updating billing records. Nobody ever has any excuse not to follow the procedure, and they’re constantly reminded of the presence of a huge swathe of useful documentation on how to do their jobs.

• If I write stuff down, you can outsource my job / fire me! If you really are just getting documentation written so you can outsource their jobs, then you’re evil and can get stuffed. If your corporate culture is such that people are legitimately worried about this, you’ve got bigger, more toxic problems than a lack of documentation. On the other hand, if it’s just rampant paranoia, I suggest that perhaps a psychiatric assessment might be of assistance.
• I don’t have time now, I’ll do it later. To this, I say: “Yes, you do” and “No, I’m pretty sure you won’t”. Documentation, like automated testing, is a dish best served with the main course, not for dessert. Budget time in your project plans for documentation, and make sure everyone knows it. Make reviewing the documentation part of the review, and be generous with people who slip the timeline due to documentation work.A closely related problem is getting dragged off to do something else as soon as the “real job” is done, without having done the documentation. To combat this, everyone needs to know that “the job’s not done ’til the wiki’s up-to-date”, staff must have the authority to say “I can’t do that other task, because I haven’t finished documenting this yet”, and management needs to learn to stop judging completion by when the service is working, but rather by when it’s documented.

Ultimately, the best way to get documentation written is to make the writing a natural a part of the workflow, like logging into your workstation when you start work. Some of the ways I think work well to move everyone in the right direction include:

• When new team members come on board, take the time to give them a guided tour of your existing documentation. Sure, these days most technical people already know what a wiki is and how to do the basics, but everyone structures their wiki content differently, and different wiki engines encourage different behaviour. Give the new geek a strong impression of how good your docs are, so they’ll be encouraged right from the start to keep that tradition going, and give them an idea of how the wiki is structured and how to find information so that their early experiences aren’t frustrating.
• Answer all questions with a wiki page — either point someone to an existing page, or write a new one and point them at that. This can be slow and tedious at first if you’ve got a huge, undocumented mess, and might have to be applied selectively at first, but communicating through the wiki both ensures that documentation gets written, and that everyone gets into the habit of using the wiki to find the answers to their problems.As an aside, the person answering the question should be writing the page, instead of the questioner getting a verbal answer and then writing things down. Whilst this undoubtedly takes more time, it means that people can’t get away with skimping on their docs and then expecting the new guys to take up the slack, and the person asking the question is unlikely to have sufficient visibility into the problem to structure a page appropriately. Also, having someone write a page (to answer the question) and then someone else immediately read it to get the answer, provides a simple informal QA mechanism.
• In your project plans, explicitly describe the documentation to be written for every single step. (You do have project plans, don’t you?) This isn’t just adding “Document as needed” to the end of each item description; that’s no where near detailed enough. Instead, write things like “Setup an internal package repository to contain all of the cluster-specific packages. Document how to build new packages, update existing packages (both existing-internal and newly-modified-from-upstream packages), and how to upload built packages to the repository and rebuild the indexes at Procedures/PackageManagement”.Yes, most of that description covers the docs rather than the core point, but that’s OK (and intentional) — it’s far more likely that whoever gets assigned that task will know how to setup a package repository and can judge when it’s done; far less likely is that they’re going to automatically think of all the possible ways that documentation should be written, especially when they’re excited to be getting on with the next task. It’s also ridiculously hard to QA a completed task if there’s no detailed list of what docs should have been written.

  When I’m not intrinsically motivated to complete a task, I break it down into steps so small that each one is “obviously trivial, I can do that in five minutes”, and then there’s no reason not to do it. It’s the same thing here — “document as needed” is big, scary, open-ended, and doesn’t have an obvious starting point. “Document how to do this, place it here” is a series of small, simple, completable tasks, and there’s no mental block over where to put the docs to get in the way.

• Consciously budget time in your project plans for docs. I prefer to make it a separate column in the project plans, but whether it’s included in your other estimates or not, it definitely needs to be included in the budget or it’ll never get done.Separate line items for documentation allows you to do your evidence-based scheduling more accurately (which is always good, as it shows you whether your doc time budgets are accurate or not) and — most importantly — says to everyone “I have explicitly set aside time to document this, so there’s no excuse not to do it”.

  On the other hand, making it explicit that you’re spending half your time writing docs gives meddlesome senior executives yet another thing to haggle over and pressure you to cut from the schedule (“What? Three weeks of your two month project is writing documentation? Are you insane? The board wants this done by last week! Get out of my office!”). Only you can decide if your corporate culture permits you to make it explicit, but if you want docs, you can’t not budget the time somewhere to make it happen.

• Include constant documentation review and improvement in all your operational processes. For example, our monitoring system alert handling procedures mandate that every alert goes into a ticket, and the ticket doesn’t get closed until it’s been through a set of review procedures, which include checking that it’s not a recurring problem (and fixing that), adjusting whatever needs adjusting if it was a false positive to make sure it doesn’t happen again, and — the key point in this discussion — fixing any documentation that wasn’t as good as it could be in helping to solve the problem. Don’t let any opportunity to improve your documentation go to waste.
• Hire for technical writing skills. Make it a mandatory item in your interview process that the candidate demonstrate the ability to string a few written technical sentences together in a comprehensible way. Also, if you’ve got anyone in the team already who can’t write, encourage them to improve their writing skills — going as far as paying for them to attend a creative writing course (on company time, of course), or even running something internally if you’ve got enough people who can’t write. All of this emphasis on good written communication skills will make it obvious that you value good writing, and will encourage people to obtain and maintain those skills.
• Be the change you wish to see in the world. Demonstrate that documentation writing is for everyone, by holding yourself to the highest possible standard. Document everything you do and know meticulously, do it as you do the work, and basically follow all the good advice you’re giving to everyone else. This doesn’t mean that you write all the documentation yourself (if you’re a manager, the chances are that you don’t know enough about everything to do it properly, anyway) but the work that you do do needs to include full and complete documentation.
• QA docs along with the rest of the deliverables. Whenever you have a QA review of work done, make sure that it explicitly includes reviewing the accompanying documentation, both for style and substance, but also that the purpose of the documentation is met. Emergency recovery procedures look very different to project planning notes.
• Ensure that people are kept up to date with changes, by having edits (including a diff of exactly what was changed) e-mailed out to everyone. I can’t believe that there are wiki systems that don’t do this, but I certainly will never use one of those systems (again) if I can avoid it. It’s just way too useful a feature to forego.The benefits are many and varied: it keeps the presence and quality of the documentation firmly in everyone’s mind; it’s an immediate QA process; people can’t get away with shoddy edits because everything they write immediately gets seen by everyone else; and, as processes change and improve over time, everyone’s kept up to date on those changes so they’re not doing things in the old, inferior way.

Given that long list of “Do’s”, here’s a don’t: espousing the need for documentation without doing anything else. “Yes boss, we all know how important docs are, but the project plan is already unrealistic even without any writing, and every time we try to write anything you roll your eyes and tell us to get on with the more important things, and you put the kybosh on hiring that extra person to help out with the extra workload, and oh yeah, you never write anything down yourself so what the hell?”

Next time, I’ll talk about how we structured the wiki for Project Starbug.

Not content to take Oracle lying down, we’ve made a couple of small changes on our systems to make life a little saner. The first is a substantial improvement to the default initscript, the second is some shell/environment hacks that should really be done by default. I alluded to these a few posts ago, but hadn’t gotten to publishing them yet.

If you’re a poor sod that has to use Oracle, head over and have a look at the improvements, feedback is always welcome (it might not be perfect, but it’s a lot better). In the interests of not being acquired by Oracle Corp, we’re publishing a diff to the initscript, rather than the full file.

Given that you’ve been suddenly and completely convinced of the need for documentation in my previous post, the question still remains: how does one make documentation appear on a consistent and ongoing basis?

If you’re really, really lucky, you’ve been spared the painful experience of putting up a wiki somewhere (or, worse, forked out a pile of cash for a “knowledge management system”), sticking some info into it at random, and then… nothing. You planted the seeds of a documentation tree, why isn’t it growing, and flowering, and solving all of your problems for now and forever?

For Project Starbug, we’re creating a whole new infrastructure, more-or-less from scratch. This is the easiest possible environment to make work, because you’re not constrained by what is already in place (and that you can’t afford to get rid of), and the whole thing isn’t in production so there’s no need to get freaked out by the thought of taking a major site off the Internet due to making an ill-advised change — and, most relevantly to this discussion, there’s no giant mass of undocumented… stuff that needs to be picked apart and documented. There’s nothing more deadly to motivation than the idea that when you’ve got this bit documented, there’s only 350,000 other bits to go.

So, if I didn’t want to end up with a shiny, new, incomprehensible and undocumented system, we needed to start focusing on documentation right off the bat and build the documentation alongside the rest of the system. This, in turn, meant that we needed to have something easy to work with, well structured, and above all ready to go before anything else could really kick off.

What to use was a no-brainer. Wikis are straightforward to access and edit, and there’s very little downside to them. We use moin internally for our documentation extensively, so it wasn’t a hard sell to spin up another copy of the wiki software to contain all of the documentation for this project. Most widely-used wiki engines these days are on much the same level, though, and it’s really just a matter of preference which one to use — mostly based around the language you’re most comfortable using (Python == Moin, PHP == MediaWiki, Perl == twiki, Ruby == instiki, Java == something useless and enterprisey), because you really want to be able to write plugins and extensions. One day I’d love to try ikiwiki, because that means I can edit wiki pages without even needing to open my web browser, which will be a particularly special kind of bliss.

Why did we use a separate wiki, though, and not an extension of our existing one? We want to communicate with the customer as well as we possibly can, and the content of the wiki is like a big, persistent communications nexus, and giving the customer (especially this customer, who really knows their stuff) direct access to be able to read all the internal procedures and technical information relating to the management of their infrastructure is a massive boon to communication. Who knows when they might see something we’ve written and say, “Hey, that’s not right!” and fix it? We’re the system administration experts, not the experts in their application, so it makes perfect sense to have them as tightly integrated as possible into the management of the whole infrastructure.

Though we may have made it over “Documentation Hurdle #1″, the race had barely even begun. Plenty of well-intentioned doc projects have gotten something started, and then withered on the vine. The key is to make sure that the documentation stays maintained, and keeping up with the growth of the infrastructure and it’s constant changes. The most important way to do this is to identify the reasons why people don’t keep a reasonably useable documentation repository maintained, and remove those reasons, leaving no possible excuse not to write docs. It needs to be easier to write docs than to not write them, otherwise they’ll get forgotten in the pressure of the moment, and playing catch-up is painful and annoying.

In the next article, we’ll examine why people don’t write docs as often as they know they should, and how to create a “documentation culture” in your team.

You’re trying to reconfigure a service to do something new. Digging into the config files, you see that everything’s been modified heavily, but it doesn’t seem to make a lot of sense. Everything’s currently working, so it must be right, but why was it done like this in the first place? It looks like this can be simplified, but… you’re not sure. What if it needs to be this complicated for a reason?

Or perhaps it’s 2am, you’ve just been notified by the monitoring system that a critical system that you don’t have a lot of experience with has gone down. Logging in to the server you thought the service was on, you realise that this isn’t the right place, so you waste precious time tracing the network through the load balancers and proxies to where the service really lives. Then you realise that you don’t know where any of the config or binaries are. By now you’ve been ferreting around for 20 minutes and your SLA is just about blown, and you still have no freaking idea what’s going on with any of this…

Maybe the problem is that you’ve got something on your network that’s causing problems from excessive broadcasts on some random IP address. All you’ve got is the MAC address, but what server does that map to? You’ve got no idea.

Then there’s the frustration of being asked by the boss to setup a new instance of some minor service that nobody’s touched in two years as a test platform for a new client. Nobody else in the office remembers how it was done last time, only that it wasn’t a whole lot of fun. Of course, the boss doesn’t see why you can’t whip this one up nice and quick, given that “we’ve already got one of these over here, why should it take too long to do another one?”

By now, you probably know what this is all about (if the post title didn’t give it all away to begin with). We’re talking Documentation. Everyone’s heard a hundred reasons for why you should write documentation. Here’s number 101:

A system really only survives as long as people understand how it works and how to maintain it. The moment that information is lost, the system is basically dead — sooner or later someone’s going to come along and tear it down and replace it with something else, something that they understand. Of course, the operation of that system will sooner or later be lost and the cycle will repeat.

If you want to maximise the life of the systems you build, then, you need to ensure that the crucial details about it aren’t lost. The only way to do that is to write things down. Your memory will fade, you won’t always be around to answer questions, and trying to figure things out post-facto is a pain in the arse.

Of course, it’s all well and good to blather on about how wonderful
documentation is, but the difficult bit is how do you write good
documentation? Well, I can’t say I’ve got all the answers, but in the next few installments of The Adventures of Project Starbug, I’ll describe how we laid out the documentation for this new project.

Long before any code gets written or any servers deployed, a quiet yet crucial job is being performed. The poor tech who is doing this work won’t get much credit, and almost certainly none of the glory, but if this job isn’t done properly, then none of what gets done later will be of much use.

I am, of course, talking about… requirements gathering (bom bom bommmmmm).

In the case of project starbug, the requirements gathering work sits at the “fairly straightforward” end of the spectrum — but it’s by no means easy. What makes the job easier than average is that the site is currently operational, and our primary job is making sure that the new server farm that we’re building will (a) match what is currently running (in terms of system setup), and (b) have enough capacity for future growth.

The system configuration to support the customer’s application is fairly easy to achieve for this project — the customer knows their software and what it requires really, really well, and we’ve got the existing setup to examine to try and work out how the pieces fit together if it isn’t immediately obvious. The main requirement here is to make sure that all the requirements are documented thoroughly. Yeah, writing docs isn’t the glamourous end of the job, but it is important, and is something that pays dividends down the line. More on that in another article, though.

The capacity issue is a lot trickier. The new architecture we’re building is completely different to the current architecture (which isn’t scaling well, hence why it’s being left behind), so it’s hard to draw any direct performance metrics by just looking at what hardware is already in use (especially since the current setup uses virtualisation a little too heavily, which makes comparisons based on hardware spec even harder).

Based on a cursory examination of the bottlenecks in the existing system, along with previous knowledge of the system behaviour, I decided that the primary bottleneck of the system is disk I/O. This site isn’t your typical large-scale website; it does a lot more file management than is typical. As a result, the key thing we need to ensure in our new hardware setup is that there is sufficient disk I/O capacity.

Memory constraints (large app servers, mostly) take a close second in the “what is going to kill us here” stakes, as the current infrastructure is using somewhere north of 100GB of RAM (spread across all the various servers that are being used). We want to provision this plus some extra, as moah RAMs == moah disk caching, and moah disk caching == better effective disk I/O. Win all round!

CPU, on the other hand, is practically never an issue. The servers run a lot of separate processes, but they’re almost always waiting on stuff coming from the disk, so with the current state of the art in server CPUs being quad core, we really shouldn’t have a CPU bottleneck.

Although I said earlier that memory and disk I/O were tied for the title of “biggest performance bottleneck”, there was really no competition for which one of these was going to keep me up at night. Solving the memory problem is easy — modern chassis can easily accomodate 32GB (or more) of RAM. There was never any doubt that we’d be using at least a half dozen machines, so stocking them all with 32GB of RAM should be plenty.

No, the worry was always going to be the file I/O capacity, and making sure that we had both the speed we needed, as well as the storage capacity. While the site doesn’t need petabytes of storage, it does need a decent amount of space, and it all needs to be pretty quick. What’s annoying (but understandable) about storage systems is that you can either have a lot of capacity (1.5TB SATA drives are common as MCSEs) or you can have a lot of speed (15k SAS drives max out at 300GB). We could get the storage space we needed with 300GB drives, but will it be quick enough?

To try and make some sort of an apples-to-apples comparison, I needed to have a number that represented how much I/O was being done at present, and which could be compared to what our new hardware infrastructure is capable of.

In the end, what I went with was running the sar tool on a number of the existing machines to try and get an idea of how much disk I/O is being requested by the machines. There are a number of things that might make this comparison inaccurate, but in the end I decided that there wasn’t really any better metric.

The key thing was to try and get the statistics at the same “layer” of the stack in both cases — in this case, when the kernel passes the I/O request off to the disk (or RAID controller, in this case). The benefits of this are that it’s a single statistic to compare, and it’s not ridiculously impossible to synthesise a load at this level for benchmarking purposes (obviously, running the live site on the new infrastructure to benchmark the new hardware isn’t a real winning strategy). When all’s said and done, though, these benchmarks are an estimate, and are unlikely to be completely accurate. That needs to be kept in mind when doing the hardware estimations later on.

All of this information gathering and benchmarking takes a pile of effort, but without it there’s no chance whatsoever that any sizeable infrastructure will be correct for the job it needs to do. I was surprised in this case at how little hardware we ended up needing, however on a previously sized system I worked on the initial guesstimates turned out to be an order of magnitude too low (the system ended up with some thirty-odd servers instead of the five initially ordered). Without a comprehensive analysis of the reality of the situation, you’re either going to end up with a poorly performing site, or else a pile of unused hardware.

Anchor recently signed a new customer. This is not normally news, but then again, this is not a normal customer. They’re fairly sizeable, and need a large scale dedicated infrastructure to handle their request volume.

Because of the scale of the development, and some of the novel approaches we’re going with, we’ve decided to blog about the experience of setting it all up. In effect, we’ll be doing the development of this infrastructure in public. Over the next couple of months, as everything comes together, I’ll be regularly writing up what we’re doing, how we’re doing it, and the good, the bad, and the ugly. Some details will need to be obscured, for customer confidentiality reasons, but as much information will be made public as we possibly can. If you’ve never been involved in a big infrastructure project, hopefully you’ll be able to get a feel for what goes into something like this.

We’re code naming this sizeable effort “Project Starbug”, and all the blog posts in this series will be tagged with “project starbug”, for ease of identification. Follow along, and watch the adventure unfold…