If you get a lot of notifications in quick succession, your Rage meter starts to build up. When it hits 100% you unleash a special attack and reboot the server.

Rachel's gauge is at 100%, circled in blue crayon. She can now reboot the server with her Static Iris

That’s pretty cool, but it turns out that customers don’t like reboots as much as us, so we looked at ways to reduce the rage. One great way to do this is with better documentation; we call it Ragewiki.

Making use of the notes_url parameter, we provide a link to our wiki documentation directly from Nagios’ web interface. There’s one page for each service, with precise instructions on how to diagnose and fix common problems, as well as a brief description of what the service actually does.

So now when you get that SMS at 3am (PROBLEM – ntype on fundle is CRITICAL), you don’t spend 20 minutes flailing through A Brief History of Time, as told by H.P. Serverbox.

To sweeten the deal a bit, we also allow for host-specific instances of a service, which might need extra-special instructions. We also have a page full of terse legacy documentation that we’d like to fallback on in case the new docs haven’t been written yet. We think it’s a cute little hack so we’d like to share with you.

The possibilities are up to your own imagination, we just went for the most straightforward option. You could always link to a big red button that reboots the server straight away.

1. Give every service a URL in the Ragewiki, using the notes_url argument. We attach this to the generic service template so that every single service automatically gets a link.

   # RageWiki ftw
   notes_url /ragewiki/$HOSTNAME$/$SERVICEDESC$

   You’ll notice that we’ve parameterised the URL so that each host-service pair is unique

2. Prepare a rewrite map to check for existence of docs
   This URL will refer to the Apache instance on the nagios server itself. It captures the request starting with /ragewiki/, extracts the hostname and servicename, then builds a suitable redirect.

   Because we want to support per-host pages that may exist, we use a RewriteCond and a smart RewriteMap to check whether the page exists, then redirect accordingly. We use moin as our documentation wiki, with HTTP access control in front of that.

   RewriteLock /var/lock/rewrite.lock
   RewriteMap RageWiki "prg:/usr/bin/xargs -n1 -d '\\\\n' /usr/bin/HEAD -sd -H 'Authorization: Basic EncodedUsernameAndPassword'"

   You may want to read up on Apache’s RewriteMap functionality to make sense of this. The short version: it contacts the wiki and returns the HTTP status line for the suggested page. A 200- or 300-series status code is considered a success – the page exists and should be used.

3. Finally, use the RewriteMap and generate a suitable redirect
   This is a basic set of cascading rewrites, the first success will terminate further processing.

   # Server-specific docs: /servers/$HOSTNAME/$SERVICENAME
   RewriteCond ${RageWiki:https://magic.ponies.anchor.net.au/servers/$1/$2} ^[23]\d\d
   RewriteRule ^/ragewiki/([^/]+)/(.+)$ https://magic.ponies.anchor.net.au/servers/$1/$2 [R,L]
   
   # Whole lotta BGP goin' on (with variable check names, a variant of generic docs)
   RewriteCond ${RageWiki:https://magic.ponies.anchor.net.au/Nagios/Services/bgp} ^[23]\d\d
   RewriteRule ^/ragewiki/[^/]+/bgp[_-].+$ https://magic.ponies.anchor.net.au/Nagios/Services/bgp [R,L]
   
   # Generic docs for normal services: /Nagios/Services/SERVICENAME
   RewriteCond ${RageWiki:https://magic.ponies.anchor.net.au/Nagios/Services/$1} ^[23]\d\d
   RewriteRule ^/ragewiki/[^/]+/(.+)$ https://magic.ponies.anchor.net.au/Nagios/Services/$1 [R,L]
   
   # Catch any checks without docs, and send them to the fallback page.
   # Funky regexes to pass the failed service name through to the fallback page.
   # FIXME: Can we use a positive-lookbehind in these things? Would make it slightly tidier.
   RewriteRule ^/ragewiki/([^/]+)$    https://magic.ponies.anchor.net.au/CommonNagiosServiceCheckReference#$1 [NE,R,L]
   RewriteRule ^/ragewiki/.*/([^/]+)$ https://magic.ponies.anchor.net.au/CommonNagiosServiceCheckReference#$1 [NE,R,L]
   

   Special cases with varied names, like our BGP checks, are easily handled by dropping a custom regex into the chain. It’s best if your service names have a consistent format that can be readily pared back to a basic name, but this method is fine for the occasional odd case.

Too easy! To give you an idea of what we think good Ragewiki docs look like:

• What servers does this apply to?
• Summarise what the nagios check is for (one sentence!)
• What’s the impact of a failure? Customer visible? Websites are down? Etc.
• A short procedure on how to confirm the notification and diagnose it further
• A procedure on how to fix it

That’s it; the page should only be a couple of screens long at the most. If you can’t include all the necessary information, it’s best to put it on a separate and link to it. We specifically don’t include information about How It Works because it detracts from fixing problems faster.

Ragewiki works great for us, so we’d be interested in hearing your thoughts and comments. It’d also be cool to know if other people have reached the same goal, but in a different way.

$USER1$/check_disk -w 5G -c 1G -p /data/foo

If you don’t actually test that this works (by artificially filling your disk and seeing what happens), you may be dismayed to find that you only get alerted when the disk has 5MB of free disk space. Why is this?

Because Nagios, despite the fact that nobody has sweated the megabytes for about a gazillion years, doesn’t support ‘G’ as a suffix for thresholds. Oh, it’ll make a good show of pretending — after all, the output formatting options have ‘GB’ as an option — but nope, for your thresholds it’s “5000M” all the way.

ROCK ON!

The end goal of all this monitoring is to ensure that the services on the servers we run are always working.

We can take this a step further though, rather than just monitor the components of the server that are required to keep the websites running, we can monitor in quite detailed ways many of the components of the websites themselves.

At the end of the day, having a monitoring system tell you that a server is healthy and all of the applications are working only goes so far. Ultimately what’s important (to our clients) is that the website is behaving the way that they expect it to.

Since we don’t build any of the websites that we host unfortunately we can’t put in place systems to monitor the innards of an application. To do so requires an intimate knowledge of how the application was built. We do however have a very powerful monitoring system and if the developers of the websites put the hooks into their code, we can monitor these hooks so that both Anchor and the developers can be alerted to the problems.

With these hooks in place, in many cases Anchor will be able to fix the problems, but if we can’t at least we notify the client that there’s a problem (even if it is at 2am in the morning).

In our world, the more monitoring we put in place the greater the uptime of services and the happier our clients are. On this one though, we need our clients’ help. For all Anchor customers on a Fully Managed support pack we do our side of the monitoring free of charge, and for everyone else we can do anything – for a small fee of course.

To help people understand what can be achieved with web application monitoring along with some implementation ideas, we’ve put together this article on website monitoring.