Keeping logs of the operation of your system(s) is really important; when something goes wrong in the middle of the night, a good log can give you all the information you need to diagnose and fix the problem before it happens again.
One area of your system that’s quite crucial to keep, but which is often forgotten, is your kernel’s dmesg output. This is all of the messages that come direct from the kernel, from “filesystem mounted” to “Aiee! Penguins on the SCSI bus!” or “lp0: on fire”. While the former isn’t so important to keep for posterity, when the kernel crashes, you really do want to capture that output, but you can’t use your system’s normal logging because when the kernel dies, it usually takes userland processes like syslogd with it.
Enter: the netconsole. This is a nice little kernel module that’s been around for years, but isn’t widely used, probably because (a) nobody knows about it, and (b) it’s not so simple to setup. While this blog post is a feeble attempt to solve (a), Karsten M. Self has done a good job of assisting with (b) in a recent post to the linux-elitists mailing list. I encourage everyone to take a look.