Tag

log Archives - AWS Managed Services by Anchor

Keeping your kernel output safe

By | Technical | No Comments

Keeping logs of the operation of your system(s) is really important; when something goes wrong in the middle of the night, a good log can give you all the information you need to diagnose and fix the problem before it happens again. One area of your system that’s quite crucial to keep, but which is often forgotten, is your kernel’s dmesg output. This is all of the messages that come direct from the kernel, from “filesystem mounted” to “Aiee! Penguins on the SCSI bus!” or “lp0: on fire”. While the former isn’t so important to keep for posterity, when the kernel crashes, you really do want to capture that output, but you can’t use your system’s normal logging because when the kernel dies, it usually takes userland processes like syslogd…

Read More

Root cause analysis using shell history timestamping

By | Technical | No Comments

Some of the problem-solving we do has a forensic component (not in the legal sense). When something breaks down, we perform what’s known as Root Cause Analysis. It’s well and good to fix the symptoms so things are running again, but it’s more important to fix the problem so it doesn’t happen again. Often enough, we’ll login to a user’s account and check out what they’ve run recently. They might have edited a config file, removed some necessary wrapper scripts (this happens on our shared hosting servers), or blindly duplicated something that really shouldn’t be copied. Whatever the case, these are good places to start looking when you need to fix something.

Read More