Tag

time Archives - AWS Managed Services by Anchor

A most unusual temporal issue

By | Technical | No Comments

Most of us have suffered time-related woes, dealing with servers that just can’t keep their clock straight even with NTP. Well we’ve just come across a new one, and NTP isn’t going to save the day – what happens when you have multiple misbehaving clocks? We first noticed the problem when reloading the firewall on the host, a KVM server that had been virtualised from hardware. We use filtergen for ruleset generation, and one of its postreload scripts restarts fail2ban, an anti-bruteforce tool. One part of fail2ban’s initscript makes a call to sleep 1 to give things time to settle. We noticed that the invocation of sleep was taking more than a second, much more. Something wasn’t right and the automatic 15sec rollback was kicking in, having not received explicit…

Read More

Hopping mad: bitten by the leap second bug

By | Technical | No Comments

It’s been an exciting weekend for those running Linux systems, with many popular websites taken offline as a result of a kernel bug relating to the addition of a leap second at the end of June. This follows hot on the heels of an outage that saw many Instagram users tragically unable to upload pictures of their breakfast. There’s no shortage of stories in the media about the problem if you care to look around, but we were rather disappointed by the lack of coverage with much substance. We’ve also not yet found an explanation that can be understood by someone who isn’t intimately familiar with the internals of the Linux kernel. So we put on our detective hat and started looking. Update: if you are familiar with the innards…

Read More

The long running process dilemma

By | Technical | No Comments

Recently we had a client on one of our webservers doing something which we consider to be bad form, inconsiderate and downright annoying – running scripts from cron every five minutes which take longer than five minutes to run. The end result? Processes that pile up, consuming resources and frequently inconveniencing other customers (if it is running on a shared webserver). If the scripts in question are competing for resources, you can frequently end up in a dead-lock or live-lock situation as well. It seems some developers are blissfully unaware of how to write their scripts with enough safety features built in. Fortunately, we at Anchor have encountered these situations enough times to know how to best combat them. In this scenario our client was using Perl in scripts which…

Read More