This is going to be a contentious one, but here at Anchor we think automatically applying updates to servers is a Good Thing. It’s definitely not for everyone, but in an environment like ours with hundreds of managed servers it’s the only way you’re going to get things done and get any sleep at night.

Sysadmin of note Tom Limoncelli advocates rolling out updates to progressively more machines with prior testing beforehand to mitigate the scope of potential problems (it’s called “one, some, many”). It’s certainly a good strategy for a large number of homogenous computers, but what we’re talking about here is a bit smaller-scale.

Specifically, we have customers with servers that we never touch, we call this Anchor Monitor. These customers often have particular environments that they’re better off supporting themselves, so we monitor the machine to ensure it’s still on the network, and leave it at that. Unfortunately they’re not always kept up to date, so one of the more recent improvements to our process has been to enable automatic updating by default – it’s up to the customer if they want to change this once it’s handed over to them.

We’ve written this into a short procedure if you’re interested. It applies directly to Debian and Redhat distributions, but it’s easily portable to other systems. If you run Windows, it’ll already be hassling you every 20min for updates.

Sampling shamelessly stolen from four fine folk, for your amusement.

We love vi, everyone at Anchor uses it.
‘Cause it’s better than emacs, yo.

.

AWW SNAP, BRO! WE JUST IMPROVED YO GESTURE!!

.

We heartily endorse this event or product!

Anchr 2.0 makes you want to reach out and touch it; hold it; feel it. Your Anchr 2.0 pulsates with a reassuring rhythm, like that of a heart, but made of silicone instead of striated cardiac muscle.

Anchr 2.0 responds.. it is alive. If you listen carefully you can hear its machinations, at speeds beyond the limits of human ken. Don’t Panic – this is normal, but a helpful voice is always close by when you need it.

Anchr 2.0 is not made, but created. Observe its perfect finish and seamless form. The dull blue glow of security, punctuated by the cerice of backups. Anchr 2.0 fits snugly in the hands. Firm, but also yielding, you cannot discern the boundary; that is the sensation of redundancy. It is comforting.

Anchr 2.0 is communal, it is shared. But! A duality of nature: There is one, but there are also many. That is your Anchr 2.0; there are many like it, but that one is yours.

Anchr 2.0 is… everything you love about webhosting, with less crap

Our current rdiff and amanda backup server, KRAKEN, is almost full, so it was time to order a new one. After much wrangling, we finally received LEVIATHAN this morning.

LEVIATHAN is, I assure you, teh hardk0rez - dual xeon 5500-series, 6gb RAM and 12TB usable storage in RAID-10

I was pushing for PHYREXIAN DREADNOUGHT personally, but LEVIATHAN is acceptable too; the upkeep effort of backup servers is pretty high after all.

This is, of course, a fantastic idea:
http://en.gentoo-wiki.com/wiki/Using_Graphics_Card_Memory_as_Swap

Anchor loves to stay abreast of the latest performance options. As such, we’re proud to announce a new range of upgrade options for our dedicated server customers that demand the absolute best in performance for their customers.

It makes sense, really. The best our current systems offer is puny DDR2 memory. Just think of what you could do with several gig of GDDR5. That’s right, FIVE! We’re now offering upgrade options with Geforce 320 and Geforce 340 cards. If you order one of our higher-specced (2RU) dedicated servers, you can have two of these puppies strapped together for insane amounts of swappiness.

Stay tuned for more news on how we’re rolling out ButterFS, phase-change cooling, overvolted Core2 Quad servers, and mass-scale SSD RAID-0 arrays for database optimisation.

I got woken up by a SMS for low diskspace the other night on one of our customer’s servers. Okay, so that’s a lie, I never sleep, but the SMS is real.

Oh great, they’re making whoopie on their mailing lists again and making some stupidly huge logfile.

Little did I know just how huge that file was. How about 735gb huge, in the space of 12hrs? This customer is already a bit of an oddball, what with 1.4TiB of usable space in their server. “Oh that’s nothing”, you say. Sure, I’ve got a few TiB of kitten pictures on my machine at home, just like you, but to put things in perspective: 300GiB of space would be “big” for most Anchor customers. SCSI disks cost about $1.70/Gb, compared to about 10c/Gb for SATA.

There was no mailout. No big processing job, and no flood of activity. With a little digging I was able to nail it down to an apache errorlog file. That was a surprise, except for the PHP errors all throughout – some things never change.

[Fri Oct 02 02:39:57 2009] [error] [client 63.82.71.139] PHP Warning: fgets(): supplied
argument is not a valid stream resource in /home/wright/public_html/script.php on
line 15, referer: XXX

Nice work there, guys. You need to learn to check your return values from failure-prone functions.

Strangely, there were no actual active connections, but the process list showed two apache processes going balls to the wall, writing the same error message to the log file ad infinitum. By my reckoning that was over 9000 lines per second – nothing a quick service-restart couldn’t fix, thankfully.

And to actually fix the problem? It’s tempting to dump the file, but we don’t like doing that; it’s just a bit too cowboy for us. I settled for a forced logrotate run, taking about 4hrs and squishing it down to just 4.3GiB – Crisis (and sleep) Averted.

And this is why we co-locate in Globalswitch, a top-tier facility with floors that AREN’T MADE OF BALSA WOOD.

Racks are pretty heavy, sure, but they totally wtfpwned those tables there

We’re gonna need another drinks fridge.

The third refrigeration unit has been ordered, ETA next week

I came across this a little while ago; it’s a good little presentation with some interesting points I’d not considered before.

http://www.slideshare.net/techdude/how-to-kill-mysql-performance

If you’re an Anchor customer, I should point out that the ARCHIVE storage engine isn’t available in Redhat’s version of MySQL, which is a damned nuisance.

************************************************************
PRESS RELEASE
FOR IMMEDIATE DISTRIBUTION
************************************************************

ANCHOR SYSTEMS SYSADMINS TO SEEK GUINNESS VERIFICATION
AS "MOST-WIRED MOFOS ON THA PLANET"