The btrfs backup experiment

By April 26, 2013 Technical 4 Comments

Today we’re talking about our experience with btrfs, the next-gen Linux filesystem. btrfs has been maturing rapidly over the last few years and offers many compelling features for modern systems, so we’ve been putting it through its paces on some of our backup servers.

How does it stack up? Read on!

We chose to test btrfs on backup servers because they can make good use of the features on offer, and the threat-level of data loss is low. For our backups, the biggest benefit comes from copy-on-write and atomic snapshots.

At Anchor we use a modified version of Dirvish with support for btrfs: instead of hardlinking directories to provide historical snapshots we just use btrfs’ snapshot facility, which is very quick. Expiring old snapshots is similarly quick – it’s a btrfs operation instead of traversing a filesystem tree.

In general btrfs has been a very positive experience. We’re looking forward to btrfs getting dedupe support in future, something that ZFS already does, which could pay off massively in our environment.

However, in recent weeks it looks like we’ve reached a sort of “critical” mass and a few near-showstoppers have cropped up.

Each backup server hosts around 100-150 backup clients. Every night at midnight they swing into action, creating dozens of snapshots at once and hammering the network for all it’s worth. It’s not perfectly reliable, but we’ve observed hung tasks coincident to snapshotting with a fair degree of regularity. When this happens it blocks I/O to the filesystem, which makes for decidedly ineffective backups.

Quota groups (qgroups) are a relatively new feature, added about six months ago in version 3.6 of the kernel. As well as enabling policy-based usage restrictions, quota groups allow for detailed usage reporting, which is very useful for the usage-based billing that we do. We believe we’ve stumbled upon a bug in the qgroups code that causes CPU soft lockups.

CPU soft lockups are basically unrecoverable, so we’re forced to reboot the system. This has a knock-on effect that, as best we can tell, corrupts btrfs’ free space cache, requiring a time-intensive rebuild after the reboot (over an hour on our 16TB filesystems). Failure to do so results in an error some hours later, with the problem being detected and forcing the filesystem into read-only mode for safety. We haven’t nailed the problem to the qgroups code with absolute certainty, but our investigations are pointing in that direction.

The final lesson for today relates to full filesystems: you never, ever want to fill up a btrfs filesystem. Normal filesystems (eg. ext3) behave fairly predictably when they fill up or get close to capacity. Your system might behave erratically, but by and large it’s easy enough to fix.

In the one instance that we filled up a btrfs filesystem due to some misplaced rsync options, it slowed everything down to the point of being unusable. It wasn’t practical to diagnose the exact reason for the slowdown, but it suffices to say that if you can’t even navigate the filesystem to fix the error then it’s a big problem. Avoiding a recurrence was actually a key motivator for enabling qgroups when they appeared, but it didn’t quite go to plan.

Summary: it’s not all doom and gloom

This probably all sounds very critical, but we think the reality is actually quite positive:

  • Staff have been using btrfs on workstations and personal servers, and it works great
  • We’ve used btrfs in conjunction with Ceph and had no problems
  • Zero incidents of data loss or corruption in our testing so far
  • btrfs has actually detected single-bit corruption in an underlying hardware RAID volume – that’s a win for data integrity!
  • btrfs has fairly gracefully managed thousands of snapshots on each of our backup servers

In the short-term we’ll be pushing most of our systems back to ext4 with hardlinking, while keeping an eye on btrfs and zfs for backups.

Development is very active, with new features and bugfixes appearing in every kernel release. Many see btrfs as the future for Linux beyond ext4, and we think it’s worth trying if you haven’t already – it probably won’t be too long until it’s the default recommended filesystem for a distro like Fedora or Ubuntu.

4 Comments

  • David Goodwin says:

    Thanks for the above.

    (This would have been a lot more readable if you theme wasn’t grey text on a grey background, and if the font size wasn’t so small.)

  • Bart Thomas says:

    font size fixed; grey background has to stay for the time being though ;-)

  • Benjamin Skapur says:

    i like te way is producing cover of. same theme…is going ti be ok.i had recomended to visit us at http://hitpilot.com

  • lsatenstein says:

    I began usng btrfs about the time of your article (1.5 years ago), as my desktop filesystem (Fedora 18). btrfs was great, for a few months, but then failed, and the file checking software was unsuccessful at recovering. Some reporting also showed btrfs orphans (unreachable blocks). A filecheck utility should have been able to return the orphans to the available pool.
    With each release of Fedora (F19,F20), btrfs improved, but the problems persisted. With small 50gig partitions, as partition use approched 80% utilisation, performance dropped. Using btrfs on a desktop system with very frequent home directory file creates /deletes and 80% file partition utilization is a reason I shy away from its use.
    On 250gig or larger partition sizes, over a 6 month period, I experienced zero problems.

Leave a Reply

Ready to talk business? Send us a note.