This weekends nerd time (Thanks Kim) had me diving into BTRFS, a new-ish combination volume manager/filesystem for Linux. SLES has gotten behind it big, it’s their default for the root filesystem now, instead of LVM, which is a problem for us at work, because Splunk doesn’t support BTRFS for the the filesystem where indexes are stored, so can’t benefit from any of it’s features, and BTRFS doesn’t support any kind of block device access, just BTRFS filesystems.
LXC (Linux Containers) have built-in support for BTRFS as a backing store, and BTRFS snapshots make it really easy to deploy nearly identical containers (all with the same software installed) in an instant. LXC creates subvolumes for each container. Combine that with puppet recipes to automatically self-configure each app server container image as it’s launched, based on its hostname, and there’s your disaster recovery plan covered. Upgrading container software ought to be just as easy as building a new base image, shutting down containers based on the old image, renaming or deleting their subvolumes, then building new containers based on the new image.
BTRFS is combination volume manager, snapshot manager, and filesystem. It’s not like LVM, you can’t create block devices to put any other types of filesystems underneath it, except maybe in a loopback fs. BTRFS is creating “subvolumes” under the physical filesystem, which can have separate quota features enabled, but they’re all BTRFS, all the way down.
BTRFS is a local filesystem, not a cluster shared or distributed filesystem. If you’re trying to share one filesystem among a cluster of containers running on multiple hosts, it’ll have to be something else.
I don’t think the Splunk Universal Forwarder would have any problem running under BTRFS or LXC, since it’s not indexing anything, it should not suffer the same restrictions as it’s general purpose indexer big brother.
The general consensus (which I agree with) seems to be that BTRFS is cool and stable, but only SuSE Linux Enterprise Server is using it for their root filesystem by default at this time. Most of the free distros include support, but at what bugfix level, is the big question. Other distros are being more cautious, but I encourage you experiment in the lab. I used it on Ubuntu 15.04 minimal server Linux, with LXC installed.
If your need is a bunch of LXC containers that will run fine on BTRFS filesystems, I think you may find BTRFS may be a big win. I’ve heard it’s reported to be big on filesystem performance, but we’ll see. I’ve also seen reports that it’s known to be really bad for filesystems which contain lots of small files. I’d like to see more published benchmarks, especially from SuSE, since they’ve committed to it so hard.