More on BTRFS, CoreOS Linux, and LXC

First off, one thing I learned was that CoreOS tried adopting BTRFS last year, and too many people tried it, and found issues (like ‘out of space’ when there’s only 30% used in df). CoreOS has since switched to OverlayFS, a union filesystem that runs on top of two EXT4 filesystems. So cool, and it solves my issue of filesystem compatibility, since it makes LVM available. Everything is compatible with EXT4 file systems.

Okay, CoreOS. CoreOS Linux is a new-ish distro, which was originally designed to run Docker containers, but has since defined their own, open source, container image format, rkt. rkt is more flexible than Docker, making Enterprise solutions much easier. CoreOS is designed to be deployed to bare metal (or any hypervisor you like – Amazon, Google, Microsoft – pick your cloud), and join your cluster instantly. The more I learn about CoreOS and fleet, the more I want to run a cluster myself.

CoreOS only supports x86_64, so no Raspberry Pi 2 cluster without cross compiling everything, which is possible, but a royal pain in the arse.  LXC works fine, though, so Ubuntu 14.04 LTS on Pi 2 is totally doable.

LXC development is being actively funded by Ubuntu’s Cananical, and seems to be trying to compete with CoreOS, running Docker containers as easily as creating any custom machine you like.

LXC also supports overlayfs, so you can base a number of containers off a read-only version of your favorite version of rootfs as the “lower” filesystem, and each with it’s own writable “upper” LVM based filesystem on top.

CoreOS seems like what we might use at work, if we were a modern company, I mean. I want to know more, I wish I could influence our future somehow, but oh well. I’m sure we’ll get on board in another 5-10 years.

Ubuntu Linux and LXC is what I want to use on my small office/home office server right now.

BTRFS is pretty cool

This weekends nerd time (Thanks Kim) had me diving into BTRFS, a new-ish combination volume manager/filesystem for Linux.  SLES has gotten behind it big, it’s their default for the root filesystem now, instead of LVM, which is a problem for us at work, because Splunk doesn’t support BTRFS for the the filesystem where indexes are stored, so can’t benefit from any of it’s features, and BTRFS doesn’t support any kind of block device access, just BTRFS filesystems.

LXC (Linux Containers) have built-in support for BTRFS as a backing store, and BTRFS snapshots make it really easy to deploy nearly identical containers (all with the same software installed) in an instant.  LXC creates subvolumes for each container.  Combine that with puppet recipes to automatically self-configure each app server container image as it’s launched, based on its hostname, and there’s your disaster recovery plan covered.  Upgrading container software ought to be just as easy as building a new base image, shutting down containers based on the old image, renaming or deleting their subvolumes, then building new containers based on the new image.

BTRFS is combination volume manager, snapshot manager, and filesystem.  It’s not like LVM, you can’t create block devices to put any other types of filesystems underneath it, except maybe in a loopback fs.  BTRFS is creating “subvolumes” under the physical filesystem, which can have separate quota features enabled, but they’re all BTRFS, all the way down.

BTRFS is a local filesystem, not a cluster shared or distributed filesystem.  If you’re trying to share one filesystem among a cluster of containers running on multiple hosts, it’ll have to be something else.

I don’t think the Splunk Universal Forwarder would have any problem running under BTRFS or LXC, since it’s not indexing anything, it should not suffer the same restrictions as it’s general purpose indexer big brother.

The general consensus (which I agree with) seems to be that BTRFS is cool and stable, but only SuSE Linux Enterprise Server is using it for their root filesystem by default at this time.  Most of the free distros include support, but at what bugfix level, is the big question.  Other distros are being more cautious, but I encourage you experiment in the lab.  I used it on Ubuntu 15.04 minimal server Linux, with LXC installed.

If your need is a bunch of LXC containers that will run fine on BTRFS filesystems, I think you may find BTRFS may be a big win.  I’ve heard it’s reported to be big on filesystem performance, but we’ll see.  I’ve also seen reports that it’s known to be really bad for filesystems which contain lots of small files.  I’d like to see more published benchmarks, especially from SuSE, since they’ve committed to it so hard.

Linux Containers are a GAME CHANGER

If you are an enterprise linux vmware or z/Linux customer, and run lots of Linux virtual machines, you are doing yourself a disservice if you aren’t at least investigating Linux Containers, LXC, and Docker.  The performance gain is significant, it makes the technology really hard to ignore. My employer uses mainframes running z/Linux guests, and it’s a significant investment as we grow.

Imagine a different datacenter, where you had an OpenStack cloud of servers, 10 of each JVM to start, but each JVM cloud growing and shrinking with the cloud as usage and load fluctuates throughout the day.  You can run thousands of JVMs on a short row of servers in your datacenter. No longer do you need to run enormous data centers to house all those machines, pay for power for all those machines, pay for people to operate all those machines.

Linux Containers are a really cool, new twist on “virtualization”.  Assuming your workload runs well on Linux, that is.

I really see containers being a game changer in the enterprise security world too. Every server will be able to isolate apps in their own containers, which can have just enough files to run your website, nothing more. Containers can be setup to limit CPU and memory resources, making it easy to grow as Java max heap sizes grow over time.  Instead of hundreds of virtual machines, you’ll have hundreds of containers running your apps, all running natively on the raw metal of your servers.  No virtualization overhead at all.

Containers don’t need every program a minimal linux install gets.  They only need enough to get your app running.  Maybe a shell, some scripts and utilities, some libraries, that’s good enough.  I’m sure Docker has figured out the best way to package a container.  LXC is capable of running a Docker container, or so I’ve read.  I’ve no real world experience with Docker, yet.  Becoming a well accepted packing format, I can’t imagine it being long before app vendors start offering a container version of their apps for download.

LXC Linux Containers

One of my more popular articles on my old website was my investigation into lxc, commands for manipulating Linux containers.  I wrote a few management scripts to make lxc admin a little easier, and deployment a little faster.  I got some nice feedback.

This week, I investigated the current state of LXC, and wow, has it advanced.  The last version I played with was v0.7.5, and now it’s up to 1.0.7 (stable version, devs keep working on the next big version).  All the ideas I had are taken care of, and more.  LVM backing store lets you isolate the containers filesystems, so they’re never mounted on the host.  Templates are a cool idea, a way to install any ditro of Linux from scratch (provided you have the neccessary build tools, such as yum and apt-get).

The container.conf file is still a mess, and there are still precious few working examples on the Internet (but more than there used to be).  Networking is still a mystery to too many people, and all Linux comes with a simple bridges.  You can do VLANs on the linux bridge, I’ve done it before, but it’s not for the newbie.

LXC came out of the box setup with all the containers on their own bridge, lxcbr0, isolated from the outside world, and a NAT/PAT firewall running on the host itself, to give them outbound only access (by default).  Nice idea, especially for a working default config, right out of the box.

I rather like the LVM backing store, but noticing the zfs and btrfs support makes we really want to start using and learning more about them now.  Endless new technology to explore!