Review: The best Linux distros for Docker and containers

Over the past six months I have reviewed five minimal Linux distributions that are optimized for running containers: Alpine Linux, CoreOS Container Linux, RancherOS, Red Hat Atomic Host, and VMware Photon OS. Generically known as “container operating systems,” these stripped down, purpose built Linux distributions are not the only way to run containers in production, but they provide a base that does not waste resources on anything besides container support.

The state of the industry with container deployment systems is very much like the early days of Linux distributions. You have one key element, in this case the Docker container, that is surrounded by a number of competing ecosystem components. Just as the traditional Linux distros bundled different package managers, desktop environments, system utilities, services, and apps, most container distributions mix and match various components to create what they consider an optimum solution. Take for example distributed configuration and service discovery. There are several solutions for this such as Etcd, Consul, and ZooKeeper.

Each distribution takes a different approach to what to include in the stack. On one extreme there are distributions designed to support only the higher levels of the stack, such as CoreOS Container Linux and Red Hat Project Atomic. So much of the functionality has been put into the proprietary management layer that there is little hope of using the OS for anything else. Other distros, such as RancherOS and VMware Photon OS, provide greater flexibility, supporting multiple ecosystem components and orchestration systems. These give administrators a good way to experiment and potentially avoid vendor lock-in.

Alpine Linux

Alpine Linux, the underlying operating system for many official Docker images, is a great choice for the task. A mere five megabytes in size, Alpine Linux is on the opposite end of the spectrum from the full-fat Linux distributions of a few years ago, which were competing with Solaris and intended to run on massive hardware systems. This new breed of Linux is designed to run on embedded hardware, and consume minimal resources, making it a perfect choice for containers.

Alpine Linux’s legacy as an embedded systems OS was evident during my review. Many of the configuration options default to embedded systems, and the documentation in many areas was sparse, or non-existent. Clearly a system designed for and primarily used by hackers, Alpine Linux will require companies to clear a few hurdles to build their applications.

Production deployments of Alpine Linux will enjoy several advantages including rapid startup, a minimalist footprint, and a secure-by-default stance not found elsewhere. Everything from the system binaries to the C libraries is designed for small, fast, and secure deployments. There is no bloat here.

Administration of Alpine Linux is different than traditional Linux systems, and will take some time to learn. Installation and package management is unique even among the container operating systems, though well thought out. If your development shop has above-average developers who are willing to make the investment, Alpine Linux will provide a solid, stable, secure base for applications for a long time to come.

CoreOS Container Linux

The CoreOS container stack draws on Etcd for distributed storage and service discovery, Flannel for networking, and Kubernetes for container orchestration, and supports its own flavor of container format, rkt (Rocket), in addition to Docker. Rocket was an attempt at a competing container format to address shortcomings in the Docker format circa 2015, but with those deficiencies addressed, Rocket hasn’t seen much uptake.

CoreOS, like Project Atomic, is not afraid of diverging radically from traditional Linux. Like Red Hat’s container OS, CoreOS Container Linux creates a mostly immutable filesystem, but does so with a disk partitioning system inspired by Google’s Chromium OS. What this does is preserve the old filesystem on a partition, meaning that rollbacks are always safe and fast.

Although the documentation is fairly good and comprehensive, I found installation somewhat cumbersome, involving a two-step process to derive the configuration file. Once installed however, CoreOS offers continuous, “no downtime” upgrades, a feature made possible by its unique disk partition layout. CoreOS has done a lot of work here, and the company offers various maintenance options that will suit most any organization, including the ability to opt out.

CoreOS, somewhat like Project Atomic, is an all-or-nothing decision. Picking apart the pieces and using the underlying OS to build your own container infrastructure is not really an option due to all of the architectural design decisions baked into the platform. If you are willing to embrace those decisions and pay for CoreOS’s commercial Kubernetes distribution, Tectonic, no doubt you could perform some seriously heavy lifting.

Rancher Labs RancherOS

Rancher Labs’ RancherOS is a Linux operating system composed entirely of containers. Even the init process (PID 1) is a Docker container. This means that there is no need for a package management system. OS upgrades (and downgrades) are managed with Docker, just like any other container.

Although this approach is equally as radical as the architectural decisions made in some of the other distributions, such as Project Atomic and CoreOS, the result is a surprising simplicity. Although learning any entirely new system administration seems daunting at first, you have to know Docker to manage containers anyway, so why not use the same system for both?

RancherOS seems to be maturing rapidly. In my review I found the documentation a bit lacking, but any developer or administrator familiar with Docker containers will already know most of the system. RancherOS has a small footprint (20MB) and makes efficient use of resources. Although the lines between Rancher the container management system and Rancher the OS are somewhat blurred, the container management system is free and open source, so there is no reason to attempt to roll your own. Organizations that need access to source code should look no further.

The Rancher platform supports nearly the entire ecosystem of tools for container management, including Docker Swarm, Kubernetes, and Mesos, and it is advancing rapidly. Although radically different than traditional UNIX, RancherOS seems to have adhered more closely to the basic UNIX philosophy than the other container OS distributions: Simple tools operating together in an elegant way.

Red Hat Project Atomic

Red Hat’s Project Atomic is firmly in the Kubernetes camp of service orchestration. Typically this type of deployment is geared toward large-scale, highly available scenarios. The downside is that, essentially, you have to “do as you are told” and architect the application by convention.

In the box you will find Flannel for networking, Etcd for distributed key-value storage, and OSTree for host management. OSTree is a relatively new way to deploy an OS at scale in a reliable and distributed fashion. Atomic combines OSTree with a new RPM package manager to create RPM-OSTree, which delivers a mostly immutable filesystem.

I found Project Atomic to be a challenge. It is very ambitious and moving fast, with many moving parts. RHEL, CentOS, Fedora, SELinux, Systemd, a custom “Docker” command-set to control the underlying host… they are all in the mix, and the documentation is disorganized and confusing. Furthermore, in my small cluster, lacking Chef, Salt, or Puppet, I had to manually configure each node.

The bottom line is that Project Atomic still needs some time to bake. If the vision is realized, then it may become the standard of the future — though not for data centers with hundreds of nodes but thousands or tens of thousands. In this respect the vision seems closer to that of Mesos than a typical container deployment system. If your company is living and breathing the Red Hat ecosystem and plans to stay there, then Project Atomic is probably worth getting started with.

VMware Photon OS

VMware’s Photon OS is a minimal Linux container host designed to have a small footprint and tuned for VMware hypervisors. As such, Photon OS only runs in virtual environments; deployment on physical hardware is not possible. Photon OS is customized to make container management easier, but not as radically as Atomic or CoreOS. Photon OS is more of an evolutionary step.

Based on my testing, Photo OS lives up to its promises in the VMware virtual environment. (Photon OS can also run on other hypervisors as well as the Google and Amazon clouds.) Because Photon OS can make assumptions about the hardware (virtual), the ecosystem looks very much like standard Linux, making the learning curve less steep. Networking and storage are Systemd compatible, and there are a range of options documented for container networking. The documentation for Photon OS may be the best among the products reviewed.

VMware is taking the lead in productionizing containers for traditional environments, and this makes sense when you think about it. How often have you been asked to describe the difference between a container and a VM? With Photon OS, there soon won’t be any difference: Containers will just be a lightweight VM, deployed and managed with the same tools. Photon OS supports virtually every major piece of the container ecosystem: Docker and Rocket containers, Docker Swarm, Kubernetes, Mesos, Google Cloud Engine, Amazon EC2, and more.

Of all the distributions I reviewed, VMware Photon OS seems both the most visionary and currently the most complete and usable. If you are a VMware shop exploring containerization, I would not think of considering anything else. If you are not a VMware shop, Photon OS is still worth a good look.

Comparing container operating systems

Alpine Linux underpins most every Docker image out there. Perfect for embedded applications, Alpine Linux should not be thought of as a way to run containers. Instead, in a way, Alpine Linux is the container. Developers familiar with building applications on Alpine Linux will write better container applications.

CoreOS, one of the early container operating systems, adopts the Google technology stack. It offers a reliable, though opinionated, way of managing container infrastructure. While CoreOS makes many of the components available as open source, the complexity of learning such a large stack effectively means users will need to purchase the proprietary Techtonic orchestration system for production deployments. If money is no object, and you need to deploy Google-sized applications, CoreOS a logical choice.

RancherOS is pure containers. If you are going to roll your own container infrastructure, or you want a minimal container management stack, RancherOS is the place to start. With open source orchestration and scheduling tools like Docker Swarm, Kubernetes, and Mesos all freely available, the Rancher stack will appeal to open source oriented, do-it-yourself companies.

Red Hat’s Project Atomic is an umbrella project that is re-architecting the way companies deploy infrastructure. This ambitious project could change the way companies think about application deployment, but the road is a long one. Project Atomic is best suited to early adopters with a large existing investment in Red Hat technologies.

VMware’s Photon OS brings that vendor’s virtual machine management technology and experience to containers. Photon OS is deployed as a virtual machine and is managed with traditional VM tools. VMware, perhaps seeing the writing on the wall for traditional VMs, has wholeheartedly embraced container technology and is rapidly advancing the state of the art. If you are a VMware shop now, you will be hard-pressed to find a better container platform than Photon OS.

Read the container Linux reviews: 

• Review: Alpine Linux is made for Docker
• CoreOS review: Linux for containers and Kubernetes
• RancherOS: A simpler Linux for Docker lovers
• Review: Red Hat does Docker the hard way
• Review: VMware’s Photon OS shines for Docker containers