Containerization is a kind of virtualization that’s surprisingly good at crowding Linux virtual machines onto hosts, including humble home servers. In this post, we’ll look at Linux containers (LXC), a powerful virtualization technology leveraged by Docker, the billion-dollar company that made containerization famous.

Overview

Containerization can be understood in contrast to hardware virtualization, in which a whole machine is emulated, complete with CPU, memory, networking and a hard disk. The latter approach, used by VirtualBox and Amazon’s EC2, conveniently runs any operating system on the virtual hardware, regardless of the host operating system.

Unfortunately, the alchemy of hardware virtualization can consume 10-30% of the host machine’s raw computational power, depending on the task at hand. In terms of density (i.e. maximum virtual machines on given hardware), hardware virtualization is severely limited by the extent to which you want to guarantee minimum performance on each instance. In practice, I’ve found that running more than a few VirtualBox machines on a consumer-grade computer is untenable.

Containers, on the other hand, can virtualize orders of magnitude more machines by deliberately reusing elements of the host operating system. Specifically, LXC (pronounced lex-cee) reuses the host’s kernel, the central core of the operating system, as stated in the introduction to LXC:

The goal of LXC is to create an environment as close as possible to a standard Linux installation but without the need for a separate kernel.

The main catch is that you can only virtualize host-kernel-compatible Linux machines with this technique. The other catch is that security (i.e. preventing container escape) is a bit harder to get right.

Creating a networked container

In setting up LXC on a Ubuntu 14.04.3 LTS (Trusty Tahr) server, I’m referring to the Ubuntu LXC server guide. It’s worth noting that Ubuntu is officially the best OS for the job, as stated in LXC’s Getting Started reference:

Ubuntu is also one of the few (if not only) Linux distributions to come by default with everything that’s needed for safe, unprivileged LXC containers.

The first step is to install LXC:

sudo apt-get install lxc

You could now simply run sudo lxc-create to create a privileged container, in other words, a container owned by the root user. However, this incurs the unnecessary risk of compromising the superuser account, should an attacker escape the container. As discussed in LXC security:

[Privileged containers are] not safe at all and should only be used in environments where unprivileged containers aren’t available and where you would trust your container’s user with root access to the host.

Creation of a so-called unprivileged container (i.e. owned by a regular user), is enabled by subordinate users and subordinate groups. On a newish Ubuntu installation, every user has been assigned a reserved range of subordinate user and group ids:

cat /etc/subuid
cat /etc/subgid

To discover the ranges for the current user:

MY_SUBUID_RANGE=`grep "^$USER:" /etc/subuid | sed 's/\w\+://' | sed 's/:/ /'`
MY_SUBGID_RANGE=`grep "^$USER:" /etc/subgid | sed 's/\w\+://' | sed 's/:/ /'`
echo "My (user=$USER) sub-user range is $MY_SUBUID_RANGE and my sub-group range is $MY_SUBGID_RANGE"

You should see output like:

My (user=ubuntu) sub-user range is 100000 65536 and my sub-group range is 100000 65536

This means that the user ubuntu reserves the right to use 65,536 ids beginning with 100000. I think this means the actual range is 100000-165535, but in any case, it’s a lot of ids. LXC needs a new subordinate user to each new container, as we’ll see later.

You can now write the necessary LXC config file:

mkdir -p ~/.config/lxc

cat << EOF > ~/.config/lxc/default.conf
# Map root uid (0) gid (0) to subordinate ranges
lxc.id_map = u 0 $MY_SUBUID_RANGE
lxc.id_map = g 0 $MY_SUBGID_RANGE

# Network configuration
lxc.network.type = veth
lxc.network.link = lxcbr0
EOF

cat ~/.config/lxc/default.conf

The above maps root (which has a uid 0 and a gid of 0) to the subordinate user and group ranges discovered above. It also instructs LXC to create a virtual network interface using bridged networking.

Next, we need to allow the user to create new virtual network interfaces:

echo "$USER veth lxcbr0 10" | sudo tee -a /etc/lxc/lxc-usernet

Finally, you can create a container:

lxc-create -n ubuntu_example -t download -- --dist ubuntu --release trusty --arch amd64

The -n parameter specifies an arbitrary name for the container. The -t parameter indicates the container template, which refers to one of the scripts found in /usr/share/lxc/templates. At the moment, the choices are:

  • alpine
  • altlinux
  • archlinux
  • busybox
  • centos
  • cirros
  • debian
  • download
  • fedora
  • gentoo
  • openmandriva
  • opensuse
  • oracle
  • plamo
  • sshd
  • ubuntu
  • ubuntu-cloud

By the way, these are non-trivial shell scripts. For example, the ubuntu template, which lives at /usr/share/lxc/templates/lxc-ubuntu, is almost 800 lines long. It actually does a from-scratch installation (using debootstrap), while the download template script (about 600 lines long) grabs and extracts a pre-built disk image from linuxcontainers.com.

Parameters after -- are passed to the template script. In the command above, download is being instructed to get 64-bit Trusty Tahr from a URL that (today) resolves to http://images.linuxcontainers.org/images/ubuntu/trusty/amd64/default/20160105_03:49/.

Latest trusty root image

Incidentally, download is the only template that currently works for an unprivileged user. You can experiment with the others, but you’ll likely get an error like:

This template can’t be used for unprivileged containers. You may want to try the “download” template instead.

Managing containers

Once a container is created, you can recall its name and status by running:

lxc-ls --fancy

The root file system for the container is located at ~/.local/share/lxc/ubuntu_example/rootfs:

du -sh ~/.local/share/lxc/ubuntu_example/rootfs

This shows that the ubuntu installation is 389M.

Now, start the container by running:

lxc-start -n ubuntu_example -d

The -d parameter runs it as a daemon (i.e. in the background). If you don’t do that, your terminal will get taken over by the booting container.

If you get an error like the following, the user probably does not have the correct cgroups.

lxc_container: lxc_start.c: main: 341 The container failed to start.

To correct the issue, just reboot and try again.

sudo shutdown -r now

Once you’ve successfully started the container, run lxc-ls again:

lxc-ls --fancy

The output should include a status like this:

ubuntu_example RUNNING 10.0.3.233

Login to the container by running:

lxc-attach -n ubuntu_example

root@ubuntu_example:/#

From here, you can install software with apt-get or set a password with passwd and so forth. For example, we can install a web server like nginx.

apt-get install nginx

Then return to the host machine:

exit

Visit to the IP address above with a browser or use curl:

curl 10.0.3.233

If everything is working correctly, the result will include:

Welcome to nginx!

As mentioned earlier, LXC is using subordinate users and groups to keep containers isolated. To see this in action, run top on the host computer (substituting 100000 for your first subordinate uid).

top -U 100000

You should see all the processes running on the container (note the newly installed nginx);

Running top for container processes

It’s worth emphasizing that these processes are transparently running on the host, using the host’s kernel, which is why they’re visible on the host’s top. This is different from the typical hardware-virtualized machine scenario, in which the processes are invisible to the host. Press q to quit top.

To stop the container, run:

lxc-stop -n ubuntu_example

To destroy the container and delete all associated data, run:

lxc-destroy -n ubuntu_example

Next steps

The idea of containers, while not exactly new (apparently around since 2000), has skyrocketed in popularity over the past two years due to the success of Docker and fuelled by the maturation of containerization technology like LXC, which reached a major milestone (version 1.0) in early 2014.

The Google Trend for Docker (see the blue line) compared to a couple other virtualization solutions (Xen and KVM), shows rising popularity of the former since 2014. LXC seems to fly a bit under the radar.

Docker popularity skyrocketing since 2013

Docker, a brilliant platform that makes it convenient to deploy software via containers, is worth a detailed exploration another time. Meanwhile, here’s a good summary of containerization and Docker and a great StackOverflow answer about how containers are different from “normal” virtual machines.

Finally, if you’re concerned about container security, here’s some additional reading: