Containerization is a kind of virtualization that’s surprisingly good at crowding Linux virtual machines onto hosts, including humble home servers. In this post, we’ll look at Linux containers (LXC), a powerful virtualization technology leveraged by Docker, the billion-dollar company that made containerization famous.
Containerization can be understood in contrast to hardware virtualization, in which a whole machine is emulated, complete with CPU, memory, networking and a hard disk. The latter approach, used by VirtualBox and Amazon’s EC2, conveniently runs any operating system on the virtual hardware, regardless of the host operating system.
Unfortunately, the alchemy of hardware virtualization can consume 10-30% of the host machine’s raw computational power, depending on the task at hand. In terms of density (i.e. maximum virtual machines on given hardware), hardware virtualization is severely limited by the extent to which you want to guarantee minimum performance on each instance. In practice, I’ve found that running more than a few VirtualBox machines on a consumer-grade computer is untenable.
Containers, on the other hand, can virtualize orders of magnitude more machines by deliberately reusing elements of the host operating system. Specifically, LXC (pronounced lex-cee) reuses the host’s kernel, the central core of the operating system, as stated in the introduction to LXC:
The goal of LXC is to create an environment as close as possible to a standard Linux installation but without the need for a separate kernel.
The main catch is that you can only virtualize host-kernel-compatible Linux machines with this technique. The other catch is that security (i.e. preventing container escape) is a bit harder to get right.
Creating a networked container
In setting up LXC on a Ubuntu 14.04.3 LTS (Trusty Tahr) server, I’m referring to the Ubuntu LXC server guide. It’s worth noting that Ubuntu is officially the best OS for the job, as stated in LXC’s Getting Started reference:
Ubuntu is also one of the few (if not only) Linux distributions to come by default with everything that’s needed for safe, unprivileged LXC containers.
The first step is to install LXC:
sudo apt-get install lxc
You could now simply run sudo lxc-create to create a privileged container, in other words, a container owned by the root user. However, this incurs the unnecessary risk of compromising the superuser account, should an attacker escape the container. As discussed in LXC security:
[Privileged containers are] not safe at all and should only be used in environments where unprivileged containers aren’t available and where you would trust your container’s user with root access to the host.
Creation of a so-called unprivileged container (i.e. owned by a regular user), is enabled by subordinate users and subordinate groups. On a newish Ubuntu installation, every user has been assigned a reserved range of subordinate user and group ids:
cat /etc/subuid cat /etc/subgid
To discover the ranges for the current user:
MY_SUBUID_RANGE=`grep "^$USER:" /etc/subuid | sed 's/\w\+://' | sed 's/:/ /'` MY_SUBGID_RANGE=`grep "^$USER:" /etc/subgid | sed 's/\w\+://' | sed 's/:/ /'` echo "My (user=$USER) sub-user range is $MY_SUBUID_RANGE and my sub-group range is $MY_SUBGID_RANGE"
You should see output like:
My (user=ubuntu) sub-user range is 100000 65536 and my sub-group range is 100000 65536
This means that the user ubuntu reserves the right to use 65,536 ids beginning with
100000. I think this means the actual range is 100000-165535, but in any case, it’s a lot of ids. LXC needs a new subordinate user to each new container, as we’ll see later.
You can now write the necessary LXC config file:
mkdir -p ~/.config/lxc cat << EOF > ~/.config/lxc/default.conf # Map root uid (0) gid (0) to subordinate ranges lxc.id_map = u 0 $MY_SUBUID_RANGE lxc.id_map = g 0 $MY_SUBGID_RANGE # Network configuration lxc.network.type = veth lxc.network.link = lxcbr0 EOF cat ~/.config/lxc/default.conf
The above maps root (which has a uid
0 and a gid of
0) to the subordinate user and group ranges discovered above. It also instructs LXC to create a virtual network interface using bridged networking.
Next, we need to allow the user to create new virtual network interfaces:
echo "$USER veth lxcbr0 10" | sudo tee -a /etc/lxc/lxc-usernet
Finally, you can create a container:
lxc-create -n ubuntu_example -t download -- --dist ubuntu --release trusty --arch amd64
-n parameter specifies an arbitrary name for the container. The
-t parameter indicates the container template, which refers to one of the scripts found in
/usr/share/lxc/templates. At the moment, the choices are:
By the way, these are non-trivial shell scripts. For example, the ubuntu template, which lives at
/usr/share/lxc/templates/lxc-ubuntu, is almost 800 lines long. It actually does a from-scratch installation (using debootstrap), while the download template script (about 600 lines long) grabs and extracts a pre-built disk image from linuxcontainers.com.
-- are passed to the template script. In the command above, download is being instructed to get 64-bit Trusty Tahr from a URL that (today) resolves to
Incidentally, download is the only template that currently works for an unprivileged user. You can experiment with the others, but you’ll likely get an error like:
This template can’t be used for unprivileged containers. You may want to try the “download” template instead.
Once a container is created, you can recall its name and status by running:
The root file system for the container is located at
du -sh ~/.local/share/lxc/ubuntu_example/rootfs
This shows that the ubuntu installation is 389M.
Now, start the container by running:
lxc-start -n ubuntu_example -d
-d parameter runs it as a daemon (i.e. in the background). If you don’t do that, your terminal will get taken over by the booting container.
If you get an error like the following, the user probably does not have the correct cgroups.
lxc_container: lxc_start.c: main: 341 The container failed to start.
To correct the issue, just reboot and try again.
sudo shutdown -r now
Once you’ve successfully started the container, run lxc-ls again:
The output should include a status like this:
ubuntu_example RUNNING 10.0.3.233
Login to the container by running:
lxc-attach -n ubuntu_example
From here, you can install software with
apt-get or set a password with
passwd and so forth. For example, we can install a web server like nginx.
apt-get install nginx
Then return to the host machine:
Visit to the IP address above with a browser or use
If everything is working correctly, the result will include:
Welcome to nginx!
As mentioned earlier, LXC is using subordinate users and groups to keep containers isolated. To see this in action, run top on the host computer (substituting
100000 for your first subordinate uid).
top -U 100000
You should see all the processes running on the container (note the newly installed nginx);
It’s worth emphasizing that these processes are transparently running on the host, using the host’s kernel, which is why they’re visible on the host’s top. This is different from the typical hardware-virtualized machine scenario, in which the processes are invisible to the host. Press
q to quit top.
To stop the container, run:
lxc-stop -n ubuntu_example
To destroy the container and delete all associated data, run:
lxc-destroy -n ubuntu_example
The idea of containers, while not exactly new (apparently around since 2000), has skyrocketed in popularity over the past two years due to the success of Docker and fuelled by the maturation of containerization technology like LXC, which reached a major milestone (version 1.0) in early 2014.
The Google Trend for Docker (see the blue line) compared to a couple other virtualization solutions (Xen and KVM), shows rising popularity of the former since 2014. LXC seems to fly a bit under the radar.
Docker, a brilliant platform that makes it convenient to deploy software via containers, is worth a detailed exploration another time. Meanwhile, here’s a good summary of containerization and Docker and a great StackOverflow answer about how containers are different from “normal” virtual machines.
Finally, if you’re concerned about container security, here’s some additional reading: