Build Kata Containers from source on x86 and arm64

Kata Containers enable containers to be seamlessly executed in Virtual Machines. Kata Containers are as light and fast as containers and integrate with the container management layers, while also delivering the security advantages of VMs. Kata Containers is the result of merging two existing open source projects: Intel Clear Containers and Hyper runV.

Kata Containers consist of several components. For amd64 machines, binaries are provided through the formal release process. However, for arm64, binary files are not available (just yet).

In this post, we will be going through the steps to build kata containers from source, both for amd64 and arm64 architectures. In follow-up posts, we go through the steps to build and configure QEMU and AWS Firecracker as VMMs for Kata Containers.

Install requirements Link to heading

To build Kata Containers we need to install Rust v1.58.1, Go v1.16.10, Docker and some apt/snap packages. The specific versions may change, so make sure to check the versions database.

Apt/Snap Packages: Link to heading

We need to install gcc, make and yq v3. containerd and runc are installed by the Docker install script, in the following steps.

1sudo apt update && sudo apt upgrade -y
2sudo apt install gcc make snapd -y
3sudo snap install yq --channel=v3/stable

Rust (version 1.58.1): Link to heading

We will use rustup to install and set Rust 1.58.1 as our default toolchain:

1down_dir=$(mktemp -d)
2pushd $down_dir
3wget -q https://static.rust-lang.org/rustup/dist/$(uname -p)-unknown-linux-gnu/rustup-init
4sudo chmod +x rustup-init
5./rustup-init -q -y --default-toolchain 1.58.1
6source $HOME/.cargo/env
7popd
8rm -rf $down_dir

Go (version 1.16.10) Link to heading

We will download the appropriate Go binaries and add them to the PATH environment variable:

1down_dir=$(mktemp -d)
2pushd $down_dir
3wget -q https://go.dev/dl/go1.16.10.linux-$(dpkg --print-architecture).tar.gz
4sudo mkdir -p /usr/local/go1.16
5sudo tar -C /usr/local/go1.16 -xzf go1.16.10.linux-$(dpkg --print-architecture).tar.gz
6echo 'export PATH=$PATH:/usr/local/go1.16/go/bin' >> $HOME/.profile
7source $HOME/.profile
8popd
9rm -rf $down_dir

Docker Link to heading

We will install Docker using the provided convenience script:

 1sudo apt-get remove docker docker-engine docker.io containerd runc -y > /dev/null 2>&1
 2sudo rm -rf /var/lib/docker/
 3down_dir=$(mktemp -d)
 4pushd $down_dir
 5curl -fsSL https://get.docker.com -o get-docker.sh
 6sudo sh get-docker.sh
 7# Optionally add user to docker group to run docker without sudo
 8# sudo usermod -aG docker $USER
 9popd
10rm -rf $down_dir

Build Kata components Link to heading

Build kata-runtime Link to heading

First, we need to set the correct Go environment variables:

1export PATH=$PATH:$(go env GOPATH)/bin && \
2  export GOPATH=$(go env GOPATH) && \
3  export GO111MODULE=off

We will use go get to download kata-containers source code:

1go get -d -u github.com/kata-containers/kata-containers

We are now ready to build the kata-runtime:

1pushd $GOPATH/src/github.com/kata-containers/kata-containers/src/runtime
2export GO111MODULE=on
3export PREFIX=/opt/kata
4make
5popd

To install the binaries to a specific path (say /opt/kata) we need to specify the PREFIX environment variable prior to installing:

1pushd $GOPATH/src/github.com/kata-containers/kata-containers/src/runtime
2export PREFIX=/opt/kata
3sudo -E PATH=$PATH -E PREFIX=$PREFIX make install
4popd

Kata binaries are now installed in /opt/kata/bin and configs are installed in /opt/kata/share/defaults/kata-containers/.

It is recommended you add a symbolic link to /opt/kata/bin/kata-runtime and /opt/kata/bin/containerd-shim-kata-v2 in order for containerd to reach these binaries from the default system PATH.

1sudo ln -s /opt/kata/bin/kata-runtime /usr/local/bin
2sudo ln -s /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin

Create a rootfs & initrd image Link to heading

We can use either a rootfs or initrd image to launch Kata Containers with Qemu. However, AWS Firecracker does not work with initrd images, so we will be using a rootfs image for Kata with Firecracker. If you do not want to use QEMU or QEMU with initrd, you can skip building the initrd image.

Create the rootfs base image:

1export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs
2cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder
3# you may change the distro (in this case we used ubuntu). to get supported distros list, run ./rootfs.sh -l
4script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true ./rootfs.sh ubuntu'

Note for arm64:

We noticed that in some instances the kata-agent compilation failed. A possible workaround was to remove the USE_DOCKER variable. This requires qemu-img command to be available on your system. You can install it with sudo apt install -y qemu-utils.

1export ROOTFS_DIR="${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs"
2sudo rm -rf ${ROOTFS_DIR}
3cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder
4script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes ./rootfs.sh ubuntu'

Build a kata rootfs image:

1cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/image-builder && \
2  script -fec 'sudo -E USE_DOCKER=true -E AGENT_INIT=yes ./image_builder.sh ${ROOTFS_DIR}'

Install the kata rootfs image:

1export PREFIX=/opt/kata
2commit=$(git log --format=%h -1 HEAD) && \
3  date=$(date +%Y-%m-%d-%T.%N%z) && \
4  image="kata-containers-${date}-${commit}" && \
5  sudo install -o root -g root -m 0640 -D kata-containers.img "$PREFIX/share/kata-containers/${image}" && \
6  (cd $PREFIX/share/kata-containers && sudo ln -sf "$image" kata-containers.img)

(OPTIONAL) Next we will build an initrd image:

1cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/initrd-builder
2script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true ./initrd_builder.sh ${ROOTFS_DIR}'

(OPTIONAL) Once the image is built, we install it:

1export PREFIX=/opt/kata
2commit=$(git log --format=%h -1 HEAD) && \
3  date=$(date +%Y-%m-%d-%T.%N%z) && \
4  image="kata-containers-initrd-${date}-${commit}" && \
5  sudo install -o root -g root -m 0640 -D kata-containers-initrd.img "$PREFIX/share/kata-containers/${image}" && \
6  (cd $PREFIX/share/kata-containers && sudo ln -sf "$image" kata-containers-initrd.img)

Build Kata Containers kernel Link to heading

First, we need some additional packages to build the kernel:

1sudo apt install -y libelf-dev bison flex

Setup the kernel source code:

1cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel
2./build-kernel.sh -d setup

Build the kernel:

1./build-kernel.sh -d build

Install the kernel in the default path for Kata:

1export PREFIX=/opt/kata
2sudo -E PATH=$PATH -E PREFIX=$PREFIX ./build-kernel.sh -d install

Note:

We noticed that in some instances the installation or build process failed with the following error: ERROR: path to kernel does not exist, use build-kernel.sh setup. We mitigated this problem by specifying the version:

1./build-kernel.sh -d -v 5.15.26 build
2export PREFIX=/opt/kata
3sudo -E PATH=$PATH -E PREFIX=$PREFIX ./build-kernel.sh -d -v 5.15.26 install

Next steps Link to heading

At this point we have successfully built all the Kata components. All the binaries we built are stored under the /opt/kata/bin dir:

1$ ls -l /opt/kata/bin/
2total 142296
3-rwxr-xr-x 1 root root 50919312 Mar  25 15:32 containerd-shim-kata-v2
4-rwxr-xr-x 1 root root    16691 Mar  25 15:32 kata-collect-data.sh
5-rwxr-xr-x 1 root root 42093616 Mar  25 15:32 kata-monitor
6-rwxr-xr-x 1 root root 52673784 Mar  25 15:32 kata-runtime

The rootfs image, the initrd image and the kernel are stored under the /opt/kata/share/defaults/kata-containers dir:

 1$ ls -l /opt/kata/share/kata-containers/
 2total 221972
 3-rw-r--r-- 1 root root     72480 Μαρ  25 15:51 config-5.15.26
 4-rw-r----- 1 root root 134217728 Μαρ  25 15:41 kata-containers-2022-03-25-15:41:55.534872004+0200-486322a0
 5lrwxrwxrwx 1 root root        59 Μαρ  25 15:41 kata-containers.img -> kata-containers-2022-03-25-15:41:55.534872004+0200-486322a0
 6-rw-r----- 1 root root  27627256 Μαρ  24 14:43 kata-containers-initrd-2022-03-24-14:43:59.501993241+0200-853dd98b
 7-rw-r----- 1 root root  27626874 Μαρ  25 15:42 kata-containers-initrd-2022-03-25-15:42:28.034074480+0200-486322a0
 8lrwxrwxrwx 1 root root        66 Μαρ  25 15:42 kata-containers-initrd.img -> kata-containers-initrd-2022-03-25-15:42:28.034074480+0200-486322a0
 9-rw-r--r-- 1 root root  38736168 Μαρ  25 15:51 vmlinux-5.15.26-90
10lrwxrwxrwx 1 root root        18 Μαρ  25 15:51 vmlinux.container -> vmlinux-5.15.26-90
11-rw-r--r-- 1 root root   5795664 Μαρ  25 15:51 vmlinuz-5.15.26-90
12lrwxrwxrwx 1 root root        18 Μαρ  25 15:51 vmlinuz.container -> vmlinuz-5.15.26-90

The configuration files are stored under the /opt/kata/share/defaults/kata-containers dir:

1ls -l /opt/kata/share/defaults/kata-containers
2total 72
3-rw-r--r-- 1 root root  9717 Μαρ  25 15:32 configuration-acrn.toml
4-rw-r--r-- 1 root root 13535 Μαρ  25 15:32 configuration-clh.toml
5-rw-r--r-- 1 root root 15364 Μαρ  25 15:32 configuration-fc.toml
6-rw-r--r-- 1 root root 25701 Μαρ  25 15:32 configuration-qemu.toml
7lrwxrwxrwx 1 root root    23 Μαρ  25 15:32 configuration.toml -> configuration-qemu.toml

Now, we need to install a hypervisor (eg QEMU or Firecracker) and configure Kata Containers and containerd accordingly. In our upcoming posts we will go through the process of building and configuring both QEMU and AWS Firecracker for x86 and arm64 hosts. Happy hacking!