Picking up from where we left in our previous post, we will now install QEMU and configure Kata Containers to use QEMU as their hypervisor.

Build QEMU Link to heading

First, we need to build qemu-system for the CPU architecture of our host machine. Kata Containers provide scripts to manage the build of QEMU both for x86 and arm64 hosts. We will be using them to make sure that our QEMU installation is suitable for usage with Kata Containers.

Build QEMU for x86 Link to heading

We need to get the correct version of QEMU from the versions file:

1export GOPATH=$(go env GOPATH) && export GO111MODULE=off
2source ${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging/scripts/lib.sh
3qemu_version=$(get_from_kata_deps "assets.hypervisor.qemu.version")

Next, we need to get the QEMU source from the matching branch:

1go get -d github.com/qemu/qemu
2cd ${GOPATH}/src/github.com/qemu/qemu
3git checkout ${qemu_version}
4your_qemu_directory=${GOPATH}/src/github.com/qemu/qemu

We need to see which version of QEMU we will be building.

1echo ${qemu_version}
2# v6.2.0

Now we will use the scripts provided to apply the necessary patches to QEMU. Make sure to change the commands below to match the version of QEMU that you are targeting.

1packaging_dir="${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging"
2$packaging_dir/scripts/apply_patches.sh $packaging_dir/qemu/patches/6.2.x/

Once the patches are successfully applied, we need to install some apt packages that are required to build qemu-system:

1sudo apt install ninja-build pkg-config \
2  libglib2.0-dev \
3  libpixman-1-dev \
4  libseccomp-dev \
5  libcap-ng-dev \
6  librados-dev \
7  libpmem-dev \
8  librbd-dev \
9  libgcrypt-dev

Now, we can finally build QEMU:

1cd $your_qemu_directory
2$packaging_dir/scripts/configure-hypervisor.sh kata-qemu > kata.cfg
3eval ./configure "$(cat kata.cfg)"
4make -j $(nproc)
5sudo -E make install

If you followed the instructions in our previous post and you have set the PREFIX variable to /opt/kata the qemu-system-x86_64 binary will be stored under the /opt/kata/bin/ directory, tools/virtiofsd/virtiofsd under the /opt/kata/libexec/kata-qemu directory and all the other files will be stored under the /opt/kata/share/kata-qemu directory.

We can also install virtiofsd and qemu-system-x86_64 under the default location using the install_qemu.sh script.

1go get -d github.com/kata-containers/tests
2script -fec 'sudo -E ${GOPATH}/src/github.com/kata-containers/tests/.ci/install_qemu.sh'

Build QEMU for aarch64 Link to heading

Install requirements:

1sudo apt-get install -y build-essential pkg-config \
2  libglib2.0-dev libpixman-1-dev \
3  libaio-dev libseccomp-dev \
4  libcap-ng-dev librados-dev \
5  librbd-dev libzstd-dev

Set required environment variables

1export PATH=$PATH:$(go env GOPATH)/bin && \
2  export GOPATH=$(go env GOPATH) && \
3  export GO111MODULE=off

Build QEMU using the install_qemu.sh script:

1sudo rm -rf ${GOPATH}/src/github.com/qemu/qemu && \
2  go get -d github.com/kata-containers/tests && \
3  script -fec 'sudo -E ${GOPATH}/src/github.com/kata-containers/tests/.ci/install_qemu.sh'

Configure Kata Containers Link to heading

Next, we need to install the Kata Containers configuration file. We will use this file to configure Kata Containers to use the initrd image we built in our previous post. You could also use a rootfs image, if you prefer. We will also enable guest seccomp.

1sudo mkdir -p /opt/kata/configs
2sudo install -o root -g root -m 0640 /opt/kata/share/defaults/kata-containers/configuration.toml /opt/kata/configs
3# comment out the image entry
4sudo sed -i 's/^\(image =.*\)/# \1/g' /opt/kata/configs/configuration.toml
5# enable seccomp
6sudo sed -i '/^disable_guest_seccomp/ s/true/false/' /opt/kata/configs/configuration.toml

Make sure that /opt/kata/configs/configuration.toml has a initrd entry pointing to the initrd we created:

117   | #image = "/opt/kata/share/kata-containers/kata-containers-image.img"
218   │ initrd = "/opt/kata/share/kata-containers/kata-containers-initrd.img"

Configure containerd Link to heading

You need to edit the containerd config file (usually /etc/containerd/config.toml.

First we need to remove cri from the disabled plugins list:

1#disabled_plugins = ["cri"]
2disabled_plugins = []

We also need to add the relevant handler for the kata runtime:

1[plugins]
2  [plugins.cri]
3    [plugins.cri.containerd]
4      [plugins.cri.containerd.runtimes]
5        [plugins.cri.containerd.runtimes.kata]
6         runtime_type = "io.containerd.kata.v2"
7         privileged_without_host_devices = true
8         [plugins.cri.containerd.runtimes.kata.options]
9           ConfigPath = "/opt/kata/configs/configuration.toml"

Once the changes are saved, restart containerd: sudo systemctl restart containerd.

To verify our installation is successful, we can launch an Ubuntu test container:

1sudo ctr images pull docker.io/library/ubuntu:latest
2sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/ubuntu:latest ubuntu-kata-test uname -a
3# Linux e678ab3c6fc3 5.15.26 #2 SMP Sat Mar 19 21:02:56 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Note 1 for aarch64:

We noticed that, in some instances, trying to launch a container produced the following error:

1# sudo ctr images pull docker.io/library/busybox:latest
2# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello1 sh
3ctr: failed to create shim: failed to launch qemu: exit status 1, error messages from qemu log: qemu-system-aarch64: warning: For GICv2 max-cpus must be equal to smp-cpus
4qemu-system-aarch64: warning: Overriding specified max-cpus(4) with smp-cpus(1)
5qemu-system-aarch64: warning: global kvm-pit.lost_tick_policy has invalid class name
6Restoring 1 CPU interfaces, kernel only has 4
7: unknown

A workaround for that problem was to set the default_vcpus = 8 in /opt/kata/configs/configuration.toml.

Note 2 for aarch64:

We tried running on an older kernel, on an NVIDIA Jetson AGX Xavier and on an NVIDIA Jetson Nano, versions 4.9.253-tegra-virt (getting KVM to work on these boards is also quite a challenge, more details on a subsequent post). Apart from the GIC issue above, we saw that virtio-fs is not backported, nor is there support for the nvdimm stuff, so the following changes to the config file are needed:

 1--- configuration-qemu-stock.toml   2022-03-24 20:55:47.010284096 +0000
 2+++ configuration-qemu.toml    2022-03-24 21:34:39.725554126 +0000
 3@@ -103,7 +103,7 @@
 4 # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
 5 # unless you know what are you doing.
 6 # NOTICE: on arm platform with gicv2 interrupt controller, set it to 8.
 7-default_maxvcpus = 0
 8+default_maxvcpus = 1
 9
10 # Bridges can be used to hot plug devices.
11 # Limitations:
12@@ -151,7 +151,7 @@
13 #   - virtio-fs (default)
14 #   - virtio-9p
15 #   - virtio-fs-nydus
16-shared_fs = "virtio-fs"
17+shared_fs = "virtio-9p"
18
19 # Path to vhost-user-fs daemon.
20 virtio_fs_daemon = "/opt/kata/libexec/kata-qemu/virtiofsd"
21@@ -292,7 +292,7 @@
22 # nvdimm is not supported when `confidential_guest = true`.
23 #
24 # Default is false
25-#disable_image_nvdimm = true
26+disable_image_nvdimm = true
27
28 # VFIO devices are hotplugged on a bridge by default.
29 # Enable hotplugging on root bus. This may be required for devices with

Interestingly enough, default_vcpus must be equal to default_maxvcpus on such platforms.