This is the first of a number of posts regarding the orchestration, deployment and scaling of containerized applications in VM sandboxes using kubernetes, kata-containers and AWS Firecracker microVMs. We have gathered some notes during the installation and configuration of the necessary components and we thought they might be useful to the community, especially with regards to the major pain points in trying out recent open-source projects and technologies.
About Orchestration, the Edge, and Kata Containers Link to heading
To manage and orchestrate containers in a cluster, the community is using kubernetes (k8s), a powerful, open-source system for automating the deployment, scaling and management of containerized applications. To accommodate the vast majority of options and use-cases, k8s supports a number of container runtimes: the piece of software that sets up all the necessary components in the system to run the containerized application. For more information on the container runtime support or k8s visit the official documentation.
k8s at the edge Link to heading
A stripped down version of k8s is available through K3s, a kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances. K3s is packaged as a single <40MB binary that reduces the dependencies and steps needed to install, run and auto-update a production Kubernetes cluster. K3s supports various architectures (amd64, ARMv8, ARMv8) with binaries and multiarch images available for all. K3s works great from something as small as a Raspberry Pi to an AWS a1.4xlarge 32GiB server. K3s is also great for trying out k8s on a local machine without messing up the OS.
container sandboxing Link to heading
Kata Containers enable containers to be seamlessly executed in Virtual Machines. Kata Containers are as light and fast as containers and integrate with the container management layers, while also delivering the security advantages of VMs. Kata Containers is the result of merging two existing open source projects: Intel Clear Containers and Hyper runV.
Kata Containers integrate with k8s easily; however, there are some (minor) pain points when adding more options to the mix. For instance, choosing AWS Firecracker as the VMM for the sandbox environment, brings a storage backend dependency: device mapper. In this post, we will be going through the steps needed to setup kata containers with Firecracker, focusing on the device mapper setup for k8s and k3s.
Kata containers and containerd Link to heading
First, lets start of by installing kata containers and add the relevant handler to containerd. Following the docs is pretty straightforward, especially this one. To make sure everything is working as expected, you could try out a couple of examples found here.
In short, the needed steps are:
install kata binaries Link to heading
Download a release from: https://github.com/kata-containers/kata-containers/releases
(using v2.1.1, released in June 2021):
1$ wget https://github.com/kata-containers/kata-containers/releases/download/2.1.1/kata-static-2.1.1-x86_64.tar.xz
Unpack the binary
1$ xzcat kata-static-2.1.1-x86_64.tar.xz | sudo tar -xvf - -C /
by default, kata is being installed in /opt/kata. Check the installed version by running:
1$ /opt/kata/bin/kata-runtime --version
It should output something like the following:
1$ /opt/kata/bin/kata-runtime --version
2kata-runtime : 2.1.1
3 commit : 0e2be438bdd6d213ac4a3d7d300a5757c4137799
4 OCI specs: 1.0.1-dev
It is recommended you add a symbolic link to /opt/kata/bin/kata-runtime and
/opt/kata/bin/containerd-shim-kata-v2 in order for containerd to reach these
binaries from the default system PATH.
1$ sudo ln -s /opt/kata/bin/kata-runtime /usr/local/bin
2$ sudo ln -s /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin
install containerd Link to heading
To install containerd, you can grab a release from
https://github.com/containerd/containerd/releases or use the package manager
of your distro:
1$ sudo apt-get install containerd
A fairly recent containerd version is recommended (e.g. we’ve only tested with
containerd versions v1.3.9 and above).
Add the kata configuration to containerd’ config.toml (/etc/containerd/config.toml):
1[plugins]
2 [plugins."io.containerd.grpc.v1.cri"]
3 [plugins."io.containerd.grpc.v1.cri".containerd]
4 default_runtime_name = "kata"
5 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
6 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
7 runtime_type = "io.containerd.kata.v2"
If /etc/containerd/config.toml is not present, create it using the following
command:
1$ sudo containerd config default > /etc/containerd/config.toml
and add the above snippet to the relevant section.
test kata with containerd Link to heading
Now, after we restart containerd:
1sudo systemctl restart containerd
we should be able to launch a Ubuntu test container using kata containers and containerd:
1$ sudo ctr image pull docker.io/library/ubuntu:latest
2docker.io/library/ubuntu:latest: resolved |++++++++++++++++++++++++++++++++++++++|
3index-sha256:82becede498899ec668628e7cb0ad87b6e1c371cb8a1e597d83a47fac21d6af3: done |++++++++++++++++++++++++++++++++++++++|
4manifest-sha256:1e48201ccc2ab83afc435394b3bf70af0fa0055215c1e26a5da9b50a1ae367c9: done |++++++++++++++++++++++++++++++++++++++|
5layer-sha256:16ec32c2132b43494832a05f2b02f7a822479f8250c173d0ab27b3de78b2f058: done |++++++++++++++++++++++++++++++++++++++|
6config-sha256:1318b700e415001198d1bf66d260b07f67ca8a552b61b0da02b3832c778f221b: done |++++++++++++++++++++++++++++++++++++++|
7elapsed: 7.2 s total: 27.2 M (3.8 MiB/s)
8unpacking linux/amd64 sha256:82becede498899ec668628e7cb0ad87b6e1c371cb8a1e597d83a47fac21d6af3...
9done
10$ ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/ubuntu:latest ubuntu-kata-test /bin/bash
11root@clr-d25fa567d2f440df9eb4316de1699b51:/# uname -a
12Linux clr-d25fa567d2f440df9eb4316de1699b51 5.10.25 #1 SMP Fri Apr 9 18:18:14 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Kata containers and AWS Firecracker Link to heading
The above example is using the QEMU/KVM hypervisor. To use a different hypervisor (eg. AWS Firecracker), we need to make sure we honor all the needed prerequisites, so that the VM sandbox is spawned correctly and is able to host the containerized application we want to run. Let’s take a step back and see how kata containers fetch the container image and run the application in the sandboxed environment.
kata containers execution flow Link to heading
As mentioned earlier, Kata containers provide a way for containerized
application to run inside a VM sandbox. Additionally, kata containers manage
the container execution via a runtime system on the Host and an agent running
in the sandbox. The containerized application is packaged in a container
image, which is pulled outside the sandbox environment, along with its metadata
description (the json file). These two components comprise the container
bundle. In order to run the container inside the sandbox environment (that is,
the VM) the container rootfs (the layer stack) must be somehow exposed from the
host to the VM. Then, the kata agent, which runs inside the VM, creates a
container from the exposed rootfs.
In the case of QEMU/KVM, the container rootfs is exposed using a virtiofs shared directory between the host and the guest. In the case of AWS Firecracker, however, virtiofs is not supported. So, the only option is to use a virtio block device.
devmapper Link to heading
In order to expose the container inside an AWS Firecracker sandbox environment, the kata runtime system expects that containerd uses the devmapper snapshotter. Essentially, the container rootfs is a device mapper snapshot, hot-plugged to AWS Firecracker as a virtio block device. The kata agent running in the VM finds the mount point inside the guest and issues the relevant command to libcontainerd to create and spawn the container.
So, in order to glue all the above together, we need containerd configured with the devmapper snapshotter. The first step is to setup a device mapper thin-pool. On a local dev environment you can use loopback devices.
A simple script from the devmapper snapshotter documentation is show below:
1#!/bin/bash
2set -ex
3
4DATA_DIR=/var/lib/containerd/io.containerd.snapshotter.v1.devmapper
5POOL_NAME=containerd-pool
6
7mkdir -p ${DATA_DIR}
8
9# Create data file
10sudo touch "${DATA_DIR}/data"
11sudo truncate -s 100G "${DATA_DIR}/data"
12
13# Create metadata file
14sudo touch "${DATA_DIR}/meta"
15sudo truncate -s 40G "${DATA_DIR}/meta"
16
17# Allocate loop devices
18DATA_DEV=$(sudo losetup --find --show "${DATA_DIR}/data")
19META_DEV=$(sudo losetup --find --show "${DATA_DIR}/meta")
20
21# Define thin-pool parameters.
22# See https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt for details.
23SECTOR_SIZE=512
24DATA_SIZE="$(sudo blockdev --getsize64 -q ${DATA_DEV})"
25LENGTH_IN_SECTORS=$(bc <<< "${DATA_SIZE}/${SECTOR_SIZE}")
26DATA_BLOCK_SIZE=128
27LOW_WATER_MARK=32768
28
29# Create a thin-pool device
30sudo dmsetup create "${POOL_NAME}" \
31 --table "0 ${LENGTH_IN_SECTORS} thin-pool ${META_DEV} ${DATA_DEV} ${DATA_BLOCK_SIZE} ${LOW_WATER_MARK}"
32
33cat << EOF
34#
35# Add this to your config.toml configuration file and restart containerd daemon
36#
37[plugins]
38 [plugins.devmapper]
39 pool_name = "${POOL_NAME}"
40 root_path = "${DATA_DIR}"
41 base_image_size = "40GB"
42EOF
This script needs to be run only once, while setting up the devmapper snapshotter for containerd. Afterwards, make sure that on each reboot, the thin-pool is initialized from the same data dir. Otherwise, all the fetched containers (or the ones that you’ve created will be re-initialized). As simple script that re-creates the thin-pool from the same data dir is show below:
1#!/bin/bash
2set -ex
3
4DATA_DIR=/var/lib/containerd/io.containerd.snapshotter.v1.devmapper
5POOL_NAME=containerd-pool
6
7# Allocate loop devices
8DATA_DEV=$(sudo losetup --find --show "${DATA_DIR}/data")
9META_DEV=$(sudo losetup --find --show "${DATA_DIR}/meta")
10
11# Define thin-pool parameters.
12# See https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt for details.
13SECTOR_SIZE=512
14DATA_SIZE="$(sudo blockdev --getsize64 -q ${DATA_DEV})"
15LENGTH_IN_SECTORS=$(bc <<< "${DATA_SIZE}/${SECTOR_SIZE}")
16DATA_BLOCK_SIZE=128
17LOW_WATER_MARK=32768
18
19# Create a thin-pool device
20sudo dmsetup create "${POOL_NAME}" \
21 --table "0 ${LENGTH_IN_SECTORS} thin-pool ${META_DEV} ${DATA_DEV} ${DATA_BLOCK_SIZE} ${LOW_WATER_MARK}"
After a containerd restart (systemctl restart containerd), we should be able
to see the plugin registered and working correctly:
1$ ctr plugins ls |grep devmapper
2io.containerd.snapshotter.v1 devmapper linux/amd64 ok
containerd handler setup Link to heading
The last step to setup AWS Firecracker with kata and containerd, is add the
relevant handler to containerd config.toml. To use a separate handler and
keep both QEMU/KVM and AWS Firecracker options as the hypervisor for kata
containers, we create a simple script that calls the kata containers runtime
with a different config file. Add a file in your path (eg.
/usr/local/bin/containerd-shim-kata-fc-v2) with the following contents:
1#!/bin/bash
2KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-fc.toml /opt/kata/bin/containerd-shim-kata-v2 $@
make it executable:
1$ chmod +x /usr/local/bin/containerd-shim-kata-fc-v2
and add the relevant section in containerd’s config.toml file (/etc/containerd/config.toml):
1[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
2 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-fc]
3 runtime_type = "io.containerd.kata-fc.v2"
After a containerd restart (systemctl restart containerd), we should be able
to launch a Ubuntu test container using kata containers and AWS Firecracker:
1root@nuc8:~# ctr images pull --snapshotter devmapper docker.io/library/ubuntu:latest
2docker.io/library/ubuntu:latest: resolved |++++++++++++++++++++++++++++++++++++++|
3index-sha256:82becede498899ec668628e7cb0ad87b6e1c371cb8a1e597d83a47fac21d6af3: done |++++++++++++++++++++++++++++++++++++++|
4manifest-sha256:1e48201ccc2ab83afc435394b3bf70af0fa0055215c1e26a5da9b50a1ae367c9: done |++++++++++++++++++++++++++++++++++++++|
5config-sha256:1318b700e415001198d1bf66d260b07f67ca8a552b61b0da02b3832c778f221b: done |++++++++++++++++++++++++++++++++++++++|
6layer-sha256:16ec32c2132b43494832a05f2b02f7a822479f8250c173d0ab27b3de78b2f058: done |++++++++++++++++++++++++++++++++++++++|
7elapsed: 7.9 s total: 27.2 M (3.4 MiB/s)
8unpacking linux/amd64 sha256:82becede498899ec668628e7cb0ad87b6e1c371cb8a1e597d83a47fac21d6af3...
9done
10root@nuc8:~# ctr run --snapshotter devmapper --runtime io.containerd.run.kata-fc.v2 -t --rm docker.io/library/ubuntu:latest ubuntu-kata-fc-test uname -a
11Linux clr-f083d4640978470da04f091181bb9e95 5.10.25 #1 SMP Fri Apr 9 18:18:14 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Configure k8s to use AWS Firecracker with kata containers Link to heading
Following the above steps, integrating kata containers with k8s is a piece of cake ;-)
Two things are needed to create a pod with kata and AWS Firecracker:
- make the containerd snapshotter default
- add the kata containers runtime class to k8s
To configure the devmapper snapshotter as the default snapshotter in containerd add the following lines to /etc/containerd/config.toml:
1[plugins.cri.containerd]
2 snapshotter = "devmapper"
To add the kata containers runtime class to k8s, we need to create a simple YAML file (kata-fc-rc.yaml) containing the name of the kata-fc handler we configured earlier:
1apiVersion: node.k8s.io/v1
2kind: RuntimeClass
3metadata:
4 name: kata-fc
5handler: kata-fc
and apply it to our k8s cluster:
1$ kubectl apply -f kata-fc-rc.yaml
After a containerd restart (systemctl restart containerd), we should be able to create a pod
to see the plugin registered and working correctly. Take for instance the following YAML file (nginx-kata-fc.yaml):
1apiVersion: v1
2kind: Pod
3metadata:
4 name: nginx-kata-fc
5spec:
6 runtimeClassName: kata-fc
7 containers:
8 - name: nginx
9 image: nginx
and apply it:
1$ kubectl apply -f nginx-kata-fc.yaml
the output should be something like the following:
1$ kubectl apply -f nginx-kata-fc.yaml
2pod/nginx-kata-fc created
Inspecting the pod in the node created should give us the following:
1$ kubectl describe pod nginx-kata-fc
2Name: nginx-kata-fc
3[snipped]
4Containers:
5 nginx:
6 Container ID: containerd://bbb6dc73c3a0c727dae81b4be0b93d853c9e2a7843e3f037b934bcb5aea89ece
7 Image: nginx
8 Image ID: docker.io/library/nginx@sha256:8f335768880da6baf72b70c701002b45f4932acae8d574dedfddaf967fc3ac90
9 Port: <none>
10 Host Port: <none>
11 State: Running
12 Ready: True
13 Restart Count: 0
14 Environment: <none>
15 Mounts:
16 /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-psxw6 (ro)
17Conditions:
18 Type Status
19 Initialized True
20 Ready True
21 ContainersReady True
22 PodScheduled True
23Volumes:
24 kube-api-access-psxw6:
25 Type: Projected (a volume that contains injected data from multiple sources)
26 TokenExpirationSeconds: 3607
27 ConfigMapName: kube-root-ca.crt
28 ConfigMapOptional: <nil>
29 DownwardAPI: true
30QoS Class: BestEffort
31Node-Selectors: katacontainers.io/kata-runtime=true
32Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
33 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
34Events:
35 Type Reason Age From Message
36 ---- ------ ---- ---- -------
37 Normal Pulling 107s kubelet Pulling image "nginx"
38 Normal Pulled 87s kubelet Successfully pulled image "nginx" in 20.37180329s
39 Normal Created 87s kubelet Created container nginx
40 Normal Started 87s kubelet Started container nginx
Digging in a bit deeper:
1$ ps -ef |grep firecracker
2root 2705170 2705157 1 [snipped] ? 00:00:00 /opt/kata/bin/firecracker --api-sock /run/vc/firecracker/00fedc54a0bb0e55bd0513e0cf3c551f/root/run/firecracker.socket --config-file /run/vc/firecracker/00fedc54a0bb0e55bd0513e0cf3c551f/root/fcConfig.json
3$ kubectl exec -it nginx-kata-fc -- /bin/bash
4root@nginx-kata-fc:/# uname -a
5Linux nginx-kata-fc 5.10.25 #1 SMP Fri Apr 9 18:18:14 UTC 2021 x86_64 GNU/Linux
6root@nginx-kata-fc:/# cat /proc/cpuinfo |head -n 10
7processor : 0
8vendor_id : GenuineIntel
9cpu family : 6
10model : 142
11model name : Intel(R) Xeon(R) Processor @ 3.00GHz
12stepping : 10
13microcode : 0x1
14cpu MHz : 3000.054
15cache size : 4096 KB
16physical id : 0
17root@nginx-kata-fc:/# cat /proc/meminfo |head -n 10
18MemTotal: 2043696 kB
19MemFree: 1988208 kB
20MemAvailable: 1997152 kB
21Buffers: 2172 kB
22Cached: 31224 kB
23SwapCached: 0 kB
24Active: 16336 kB
25Inactive: 21264 kB
26Active(anon): 40 kB
27Inactive(anon): 4204 kB
Looking into the container id via ctr, we can see that is using the kata-fc handler:
1$ ctr -n k8s.io c ls |grep bbb6dc73c3a0c727dae81b4be0b93d853c9e2a7843e3f037b934bcb5aea89ece
2bbb6dc73c3a0c727dae81b4be0b93d853c9e2a7843e3f037b934bcb5aea89ece docker.io/library/nginx:latest io.containerd.kata-fc.v2
As a side note, containerd matches the handler notation with an executable like this:
1io.containerd.kata-fc.v2 -> containerd-shim-kata-fc-v2
so the name of the script we created above
(/usr/local/bin/containerd-shim-kata-fc-v2) must match the name of the
handler in containerd config.toml using the above notation.
Configure k3s to use AWS Firecracker with kata containers Link to heading
k3s ships with a minimal version of containerd where the devmapper snapshotter plugin is not included.
As a result, in order to add AWS Firecracker support to k3s, we either need to
configure k3s to use a different containerd binary or build k3s-containerd from
source, patch it and inject it to
/var/lib/rancher/k3s/data/current/bin.
Use external containerd Link to heading
Make sure you have configured containerd to use the cri devmapper plugin correctly (see above). Then just install k3s using the system’s containerd.
1curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--container-runtime-endpoint unix:///run/containerd/containerd.sock" sh -
Build k3s’ containerd Link to heading
To build k3s’ containerd we need to patch the source in order to enable the devmapper snapshotter. So:
(i) get the source:
1$ git clone https://github.com/k3s-io/k3s
(ii) patch with the following file (devmapper_patch.txt):
1diff --git a/pkg/containerd/builtins_linux.go b/pkg/containerd/builtins_linux.go
2index 280b09ce..89c700eb 100644
3--- a/pkg/containerd/builtins_linux.go
4+++ b/pkg/containerd/builtins_linux.go
5@@ -25,5 +25,6 @@ import (
6 _ "github.com/containerd/containerd/runtime/v2/runc/options"
7 _ "github.com/containerd/containerd/snapshots/native"
8 _ "github.com/containerd/containerd/snapshots/overlay"
9+ _ "github.com/containerd/containerd/snapshots/devmapper"
10 _ "github.com/containerd/fuse-overlayfs-snapshotter/plugin"
11 )
12diff --git a/vendor/modules.txt b/vendor/modules.txt
13index 91554f14..054d04bf 100644
14--- a/vendor/modules.txt
15+++ b/vendor/modules.txt
16@@ -307,6 +307,8 @@ github.com/containerd/containerd/services/snapshots
17 github.com/containerd/containerd/services/tasks
18 github.com/containerd/containerd/services/version
19 github.com/containerd/containerd/snapshots
20+github.com/containerd/containerd/snapshots/devmapper
21+github.com/containerd/containerd/snapshots/devmapper/dmsetup
22 github.com/containerd/containerd/snapshots/native
23 github.com/containerd/containerd/snapshots/overlay
24 github.com/containerd/containerd/snapshots/proxy
1$ cd k3s
2$ patch -p1 < ../devmapper_patch.txt
3patching file pkg/containerd/builtins_linux.go
4patching file vendor/modules.txt
and (iii) build:
1$ sudo apt-get install btrfs-progs libbtrfs-dev
2$ mkdir -p build/data && ./scripts/download && go generate
3$ go get -d github.com/containerd/containerd/snapshots/devmapper
4$ SKIP_VALIDATE=true make # because we have local changes
Hopefully, you’ll be presented with a k3s binary ;-)
install kata-containers on k3s Link to heading
After a successful build & run of k3s, you should be able to list the available nodes for your local cluster:
1$ k3s kubectl get nodes --show-labels
2NAME STATUS ROLES AGE VERSION LABELS
3mycluster.localdomain Ready control-plane,master 11m v1.21.3+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=mycluster.localdomain,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s
Apart from the manual install we saw above for k8s, we could just use the well documented process in the kata-deploy section of the kata-containers repo.
So, just run the following:
1$ k3s kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-rbac/base/kata-rbac.yaml
2serviceaccount/kata-label-node created
3clusterrole.rbac.authorization.k8s.io/node-labeler created
4clusterrolebinding.rbac.authorization.k8s.io/kata-label-node-rb created
5$ k3s kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
6daemonset.apps/kata-deploy created
This will fetch the necessary binaries and install them on /opt/kata (via the
daemonset). It will also add the relevant handlers on
/etc/containerd/config.toml. However, in order to use these handlers we need to
create the respective runtime classes. To add them run the following:
1$ k3s kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/runtimeclasses/kata-runtimeClasses.yaml
2runtimeclass.node.k8s.io/kata-qemu-virtiofs created
3runtimeclass.node.k8s.io/kata-qemu created
4runtimeclass.node.k8s.io/kata-clh created
5runtimeclass.node.k8s.io/kata-fc created
Query the system to see if kata-deploy was successful:
1$ k3s kubectl get nodes --show-labels
2NAME STATUS ROLES AGE VERSION LABELS
3mycluster.localdomain Ready control-plane,master 62m v1.21.3+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=mycluster.localdomain,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s,katacontainers.io/kata-runtime=true
See the last label? its there because kata containers installation was successful! So we’re ready to spawn a sandboxed container on k3s:
1$ k3s kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-fc.yaml
2deployment.apps/php-apache-kata-fc created
3service/php-apache-kata-fc created
Lets see what has been created:
1$ k3s kubectl get pods
2NAME READY STATUS RESTARTS AGE
3php-apache-kata-fc-5ccb8df89-v8hqj 1/1 Running 0 31s
1$ ps -ef |grep firecracker
2root 128201 128187 3 15:21 ? 00:00:02 /opt/kata/bin/firecracker --api-sock /run/vc/firecracker/2050be93ed33df24537f6f8274e3c66c/root/run/firecracker.socket --config-file /run/vc/firecracker/2050be93ed33df24537f6f8274e3c66c/root/fcConfig.json
1$ k3s kubectl describe pod php-apache-kata-fc-5ccb8df89-v8hqj
2Name: php-apache-kata-fc-5ccb8df89-v8hqj
3Namespace: default
4Priority: 0
5Node: mycluster.localdomain
6Labels: pod-template-hash=5ccb8df89
7 run=php-apache-kata-fc
8Annotations: <none>
9Status: Running
10IP: 10.42.0.192
11IPs:
12 IP: 10.42.0.192
13Controlled By: ReplicaSet/php-apache-kata-fc-5ccb8df89
14Containers:
15 php-apache:
16 Container ID: containerd://13718fd3820efb1c0260ec9d88c039c37325a1e76a3edca943c62e6dab02c549
17 Image: k8s.gcr.io/hpa-example
18 Image ID: k8s.gcr.io/hpa-example@sha256:581697a37f0e136db86d6b30392f0db40ce99c8248a7044c770012f4e8491544
19 Port: 80/TCP
20 Host Port: 0/TCP
21 State: Running
22 Ready: True
23 Restart Count: 0
24 Requests:
25 cpu: 200m
26 Environment: <none>
27 Mounts:
28 /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k62m5 (ro)
29Conditions:
30 Type Status
31 Initialized True
32 Ready True
33 ContainersReady True
34 PodScheduled True
35Volumes:
36 kube-api-access-k62m5:
37 Type: Projected (a volume that contains injected data from multiple sources)
38 TokenExpirationSeconds: 3607
39 ConfigMapName: kube-root-ca.crt
40 ConfigMapOptional: <nil>
41 DownwardAPI: true
42QoS Class: Burstable
43Node-Selectors: katacontainers.io/kata-runtime=true
44Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
45 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
46Events:
47 Type Reason Age From Message
48 ---- ------ ---- ---- -------
49 Normal Scheduled 82s default-scheduler Successfully assigned default/php-apache-kata-fc-5ccb8df89-v8hqj to mycluster.localdomain
50 Normal Pulling 80s kubelet Pulling image "k8s.gcr.io/hpa-example"
51 Normal Pulled 80s kubelet Successfully pulled image "k8s.gcr.io/hpa-example" in 745.344896ms
52 Normal Created 80s kubelet Created container php-apache
53 Normal Started 79s kubelet Started container php-apache
To tear things down on k3s run the following:
1$ k3s kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-fc.yaml
2deployment.apps "php-apache-kata-fc" deleted
3service "php-apache-kata-fc" deleted
4$ k3s kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/runtimeclasses/kata-runtimeClasses.yaml
5runtimeclass.node.k8s.io "kata-qemu-virtiofs" deleted
6runtimeclass.node.k8s.io "kata-qemu" deleted
7runtimeclass.node.k8s.io "kata-clh" deleted
8runtimeclass.node.k8s.io "kata-fc" deleted
9$ k3s kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
10daemonset.apps "kata-deploy" deleted
11$ k3s kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-rbac/base/kata-rbac.yaml
12serviceaccount "kata-label-node" deleted
13clusterrole.rbac.authorization.k8s.io "node-labeler" deleted
14clusterrolebinding.rbac.authorization.k8s.io "kata-label-node-rb" deleted
This concludes a first take on running containers as microVMs in k8s using Kata Containers and AWS Firecracker. In the next posts we will explore hardware acceleration options, serverless frameworks and unikernel execution! Stay tuned for more!
NOTE I: If you choose the first option for k3s, you’ll need to manually
append the handlers in /etc/containerd/config.toml as the k3s denotes a
relative path for containerd’s config file, so it resides in
/var/lib/rancher/k3s/agent/etc/containerd/config.toml. The easiest way to
work around this is:
1cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml >> /etc/containerd/config.toml
2systemctl restart containerd
NOTE II: If you are already running k8s/k3s with a different containerd
snapshotter, after configuring devmapper and restarting containerd, you
probably need to restart kubelet/k3s services and make sure that all the
containers running in k8s.io containerd namespace are now running using
devicemapper.