Since we got our hands on the new Raspberry Pi 4, we started exploring how various virtualization technologies behave on the board. First thing we tried is how to run Nabla on it and how it compares to native KVM.
Next thing we wanted to try is firecracker, the notorious micro-VMM that Amazon Lambda & Fargate run on. To our disappointment, firecracker was not yet running on RPi4. So we started looking into coding in the necessary changes :)
After a bit of investigation, we figured out that the key missing piece is support for the GICv2 ARM interrupt controller. In fact, firecracker only supports GIC version 3 since it was open-sourced, but not version 2, which is the version appearing in the Raspberry Pi series, as well as in other popular boards, like the Hikey 970 we got our hands on, or Khadas VIM3. A bit more digging into the internals of firecracker and the Linux kernel, helped us work out the details and open a pull-request which adds support for the missing parts.
Changes in Firecracker Link to heading
The Generic Interrupt Controller (GIC) is the IP block in the ARM processors that implements interrupt handling. The Linux kernel supports user- and kernel-space emulation for GICv2 as well as GICv3 and v4. However, Firecracker only handles the GICv3-related configuration of the virtual GIC (VGIC). Similarly, setting up the FDT for the guest microVM from Firecracker only handles GIC-v3 devices.
In terms of code organization, the GIC related code currently exposes a function
for setting up GICv3 performing the corresponding ioctl KVM commands. The
first part of our PR changes this by introducing a GIC Trait which defines the
common interface for all GIC implementations. Even though it is still under
discussion, what exactly will be in the Trait in its final form, it will be
something along the following lines:
1/// Trait for GIC devices.
2pub trait GICDevice {
3 /// Returns the GIC version of the device
4 fn version(&self) -> u32;
5
6 /// Returns the file descriptor of the GIC device
7 fn device_fd(&self) -> &DeviceFd;
8
9 /// Returns an array with GIC device properties
10 fn device_properties(&self) -> &[u64];
11}
With this in place, we can define objects per GIC version, which implement this
Trait, and still have the rest of the code deal with the GICDevice Trait,
which is transparent to the GIC version.
The implementations for each version, implement the Trait and a new function
which is used to create the new object:
1pub fn new(vm: &VmFd, vcpu_count: u64) -> Result<Box<GICDevice>>
The differentiation between the two versions lays in the VGIC register
attributes that each device exposes. As a result, the work we need to do is essentially to mmap their addresses.
GICv2 relevant addresses include the distributor and cpu groups:
1/* Setting up the distributor attribute.
2 We are placing the GIC below 1GB so we need to substract the size of the distributor.
3*/
4set_device_attribute(
5 &vgic_fd,
6 kvm_bindings::KVM_DEV_ARM_VGIC_GRP_ADDR,
7 u64::from(kvm_bindings::KVM_VGIC_V2_ADDR_TYPE_DIST),
8 &get_dist_addr() as *const u64 as u64,
9 0,
10)?;
11
12/* Setting up the CPU attribute.
13 */
14set_device_attribute(
15 &vgic_fd,
16 kvm_bindings::KVM_DEV_ARM_VGIC_GRP_ADDR,
17 u64::from(kvm_bindings::KVM_VGIC_V2_ADDR_TYPE_CPU),
18 &get_cpu_addr() as *const u64 as u64,
19 0,
20)?;
whereas the GICv3 includes the distributor and redistributor groups:
1/* Setting up the distributor attribute.
2 We are placing the GIC below 1GB so we need to substract the size of the distributor.
3*/
4set_device_attribute(
5 &vgic_fd,
6 kvm_bindings::KVM_DEV_ARM_VGIC_GRP_ADDR,
7 u64::from(kvm_bindings::KVM_VGIC_V3_ADDR_TYPE_DIST),
8 &get_dist_addr() as *const u64 as u64,
9 0,
10)?;
11
12/* Setting up the redistributors' attribute.
13We are calculating here the start of the redistributors address. We have one per CPU.
14*/
15set_device_attribute(
16 &vgic_fd,
17 kvm_bindings::KVM_DEV_ARM_VGIC_GRP_ADDR,
18 u64::from(kvm_bindings::KVM_VGIC_V3_ADDR_TYPE_REDIST),
19 &get_redists_addr(u64::from(vcpu_count)) as *const u64 as u64,
20 0,
21)?;
Finally, for both versions of GIC we finalize the device initialization by setting the number of supported interrupts and the control_init group.
1/// Finalize the setup of a GIC device
2pub fn finalize_device(fd: &DeviceFd) -> Result<()> {
3 /* We need to tell the kernel how many irqs to support with this vgic.
4 * See the `layout` module for details.
5 */
6 let nr_irqs: u32 = super::layout::IRQ_MAX - super::layout::IRQ_BASE + 1;
7 let nr_irqs_ptr = &nr_irqs as *const u32;
8 set_device_attribute(
9 fd,
10 kvm_bindings::KVM_DEV_ARM_VGIC_GRP_NR_IRQS,
11 0,
12 nr_irqs_ptr as u64,
13 0,
14 )?;
15
16 /* Finalize the GIC.
17 * See https://code.woboq.org/linux/linux/virt/kvm/arm/vgic/vgic-kvm-device.c.html#211.
18 */
19 set_device_attribute(
20 fd,
21 kvm_bindings::KVM_DEV_ARM_VGIC_GRP_CTRL,
22 u64::from(kvm_bindings::KVM_DEV_ARM_VGIC_CTRL_INIT),
23 0,
24 0,
25 )?;
26
27 Ok(())
28}
Regarding the FDT creation, there are differences between v2 and v3 regarding
the interrupt controller intc node.
First, we need to define the GICv2 compatible property to be arm,gic-400,
since it is the GIC-400 chip which implements GICv2. Next is the reg
property of the FDT, which includes the addresses and the corresponding sizes of
the GIC registers, i.e. distributor and CPU for GICv2, distributor and
redistributor for GICv3. Finally, we fix the interrupts property which
determines the interrupt source of the VGIC maintenance interrupt. The
corresponding values for GICv2 were taken from the respective Linux Kernel entries.
Build firecracker with RPi4 support Link to heading
While waiting for the patch to merge upstream you can go ahead and check out it yourself.
Clone and build our fork of firecracker (keep in mind that while the PR review is going on, we might force-update the branch).
1$ git clone https://github.com/cloudkernels/firecracker.git
2$ cd firecracker
3$ ./tools/devtool build
Next you need a kernel image and a root filesystem to run with your firecracker build. You can grab the pre-built kernel image and rootfs here:
1$ wget https://s3.amazonaws.com/spec.ccfc.min/img/aarch64/ubuntu_with_ssh/kernel/vmlinux.bin
2$ wget https://s3.amazonaws.com/spec.ccfc.min/img/aarch64/ubuntu_with_ssh/fsfiles/xenial.rootfs.ext4
Alternatively, you can build your own kernel and rootfs using the following steps provided by the firecracker docs here
Launch our image Link to heading
To launch our image we will use the firectl tool. We need, to setup a tap device for the networking or our firecracker microVM.
1$ sudo ip tuntap add dev tap0 mode tap
2$ sudo ip addr add 172.16.0.1/24 dev tap0
3$ sudo ip link set tap0 up
And we launch the microVM using firectl:
1firectl --firecracker-binary=${PATH_TO_FC_BIN} --kernel=vmlinux.bin \
2 --tap-device=tap0/aa:fc:00:00:00:01 \
3 --kernel-opts="console=ttyS0 reboot=k panic=1 pci=off ip=172.16.0.42::172.16.0.1:255.255.255.0::eth0:off" \
4 --root-drive=./nginx_fc.ext4
Here, we used here the ip kernel boot parameter to give the 172.16.0.42.
Finally, we try out our nginx server:
1curl 172.16.0.42
2<!DOCTYPE html>
3<html>
4<head>
5<title>Welcome to nginx!</title>
6<style>
7 body {
8 width: 35em;
9 margin: 0 auto;
10 font-family: Tahoma, Verdana, Arial, sans-serif;
11 }
12</style>
13</head>
14<body>
15<h1>Welcome to nginx!</h1>
16<p>If you see this page, the nginx web server is successfully installed and
17working. Further configuration is required.</p>
18
19<p>For online documentation and support please refer to
20<a href="http://nginx.org/">nginx.org</a>.<br/>
21Commercial support is available at
22<a href="http://nginx.com/">nginx.com</a>.</p>
23
24<p><em>Thank you for using nginx.</em></p>
25</body>
26</html>