[UPDATE: Revise instructions to reflect upstream nabla changes.]

In this post, we will walk through the steps of compiling, baking, and running an application as a rumprun unikernel on a Raspberry Pi 3.

In our previous post, we provided some background for Rumprun/rump kernels and Solo5. In short, Rumprun provides the necessary components to run a POSIX compatible application as a unikernel. Solo5 is, essentially, a hardware abstraction layer that provides a very thin interface, or else a minimal attack surface.

Rumprun High-level stack

The Nabla Containers fork of Rumprun provides a solo5 target for the rump kernel unikernel stack. Additionally, they provide a Docker runtime that spawns unikernel images as containers. We will be talking about Nabla Containers in more detail in future posts. Stay tuned :)

For this tutorial, we will use our current aarch64 port. We will show you how to build and run a unikernel application on a RPi3. Keep in mind, that our port should work without issues on any aarch64 platform. Feel free to run this tutorial on your favourite ARM board and let us know about your experience!

However, enough with the boring theory, let’s get our hands dirty.

Hands-on Link to heading

First of all, we need the rumprun toolchain to bake our unikernel image. One option is to use our docker image with the toolchain preinstalled.

Another option is to build the rumprun toolchain from source. We will provide the necessary information on how to build all components for aarch64 in the coming days.

So, log in to your RPi3 and, after you’ve installed docker-ce, try running the following command:

1docker run -ti --rm -v /build:/build cloudkernels/debian-rumprun-build:aarch64 /bin/bash

you should be presented with something like the following:

 1Unable to find image 'cloudkernels/debian-rumprun-build:aarch64' locally
 2aarch64: Pulling from cloudkernels/debian-rumprun-build
 3e10919c546c2: Pull complete 
 46b3f0a4d7b10: Pull complete 
 5473e207e8cf0: Pull complete 
 60deecc1ceca2: Pull complete 
 7628025a81431: Pull complete 
 825fd95c63d4f: Pull complete 
 9Digest: sha256:0221ba1c3a120bde1fd83b6d9c267fb4379c33d8f0012b9a64826afd476faf72
10Status: Downloaded newer image for cloudkernels/debian-rumprun-build:aarch64
11root@184fa9ecd15d:/#

now move to the build directory, and clone the rumprun-packages repo:

1root@184fa9ecd15d:/# cd build
2root@184fa9ecd15d:/build# git clone https://github.com/cloudkernels/rumprun-packages

Make sure your config.mk file contains the correct toolchain tuple:

1root@184fa9ecd15d:/build# cd rumprun-packages
2root@184fa9ecd15d:/build/rumprun-packages# echo RUMPRUN_TOOLCHAIN_TUPLE=aarch64-rumprun-netbsd > config.mk

and go to an example package, say hello, and type make

1root@184fa9ecd15d:/build/rumprun-packages# cd hello
2root@184fa9ecd15d:/build/rumprun-packages# make

The output should be the following:

 1# make
 2mkdir -p build
 3cp src/* build
 4make -C build hello.spt
 5make[1]: Entering directory '/build/rumprun-packages/hello/build'
 6aarch64-rumprun-netbsd-gcc hello.c -o hello-rumprun
 7rumprun-bake solo5_spt hello.spt hello-rumprun
 8
 9!!!
10!!! NOTE: rumprun-bake is experimental. syntax may change in the future
11!!!
12
13make[1]: Leaving directory '/build/rumprun-packages/hello/build'
14mkdir -p bin
15cp build/hello.spt bin/hello.spt
16rumprun-bake solo5_hvt hello.hvt hello-rumprun
17
18!!!
19!!! NOTE: rumprun-bake is experimental. syntax may change in the future
20!!!
21
22make[1]: Leaving directory '/build/rumprun-packages/hello/build'
23mkdir -p bin
24cp build/hello.hvt bin/hello.hvt

Now, exit the container (Ctrl-D) and cd into this directory:

1root@rpi3:~# cd /build/rumprun-packages/hello
2root@rpi3:/build/rumprun-packages/hello#

Make sure there’s a dummy file for the disk image, and a tap interface:

1root@rpi3:/build/rumprun-packages/hello# dd if=/dev/zero of=dummy count=1 bs=512
2root@rpi3:/build/rumprun-packages/hello# ip tuntap add tap0 mode tap
3root@rpi3:/build/rumprun-packages/hello# ip link set dev tap0 up

And make sure you’ve got the solo5-hvt/spt binaries (solo5). If not, do the following:

1root@rpi3:/build# git clone https://github.com/solo5/solo5
2root@rpi3:/build# cd solo5
3root@rpi3:/build/solo5# apt-get install libseccomp-dev && make

If everything was successful, you should have two binaries: solo5-spt (seccomp tender) and solo5-hvt (KVM tender) at:

1/build/solo5/tenders/spt/solo5-spt
2/build/solo5/tenders/hvt/solo5-hvt

So, returning to the previous dir / environment, you can execute the hello unikernel:

 1root@rpi3:/build/rumprun-packages/hello# /build/solo5/tenders/spt/solo5-spt --mem=32 --net=tap0 --disk=dummy ./bin/hello.spt
 2            |      ___|
 3  __|  _ \  |  _ \ __ \
 4\__ \ (   | | (   |  ) |
 5____/\___/ _|\___/____/
 6Solo5: Memory map: 32 MB addressable:
 7Solo5:     unused @ (0x0 - 0xfffff)
 8Solo5:       text @ (0x100000 - 0x30dfff)
 9Solo5:     rodata @ (0x30e000 - 0x356fff)
10Solo5:       data @ (0x357000 - 0x3c3fff)
11Solo5:       heap >= 0x3c4000 < stack < 0x2000000
12rump kernel bare metal bootstrap
13
14[   1.0000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
15[   1.0000000]     2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
16[   1.0000000]     2018 The NetBSD Foundation, Inc.  All rights reserved.
17[   1.0000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
18[   1.0000000]     The Regents of the University of California.  All rights reserved.
19
20[   1.0000000] NetBSD 8.99.25 (RUMP-ROAST)
21[   1.0000000] total memory = 14328 KB
22[   1.0000000] timecounter: Timecounters tick every 10.000 msec
23[   1.0000080] timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
24[   1.0000090] cpu0 at thinair0: rump virtual cpu
25[   1.0000090] root file system type: rumpfs
26[   1.0000090] kern.module.path=/stand/evbarm/8.99.25/modules
27[   1.0200090] mainbus0 (root)
28[   1.0200090] timecounter: Timecounter "bmktc" frequency 1000000000 Hz quality 100
29rumprun: could not find start of json.  no config?
30mounted tmpfs on /tmp
31
32=== calling "rumprun" main() ===
33
34This is the Rumprun Hello World ...
35... using the Solo5 framework ...
36... in a Nabla container via runnc!
37
38=== main() of "rumprun" returned 0 ===
39
40=== _exit(0) called ===
41[   3.0285155] rump kernel halting...
42[   3.0285155] syncing disks... done
43[   3.0285155] unmounting file systems...
44[   3.0412055] unmounted tmpfs on /tmp type tmpfs
45[   3.0412055] unmounted rumpfs on / type rumpfs
46[   3.0412055] unmounting done
47halted
48Solo5: solo5_exit(0) called

In case you want to run the non-seccomp tender (KVM_RUN), then all you need to do is bake the hello-rumprun binary with solo5_hvt. Return to the docker environment and bake the file accordingly:

1docker run -ti --rm -v /build:/build cloudkernels/debian-rumprun-build:aarch64 /bin/bash
2root@184fa9ecd15d:/# cd /build/rumprun-packages/hello
3root@184fa9ecd15d:/build/rumprun-packages/hello# rumprun-bake solo5-hvt ./bin/hello.hvt ./build/hello-rumprun

and now, exit the docker environment (Ctrl-D) and run the hello.hvt unikernel image with the solo5-hvt binary:

 1root@rpi3:~# cd /build/rumprun-packages/hello
 2root@rpi3:~/build/rumprun-packages/hello# /build/solo5/tenders/hvt/solo5-hvt --mem=32 --net=tap0 --disk=dummy ./bin/hello.hvt
 3solo5-hvt: ./bin/hello.hvt: Warning: phdr[0] requests WRITE and EXEC permissions
 4            |      ___|
 5  __|  _ \  |  _ \ __ \
 6\__ \ (   | | (   |  ) |
 7____/\___/ _|\___/____/
 8Solo5: Memory map: 32 MB addressable:
 9Solo5:     unused @ (0x0 - 0xfffff)
10Solo5:       text @ (0x100000 - 0x30ffff)
11Solo5:     rodata @ (0x310000 - 0x359fff)
12Solo5:       data @ (0x35a000 - 0x3c6fff)
13Solo5:       heap >= 0x3c7000 < stack < 0x2000000
14rump kernel bare metal bootstrap
15
16[   1.0000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
17[   1.0000000]     2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
18[   1.0000000]     2018 The NetBSD Foundation, Inc.  All rights reserved.
19[   1.0000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
20[   1.0000000]     The Regents of the University of California.  All rights reserved.
21
22[   1.0000000] NetBSD 8.99.25 (RUMP-ROAST)
23[   1.0000000] total memory = 14322 KB
24[   1.0000000] timecounter: Timecounters tick every 10.000 msec
25[   1.0000080] timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
26[   1.0000090] cpu0 at thinair0: rump virtual cpu
27[   1.0000090] root file system type: rumpfs
28[   1.0000090] kern.module.path=/stand/evbarm/8.99.25/modules
29[   1.0200090] mainbus0 (root)
30[   1.0200090] timecounter: Timecounter "bmktc" frequency 1000000000 Hz quality 100
31rumprun: could not find start of json.  no config?
32mounted tmpfs on /tmp
33
34=== calling "rumprun" main() ===
35
36This is the Rumprun Hello World ...
37... using the Solo5 framework ...
38... in a Nabla container via runnc!
39
40=== main() of "rumprun" returned 0 ===
41
42=== _exit(0) called ===
43[   3.0408694] rump kernel halting...
44[   3.0408694] syncing disks... done
45[   3.0408694] unmounting file systems...
46[   3.0548072] unmounted tmpfs on /tmp type tmpfs
47[   3.0548072] unmounted rumpfs on / type rumpfs
48[   3.0548072] unmounting done
49halted
50Solo5: solo5_exit(0) called

Now, to run an end-to-end example, take a look at runnc, the runtime container environment to spawn Nabla containers.

1root@rpi3:/build# git clone https://github.com/cloudkernels/runnc
2root@rpi3:/build# cd runnc
3root@rpi3:/build/runnc# make container-build
4root@rpi3:/build/runnc# make container-install

Now setup the daemon.json file and restart docker

 1root@rpi3:~# cat >> /etc/docker/daemon.json << EOF
 2{
 3    "runtimes": {
 4        "runnc": {
 5                "path": "/usr/local/bin/runnc"
 6        }
 7    }
 8}
 9EOF
10
11root@rpi3:~# systemctl restart docker

And spawn an example nabla container via the docker cli:

 1root@rpi3:~# docker run --rm --runtime=runnc cloudkernels/hello-nabla:aarch64
 2Unable to find image 'cloudkernels/hello-nabla:aarch64' locally
 3aarch64: Pulling from cloudkernels/hello-nabla
 4271c6521e5a2: Pull complete 
 5Digest: sha256:309f8cd277a833958ec5055c0e7b0c7ca84c0b0413c691b8bd6392e30fbd9b71
 6Status: Downloaded newer image for cloudkernels/hello-nabla:aarch64
 7Running with args: [/opt/runnc/bin/runnc-cont -k8s -nabla-run /opt/runnc/bin/nabla-run -tap tap5cc88ff904a1 -cwd / -volume /var/run/docker/runtime-runnc/moby/5cc88ff904a1f97eaea9b0cb590575e57fce5227854ef77ecd[
 8            |      ___|
 9  __|  _ \  |  _ \ __ \
10\__ \ (   | | (   |  ) |
11____/\___/ _|\___/____/
12Solo5: Memory map: 512 MB addressable:
13Solo5:     unused @ (0x0 - 0xfffff)
14Solo5:       text @ (0x100000 - 0x30dfff)
15Solo5:     rodata @ (0x30e000 - 0x356fff)
16Solo5:       data @ (0x357000 - 0x3c3fff)
17Solo5:       heap >= 0x3c4000 < stack < 0x20000000
18rump kernel bare metal bootstrap
19
20[   1.0000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
21[   1.0000000]     2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
22[   1.0000000]     2018 The NetBSD Foundation, Inc.  All rights reserved.
23[   1.0000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
24[   1.0000000]     The Regents of the University of California.  All rights reserved.
25
26[   1.0000000] NetBSD 8.99.25 (RUMP-ROAST)
27[   1.0000000] total memory = 253 MB
28[   1.0000000] timecounter: Timecounters tick every 10.000 msec
29[   1.0000080] timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
30[   1.0000090] cpu0 at thinair0: rump virtual cpu
31[   1.0000090] root file system type: rumpfs
32[   1.0000090] kern.module.path=/stand/evbarm/8.99.25/modules
33[   1.0200090] mainbus0 (root)
34[   1.0200090] timecounter: Timecounter "bmktc" frequency 1000000000 Hz quality 100
35[   1.0200090] ukvmif0: Ethernet address 02:42:ac:11:00:02
36[   1.0695796] /dev//dev/ld0a: hostpath XENBLK_/dev/ld0a (19710 KB)
37mounted tmpfs on /tmp
38
39=== calling "/dev/shm/docker/overlay2/18f96a896573deb8de0883c46ea1b077be4dceeec10aaa00fb1d75565c642d47/merged/hello.nabla" main() ===
40
41This is the Rumprun Hello World ...
42... using the Solo5 framework ...
43... in a Nabla container via runnc!
44
45=== main() of "/dev/shm/docker/overlay2/18f96a896573deb8de0883c46ea1b077be4dceeec10aaa00fb1d75565c642d47/merged/hello.nabla" returned 0 ===
46
47=== _exit(0) called ===
48[   3.0895680] rump kernel halting...
49[   3.0895680] syncing disks... done
50[   3.0895680] unmounting file systems...
51[   3.0895680] unmounted tmpfs on /tmp type tmpfs
52[   3.0895680] unmounted /dev//dev/ld0a on / type cd9660
53[   3.0895680] unmounted rumpfs on / type rumpfs
54[   3.0895680] unmounting done
55halted
56Solo5: solo5_exit(0) called

Easy ? :D Let us know how it went!