Fast system calls in dynamically linked programs

To enable fast system calls in dynamically linked binaries, we load at runtime a C library that is "unikernel-aware" and employs function calls into the kernel for system calls invocations. This C library is a special version of Musl libc that can be created fully automatically from vanilla Musl. This is done with the Coccinelle code transformation tool. To build it and test it out, use the following steps.

First, install coccinelle on your system, for Debian/Ubuntu:

sudo apt install coccinelle

In your local installation of HermiTux, checkout the htux-substitution branch for the Musl source directory (if this directory is absent just compile everything once by running make in the root folder):

cd musl
git checkout htux-substitution

The coccinelle semantic patch describing the code transformation rules to "unikernelize" Musl for HermiTux is in hermitux/hermituxify.cocci. We apply it to the Musl code base in order to generate the concrete patch used next to actually apply the transformation:

cd musl
spatch --sp-file hermitux/hermituxify-x86-64.cocci . > hermitux.patch
patch -p1 < hermitux.patch

You should use the right semantic patch according to your architecture:

• hermitux/hermituxify-x86-64.cocci for x86-64
• hermitux/hermituxify-aarch64.cocci for aarch64

Once the codebase is patched a last thing our unikernel-aware libc needs is to know the location of each syscall implementation's code in the virtual address space. This generally changes each time we compile a kernel, so there is a script to generate a source file containing that information.

To use that script, first edit it and set the HERMITUX_BASE variable to the folder containing the base HermiTux directory:

vim musl/hermitux/htux_syscalls_generate

Next make sure the kernel binary is compiled and installed:

make -C hermitux-kernel/build install -j`nproc`

Run the script and have its output written into the corresponding source file:

cd musl/hermitux
./htux_syscalls_generate.sh x86-64 > ../src/misc/htux_syscalls.c.generated

Set the script's parameter according to the target architecture: x86-64 or aarch64.

Finally compile and install musl:

cd musl
make install -B -j`nproc`

Note that is is important to fully recompile Musl (-B flag for the make command) each time the system call location information is regenerated (which needs to be done each time the kernel is recompiled.

Next one can use the test-dyn target of the template Makefile to launch dynamically compiled binaries with our transformed version of Musl. Note that in the current state of things, using that template Makefile to compile static binaries (for example with make test) will succeed and create static binaries with that same unikernelized libc: they will have fast syscalls but will not work on Linux.