torokernel / torov

ToroV allows user's applications to run as VMs and to communicate with the host OS by relying on a POSIX hypercall interface.

ToroV

ToroV enables applications to run as Virtual Machines. It is a Virtual Machine Monitor (VMM) that exposes a POSIX API to the guest. The guest communicates with the VMM by using syscalls.

Architecture

The architecture is made of three components: the guest, the Runtime Library (RTL) and the VMM. The guest is a normal user application that is compiled with the RTL. The RTL contains the required code to run the application as a guest. For example, it allows to correctly boots the application. The guest is an user application that requires services from the OS by using syscalls. In ToroV, the VMM acts as the OS that provides such services. When the application guest invokes a syscall, this produces a VMExit that the VMM catches, processes, and finally returns to the application. This technology is similar than gVisor. The main difference with ToroV is that in gVisor syscalls are first trapped by the guest os, and then, forward them to the host. In ToroV, syscalls are trapped by the host first.

Features

• Configurable syscalls per application
• Fast migration of applications
• Fast booting time
• Reduced memory footprint
• POSIX interface
• Fast syscalls

Drawbacks

In ToroV, applications trigger a VMEXIT by using the out instruction. This instruction replaces the use of the syscall instruction. This requires that applications are compiled with a STDLIB in which the syscall instruction has been replaced.

How to try it?

You require a Linux host with KVM to run the VMM. To check if KVM is enabled, you can execute lsmod to list the loaded module. If KVM is in the list, you can move forward, if not, you need to first install it.

Try by using a ToroV Docker image (Recommended)

To simple try ToroV, you can build an image in docker with the required tools to build the vmm and the examples. First, you have to build the docker image by running:

wget https://raw.githubusercontent.com/torokernel/torov/master/ci/Dockerfile
docker build -t torov-dev .

Then, run the HelloWorld example by running:

docker run --privileged -it torov-dev
cd examples/HelloWorld
../build.sh HelloWorld
../../src/vmm/vmm helloworld.json

Note that docker runs with the --privileged flag to be able to use Kvm from the container.

Try step by step

Step 0. Clone ToroV

git clone [email protected]:torokernel/torov.git

Step 1. Install Freepascal 3.2.0

wget https://sourceforge.net/projects/lazarus/files/Lazarus%20Linux%20amd64%20DEB/Lazarus%202.0.10/fpc-laz_3.2.0-1_amd64.deb/download
mv download fpc-laz_3.2.0-1_amd64.deb
apt install ./fpc-laz_3.2.0-1_amd64.deb -y

Step 2. Get the RTL for ToroV

git clone https://github.com/torokernel/freepascal.git -b fpc-3.2.0-for-torov fpc-3.2.0-for-torov

Step 3. Edit path in build.sh

Go to torov/examples and edit build.sh to set the correct paths to fpc. The path corresponds with the directory where the freepascal compiler is stored in step 2.

Step 4. Build the VMM

Go to torov/src/vmm and run build.sh. This generates the binary named vmm which contains the VMM.

Step 5. Build HelloWorld example

First, go to torov/src/rtl and execute:

fpc -s ToroVSys.pas

Second, go to torov/examples and execute:

nasm -f elf64 boot.s -o boot.o

Finally, go to torov/examples/HelloWorld/HelloWorld.ld.elf and edit the path to freepascal RTL objects. Then, run:

../build.sh HelloWorld

If the command successes, it generates three files: HelloWorld.elf, HelloWorld.bin and HelloWorld.dbg. You can run this example by running:

../../src/vmm/vmm ./helloworld.json

You will get something like:

Hello World, I am ToroV!

How to debug an application

You can debug your application by using a gdb client. To do this, follow the steps:

Step 1. Generate debug symbols

Edit torov/examples/build.sh, uncomment line 5, and comment line 6. Then, compile the HelloWorld example from its directory:

../build.sh HelloWorld

Step 2. Run the VMM with a GdbStub

You have to edit helloworld.json and set the Allowed to true in the debug section. Then, run the VMM with the gdbstub:

../../src/vmm/vmm helloworld.json

The gdb server waits for the gdb client at port 1234.

Step 3. Launch the gdb client

gdb HelloWorld.dbg
target remote localhost:1234
c

Profiling

In the folder torov/src/vmm, you can find the script profile.py that is meant to measure the running time of any application. To run it, you have just to save the binary and the correspoding json in this directory, and then run the script as follows:

python3.5 ./profile.py 1000 helloworld.json

In this case, the test measures the average running time of 1000 executions of the HelloWorld example. The scripts outputs a gnuplot command to plot the result.

References

[0] ToroV, a kernel in user-space, or sort of. FOSDEM 2022.

[1] ToroV, a kernel in user-space to deploy server-less applications. 17th Workshop on Virtualization in High-Performance Cloud Computing.

License

GPLv3

torov's Issues

Set up idt and gdt

See kvm_setup_gdt() at https://github.com/torvalds/linux/blob/c00c5e1d157bec0ef0b0b59aa5482eb8dc7e8e49/tools/testing/selftests/kvm/lib/x86_64/processor.c

Add seccomp mode to only allows exit(), read() and write() to already-open file descriptors

Copy_to/from the user requires verification

The copying from and to the user requires verification to prevent writing to invalid memory regions.

Add an option to set the max memory

Set up user space (ring3)

Use ioeventfd instead of VMExit for syscalls

This issue shall replace the current mechanism of trapping syscalls that is based on a heavy VMExit with ioeventfd. By using this mechanism, the VMM will be notified asynchronously that the app requires a syscall. This happens when the app writes to a specific mmio register. This is the mechanism used in vhost and for the virtqueue notification. This seems to perform better than user-space VMExits. The new mechanism would be:

1. app guest writes to special mmio reg
2. Event is triggered in the thread that processes eventfd
3. The thread processes the request and tells the app guest that is done
4. app guest resumes execution. Here I am sure how the VMM should tell the guest that the syscall is done.

• Implement eventfd in fpc linux unit (https://fpc-pascal.freepascal.narkive.com/w3YSd13n/using-eventfd)
• Add struct kvm_ioeventfd
• Implement int ioeventfd__init(struct kvm *kvm)
• Implement static int ioeventfd__start(void)

[1] see virtio_mmio_init_ioeventfd() at https://github.com/clearlinux/kvmtool/blob/b5891a4337eb6744c8ac22cc02df3257961ae23e/virtio/mmio.c
[2] see ioeventfd__add_event() at https://github.com/clearlinux/kvmtool/blob/master/ioeventfd.c
[3] http://blog.allenx.org/2015/07/05/kvm-irqfd-and-ioeventfd

Document guest memory maping

To document the following layout:

Set TSS

See https://patchwork.kernel.org/project/kvm/patch/[email protected]/

To rely on json to configure the vmm

The vmm shall rely on a .json that is passed as a command-line argument to get the parameters to deploy the application.

Locate binary at 0x400200

To be agreed with the binary memory layout for Linux, the user binary must be located at 0x400200. The heap shall be located at 0x600000.