torokernel / torokernel

This repository contains the source code of toro unikernel

ToroMicroVM

Introduction

ToroMicroVM is a unikernel dedicated to deploy microservices as microVMs. ToroMicroVM leverages on virtio-fs and virtio-vsocket to provide a minimalistic architecture. Microservices are deployed as Toro guests in which binaries and files are distributed in a Ceph cluster. The common fileystem allows to easely launch microvms from any node of the cluster.

Features

• Support x86-64 architecture
• Support up to 512GB of RAM
• Support QEMU-KVM microvm and Firecracker
• Cooperative and I/O bound threading scheduler
• Support virtio-vsocket for networking
• Support virtio-fs for filesystem
• Fast boot up
• Tiny image
• Built-in gdbstub

How try ToroMicroVM?

You can quickly get a first taste of ToroMicroVM by running the HelloWorld example by building a docker image that includes all the required tools. To do so, execute the following commands in a console (These steps require KVM and Docker):

wget https://raw.githubusercontent.com/torokernel/torokernel/master/ci/Dockerfile
sudo docker build -t torokernel-dev .
sudo docker run --privileged --rm -it torokernel-dev
cd examples/HelloWorld
python3 ../CloudIt.py -a HelloWorld

If these commands execute successfully, you will get the output of the HelloWorld example. You can also pull the image from dockerhub instead of building it:

sudo docker pull torokernel/toro-kernel-dev-debian-10
sudo docker run --privileged --rm -it torokernel/toro-kernel-dev-debian-10

You can share a directory from the host by running:

sudo docker run --privileged --rm --mount type=bind,source="$(pwd)",target=/root/torokernel-host -it torokernel/toro-kernel-dev-debian-10

You will find $pwd from host at /root/torokernel-host in the container.

How build ToroMicroVM locally?

Step 1. Install Freepascal 3.2.0

wget https://sourceforge.net/projects/lazarus/files/Lazarus%20Linux%20amd64%20DEB/Lazarus%202.0.10/fpc-laz_3.2.0-1_amd64.deb/download
mv download fpc-laz_3.2.0-1_amd64.deb
apt install ./fpc-laz_3.2.0-1_amd64.deb -y

Step 2. Build Qemu-KVM (qemu 5.2.50 or #51204c2f)

apt-get update
apt-get install python3-pip make git libcap-dev libcap-ng-dev libcurl4-gnutls-dev libgtk-3-dev libglib2.0-dev libpixman-1-dev libseccomp-dev -y
pip3 install ninja
# uncomment to change PATH permanently
# echo 'export PATH=/home/debian/.local/bin:$PATH' >>~/.bashrc
export PATH="/home/debian/.local/bin:$PATH"
git clone https://github.com/qemu/qemu.git qemuforvmm
cd qemuforvmm
git checkout 51204c2f
mkdir build 
cd build
../configure --target-list=x86_64-softmmu
make

Step 3. Get ToroMicroVM

git clone https://github.com/torokernel/torokernel.git

Step 4. Get the RTL for ToroMicroVM

git clone https://github.com/torokernel/freepascal.git -b fpc-3.2.0 fpc-3.2.0

Note that Step 1, 2, 3 and 4 can be found in the script at ci/prepare_host.sh.

Step 5. Edit path to Qemu and FPC in CloudIt.py

Go to torokernel/examples and edit CloudIt.py to set the correct paths to Qemu and fpc. Optionally, you can install vsock-socat from here.

Run the HelloWorld Example

You have to go to examples/HelloWorld/ and execute:

python3 ../CloudIt.py -a HelloWorld

Run the StaticWebServer Example

You can easily get the StaticWebServer up and running by following the tutorial at here. This would require only a Debian 10 installation. For example, you can get a s1-2 host from OVH. If you prefer to run it step by step, follow the next instructions. You have first to compile vsock-socat and virtiofds. The latter is built during the building of Qemu. The former can be built by executing:

git clone [email protected]:stefano-garzarella/socat-vsock.git
cd socat-vsock
autoreconf -fiv
./configure
make socat

Then, launch vsock-socat by executing:

./socat TCP4-LISTEN:4000,reuseaddr,fork VSOCK-CONNECT:5:80

In a second terminal, execute:

./virtiofsd -d --socket-path=/tmp/vhostqemu1 -o source=/root/qemulast/build/testdir/ -o cache=always

Replace source with the directory to serve. Finally, launch the static webserver by executing:

python3 ../CloudIt.py -a StaticWebServer

Run the Intercore Communication example

This example shows how cores can communicate by using the VirtIOBus device. In this example, core #0 sends a packet to every core in the system with the ping string. Each core responds with a packet that contains the message pong. This example is configured to use three cores. To launch it, simply executes the following commands in the context of the container presented above:

python3 ../CloudIt.py -a InterCoreComm

You will get the following output:

Building Toro in Windows by using Lazarus

First you have to follow this tutorial to get a FPC cross-compiler from Windows to Linux. Then, you have to execute the following script which compiles the RTL for Toro and outputs the generated files in the x86_64-linux directory. Note that this script overwrites the RTL for Linux, which is used when the -TLinux parameter is passed. This script requires three paths to set up:

1. fpcrtlsource, which is the path to the repository from https://github.com/torokernel/freepascal
2. fpcrtllinuxbin, which is the path to the cross-compiled linux RTL
3. fpcbinlinux, which is the path to the fpc compiler.

fpcrtlsource="c:\Users\Matias\Desktop\fpc-3.2.0\rtl"
fpcrtllinuxbin="c:\fpcupdeluxefortoromicrovm\fpc\bin\x86_64-linux"
fpcbinlinux="c:\fpcupdeluxefortoromicrovm\fpc\bin\x86_64-win64"

$fpcbinlinux/ppcx64.exe -TLinux -dFPC_NO_DEFAULT_MEMORYMANAGER -dHAS_MEMORYMANAGER -uFPC_HAS_INDIRECT_ENTRY_INFORMATION -dx86_64 -I$fpcrtlsource/objpas/sysutils/ -I$fpcrtlsource/linux/x86_64/ -I$fpcrtlsource/x86_64/ -I$fpcrtlsource/linux/ -I$fpcrtlsource/inc/ -I$fpcrtlsource/unix/ -Fu$fpcrtlsource/unix/ -Fu$fpcrtlsource/linux/ -MObjfpc $fpcrtlsource/linux/si_prc.pp -Fu$fpcrtlsource/objpas -Fu$fpcrtlsource/inc -FE$fpcrtllinuxbin

$fpcbinlinux/ppcx64.exe -Us -TLinux -dx86_64 -I$fpcrtlsource/objpas/sysutils/ -I$fpcrtlsource/linux/x86_64/ -I$fpcrtlsource/x86_64/ -I$fpcrtlsource/linux/ -I$fpcrtlsource/inc/ -I$fpcrtlsource/unix/ -Fu$fpcrtlsource/unix -Fu$fpcrtlsource/linux -Fu$fpcrtlsource/objpas -Fu$fpcrtlsource/inc $fpcrtlsource/linux/system.pp -FE$fpcrtllinuxbin

Then, you have to go to Lazarus and open the project HelloWorld.lpi. You are able to compile the project from compile.

Create your own distributed filesystem with CephFS

To create a CephFS cluster you can follow these instructions.

Contributing

You have many ways to contribute to Toro. One of them is by joining the Google Group here. In addition, you can find more information here.

License

GPLv3

References

[0] A Dedicated Kernel named Toro. Matias Vara. FOSDEM 2015.

[1] Reducing CPU usage of a Toro Appliance. Matias Vara. FOSDEM 2018.

[2] Toro, a Dedicated Kernel for Microservices. Matias Vara and Cesar Bernardini. Open Source Summit Europe 2018.

[3] Speeding Up the Booting Time of a Toro Appliance. Matias Vara. FOSDEM 2019.

[4] Developing and Deploying Microservices with Toro Unikernel. Matias Vara. Open Source Summit Europe 2019.

[5] Leveraging Virtio-fs and Virtio-vsocket in Toro Unikernel. Matias Vara. DevConfCZ 2020.

[6] Building a Cloud Infrastructure to Deploy Microservices as Microvm Guests. Matias Vara. KVM Forum 2020.

[7] Running MPI applications on Toro unikernel. Matias Vara. FOSDEM 2023.

[8] Is Toro unikernel faster for MPI?. Matias Vara. FOSDEM 2024.

torokernel's Issues

Investigate RemoteDebugging through Lazarus

http://wiki.freepascal.org/Remote_Debugging

ext2 driver only supports 4mb files

Currently the ext2 driver does not handle double indirect inodes.

Support for FAT32 fs

Investigate the use of NUMA in qemu

Investigate toro support for NUMA emulated hardware in Qemu.

Investigate the use of MOVNTDQA when accessing MMIO regions

To rename WriteConsole()

To rename WriteConsole() to show when the output is formatted or not.

Ping from host to guest does not work

To check this.

Test1

Investigate the use op no-ops operations in loops

Buffer-Cache should implement different kind of cache

The Buffer cache should allow to setup different kind of cache depending on the use. For example, it should be possible to define a cache that is "write-through" in order to force the updating to disk immediately. It could be nice also to document the buffer mechanism.

Investigate networking in KVM

To investigate how make networking work by a KVM guest.

Add boot.bin to repo

Boot.bin should come by default when repo is clone it. This would avoid to download YASM.

Investigate the use of torokernel in Xen

The idea is to demonstrate the use of torokernel in Xen. I tried with vmdk format and it can be imported from XenCenter (HVM guest). However, the kernel crash after detecting the CPUs. We should verify why it crashes. It could be interesting to read the serial console.

Toro in KVM

To check PrintDecimal() procedure in the Console

Console.pas has the function PrintDecimal() which has an obscure implementation details around if (Len <> 10) then It is not clear what the function is doing when dealing with a 10 digit number. This needs to be investigated.

cannot build under fpc 3.0.0 on Windows

rtl cannot be compiled with fpc 3.0.0

Add support for a DHCP

This is needed to support to redirect port. In that case qemu runs a dhcp server that provides the ip for guest. We need to investigate how DHCP can be implemented.

Done:

• Toro is able to configure the ip by using a dhcp.

Investigate what is needed to make e1000 works

I have implemented the driver for e1000, however, last tests seem that it does not work yet. I have to investigate what is needed to do to make it work.

Toro on KVM does not detect multiple cores

Investigate the use of containers to build Toro

Investigate the use of fpprofiler for toro code

https://github.com/graemeg/fpprofiler

Add wiki entry about how to compile tests

Adapt CloudIt.sh to run with KVM

This work is in progress in the branch toroonkvm

EXT2 driver does not support writing operations

Test examples should compile directly from Lazarus

Create a project for every test and make compile directly from lazarus IDE

Investigate Toro in KVM

First tests seems very interesting.

Create watchdog for a thread

It should be possible to create watchdogs for certain threads in order to do something if they spend too much time without calling the scheduler.

Add CI for tests

Add an script used by travis to compile tests after push. This represents a first step towards the use of CI in torokernel.

e1000 transmission is very slow

By pinging to the VM, I saw that ne2000 driver is much more faster than e1000. There could be a problem in the transmission of packets in the e1000 driver.

Investigate CI for torokernel

I have to automate the execution of tests before a branch is merged. The examples should be written in a way that can run as tests and gets results which should be validated.

Review how delay() work and how time is managed

We have to review how the delays are calculated.

Rename tests for examples

Add support for DHCP client

This is needed to run QEMU in KVM. It is not mandatory to set up a DHCP client but it will ease a lot the configure tcp-ip stack parameters.

Write a blog entry talking about writing operations

Build makes an exception when file is not found

Build should catch the exception and shows the corresponding message

Custom usage of ToroKernel

Is there a means to specify the upper bounds of the memory which is available to the heap? I'd like to reserve say 1MB as an I/O region accessible to the qemu monitor. Thanks, Mark.

cannot compile toro by using fpc 3.0.0 in Linux

cannot compile toro by using fpc 3.0.0 in Linux

Add test example that uses the filesystem

Add .xml to import examples directly from KVM command line

Add Readme to repo

Move pci detection to a new unit

Currently pci detection happens in Filesystem unit which it is not a good place to do it. Instead, we should move the code related with the pci bus to a new unit.

DoD:

• Pci detection is moved to a new unit

Investigate StackTrace

To investigate stacktrace to implemented in Toro

Investigate toro template for Lazarus

Update wiki to explain the population of ext2 images in Windows

The wiki must be updated with all the needed information to compile and run TorowithFilesystem.pas example. It is necessary to explain how to use genext2fs.exe.

Stack does not implement ACK + RST packet

The tcp stack just ignores such packets. This is important, for example, when it is trying to connect to a port which is not listening. TORO stack must support such a behavior.

Move tools to tools folder

Tools like boot.bin, build.pas and other tools should be in tools folder. Update the projects in examples to point to the right tools. This ticket should be done after #27

Replace the simple list of network packages for something more optimized

Create a tool in Windows to create images by using ext2 fs

Code memory region must be read-only

Create a plugin for Lazarus to see serial console from QEMU