intellabs / kafl.linux

Linux kernel branches for confidential compute research

Linux branches for fuzzing with kAFL

WARNING - this project contains experimental and unsupported software for purpose of research. Do not use for production.

Overview:

• kvm-nyx-N tracks the Nyx KVM tree at https://github.com/nyx-fuzz/KVM-Nyx/
• kafl/sdv-N are variants of the Nyx KVM tree with TDX 'SDV' emulation + kAFL support
• guest tracks upstream TDX guest kernel at https://github.com/intel/tdx/
• kafl/fuzz-N are variants of Linux guest kernels for fuzzing with kAFL

See also:

• IntelLabs/kAFL
• KVM-Nyx
• Linux Kernel Hardening for Intel TDX

kafl.linux's Issues

Regression: libxdc decode error when fuzzing with 6.0 Nyx kernel

We've spotted a regression starting from the 6.0 Nyx kernel where libxdc would continously throw libxdc_decode errors as soon as fuzzing starts for certain targets:

A complete repro is available for the Zephyr target, in development:

git clone https://github.com/IntelLabs/kAFL.git -b regression/6.0_tnt_zephyr
cd kAFL
make deploy
# let kafl setup, reboot ...
cd kAFL/kafl/examples/zephyr_x86_32
docker build -t kafl_zephyr .
docker run -ti --rm -v $PWD:/workdir -v $PWD/../nyx_api.h:/usr/local/include/nyx_api.h kafl_zephyr
./run.sh build TEST
# CTRL-D (exit container)
./run.sh fuzz

On the previous 5.10.73 kernel, these errors doesn't appear.

ping @schumilo, since you worked on the 6.0 patches, do you have any idea what could be the cause ? (the main changes with 6.0 is that we remove custom FDL code)

kvm-nyx-6.0: ERROR: vmx_set_fdl_addr

When testing the new kvm-nyx-6.0 branch with kAFL, we realized that dmesg was spammed with this error message:

ERROR: vmx_set_fdl_addr ffffe000

Originated from arch/x86/kvm/vmx/vmx_fdl.c

The debian package can be found as a build artefact here:
https://github.com/IntelLabs/kafl.linux/actions/runs/3948787949

cc @schumilo, if you have any ideas regarding that code section and why this error message is being triggered, feel free !

Cannot allocate main ToPA buffer!

Sometimes kAFL workers will raised an assertion failure message in QEMU:
https://github.com/IntelLabs/kafl.qemu/blob/kafl_stable/nyx/pt.c#L327

This is due to an failed allocation in the linux kernel, and looking at dmesg logs, a few messages
like Cannot allocate main ToPA buffer! appears.

These messages originates from here:
https://github.com/IntelLabs/kafl.linux/blob/kvm-nyx-5.10.73/arch/x86/kvm/vmx/vmx_pt.c#LL775C15-L775C48

A possible fix is to drop your kernel file cache:

sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

The flag __GFP_NOWARN might be in question for this allocation failure.
See also kernel memory allocation guide

[KVM-NYX] Error: IP Payloads are LIP!

While trying to follow the installation process, kafl fuzz fails with

Intel PT is not supported on this CPU!

error. Permissions to /dev/kvm should be set correctly. In my dmesg output there is BOOT_IMAGE=/boot/vmlinuz-6.0.0-nyx+ but also [KVM-NYX] Error: IP Payloads are LIP! error. I use a fairly new CPU - i5-1335U and Intel PT should be, in fact, supported. Is there any way how can I resolve this?