jtcriswell / sva

Secure Virtual Architecture

SVA: Secure Virtual Architecture

Introduction:

This is the open-source release of the Secure Virtual Architecture (SVA). SVA creates an extended version of the LLVM IR that is capable of supporting a commodity operating system kernel (such as FreeBSD and Linux). By controlling the expression of operating system code, SVA can reliably control operating system kernel behavior through compiler instrumentation.

This release is the second version of SVA that works on 64-bit x86 systems and supports FreeBSD 9.0. It contains the source code for KCoFI (for enforcing control-flow integrity) and Virtual Ghost (which protects applications from a compromised operating system kernel).

License:

The file LICENSE.TXT describes the licenses under which the source code is covered.

Authors:

The file CREDITS.TXT lists individual authors of the SVA source code.

Source Code Layout:

SVAOS: The source code for the SVA-OS run-time library that implements the SVA-OS instructions.

llvm: The source code for the modified version of LLVM used for compiling the SVA-OS run-time library and the FreeBSD kernel.

freebsd9_patch: A patch that will modify the FreeBSD 9.0 kernel source code to work on SVA.

autoconf: The source code to the AutoConf configure script.

How to Compile SVA:

Given that $SRC_ROOT is the absolute pathname to the SVA source code, do the following:

o Run the configure script in the source tree to create make.conf. If you want to enable the Virtual Ghost features, add the --enable-vg option.

• cd $SRC_ROOT

• ./configure --enable-targets=host --enable-vg

o Change directory to the llvm directory and Build the modified Clang/LLVM compiler. Be sure to use GNU Make (gmake):

• cd llvm ; gmake

o Change directory to the SVA-OS subdirectory and Build the SVA-OS run-time library

• cd ../SVA ; make

o If you do not have write access to /usr/obj, create a directory for storing object files created during the kernel build and set the MAKEOBJDIRPREFIX variable to refer to this directory:

• cd $SRC_ROOT ; mkdir obj

• MAKEOBJDIRPREFIX=$SRC_ROOT/obj

o Download and extract the FreeBSD 9.0 source code:

• fetch ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/amd64/9.0-RELEASE/src.txz

• xzcat src.txz | tar -xvf -

o Apply the SVA patches to the FreeBSD source code

• cd usr/src

• patch -p1 < ../../freebsd9_patch

o Build the kernel, setting INSTKERNNAME to the name of the kernel

• make buildkernel INSTKERNNAME=svaKernel __MAKE_CONF=$SRC_ROOT/make.conf

o As the root user, install the kernel

• make installkernel INSTKERNNAME=svaKernel __MAKE_CONF=$SRC_ROOT/make.conf

o Build the C library with Ghost Memory support

• cd lib/libc
• make

o As the root user, install the new C library

• make install

o As the root user, compile the programs in the sbin directory and install them

• cd ../..
• cd sbin
• make
• make install

Incremental Kernel Compiles

Once the FreeBSD SVA kernel has been compiled, you can add the following four lines in make.conf to avoid reconfiguring the kernel and to prevent the kernel from being rebuilt from scratch:

NO_KERNELCLEAN=true

NO_KERNELCONFIG=true

NO_KERNELDEPEND=true

NO_KERNELOBJ=true

Note that the FreeBSD Makefiles do not detect when the SVA Clang compiler has been modified. If you modify the compiler, you will need to rebuild the kernel from scratch.

Running the SVA FreeBSD Kernel

The SVA FreeBSD kernel only runs in single-user mode at present. When booting, exit to the boot loader prompt (option 2 by default in the FreeBSD boot loader) and use "boot -s" to boot in single user mode. The name in the examples above is svaKernel.

sva's Issues

Interrupt Context to Trapframe Hack exposes application registers

The hacks in SVA/lib/debug.c that convert the Interrupt Context into a FreeBSD trapframe so that we don't need to recode the FreeBSD trap handling functions exposes application registers that Virtual Ghost is designed to hide (the full port of Linux 2.4.22 to SVA demonstrated that such hacks were unnecessary). Furthermore, for all SVA-based systems, it unnecessarily increases interrupt, trap, and system call latency.

Make SVA Support Thread Local Storage (TLS)

The current SVA implementation does not support Thread Local Storage (TLS). This requires us to modify the C library so that malloc() does not use TLS. This, in turn, is a problem as later FreeBSD C libraries do not provide a simple option to disable TLS.

The SVA implementation should be enhanced so that existing binaries that use TLS work.

It is not necessary to call swapgs at the end of Trap and SVAsyscall

It is not necessary to call swapgs in the following two places, since we are returning back to user space:
https://github.com/jtcriswell/SVA/blob/master/SVA/lib/handlers.S#L392
https://github.com/jtcriswell/SVA/blob/master/SVA/lib/handlers.S#L667

Error when building llvm

Hello!
I'm trying to follow the instruction in README.md to install SVA, but an error occured when running 'make' in ${REPO_DIR}/llvm. The logs are as follows:

llvm[2]: Constructing LLVMBuild project information.
Traceback (most recent call last):
  File "/home/yan_ice/Desktop/Teecert/SVA/llvm/utils/llvm-build/llvm-build", line 3, in <module>
    import llvmbuild
  File "/home/yan_ice/Desktop/Teecert/SVA/llvm/utils/llvm-build/llvmbuild/__init__.py", line 1, in <module>
    from main import main
ModuleNotFoundError: No module named 'main'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/yan_ice/Desktop/Teecert/SVA/llvm/lib/DebugInfo'
make[1]: Leaving directory '/home/yan_ice/Desktop/Teecert/SVA/llvm/lib'
make[1]: Entering directory '/home/yan_ice/Desktop/Teecert/SVA/llvm/tools/llvm-config'
llvm[1]: Constructing LLVMBuild project information.
Traceback (most recent call last):
  File "/home/yan_ice/Desktop/Teecert/SVA/llvm/utils/llvm-build/llvm-build", line 3, in <module>
    import llvmbuild
  File "/home/yan_ice/Desktop/Teecert/SVA/llvm/utils/llvm-build/llvmbuild/__init__.py", line 1, in <module>
    from main import main
ModuleNotFoundError: No module named 'main'
llvm[1]: Compiling llvm-config.cpp for Release+Asserts build
llvm-config.cpp:45:10: fatal error: 'LibraryDependencies.inc' file not found
#include "LibraryDependencies.inc"
         ^~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

It seems that there is something wrong with my python environment. I've tried to find some solutions about it, but they were all failed.
I'm using Ubuntu 20.04, and my python version is 3.9.16 with minoconda installed. How to solve it?
Thanks for your help!

SVA-OS trap registration instructions missing bounds checks

The sva_register_general_exception() and sva_register_interrupt() instructions are missing bounds checks on the interrupt/trap number passed in from the operating system kernel.

Building /sbin fails when following build directions

Building /sbin fails when following the build directions in README.md. The problem is that one of the network utilities includes proc.h from the kernel which, in turn, includes an SVA header file which cannot be found because the user-space Makefiles aren't looking in the right directories for SVA header files.

Ghost Memory deallocation assumes Ghost Memory is mapped.

The Virtual Ghost Ghost Memory deallocator "hypercall" assumes that all Ghost Memory is backed by physical memory. This is not necessarily the case.

Is there any document about mips JIT?

Hello dear everyone,
Recently I want to add JIT in RISCV, but I don't know how to do it. I notice that here is the definition of MIPS JIT [1], and I want to ask you that if there is any document about the implementation of MIPS JIT?

[1]https://github.com/jtcriswell/SVA/blob/master/llvm/lib/Target/Mips/MipsCodeEmitter.cpp

Best regards
Many thanks to you
William

SVA-OS Intrinsics Should Vet State ID

SVA-OS intrinsics that take, as input, a state ID number should vet that the ID is valid. At least one intrinsic (sva_release_stack()) does not; there may be others.

Make SVAThread Allocation Dynamic

The current SVA implementation statically allocates SVAThread structures; it is possible to run out of SVAThreads even if the system has ample physical memory. The SVA implementation should be changed so that SVAThreads are allocated and freed dynamically.

the check in trap_pfault_sva() in freebsd src sys/amd64/amd64/trap.c is incorrect

We have moved the mapping of ghost memory to another place and we should reflect the change in trap_pfault_sva():

• if ((0xffffff0000000000u <= va) && (va < 0xffffff8000000000u)) {

• if ((0xfffffd0000000000 <= va) && (va < 0xfffffd8000000000)) {

SVA ghost memory free hyper call does not free all ghost memory

The SVA VM "hyper call" for deallocating ghost memory assumes that it is only freeing a single page of ghost memory. Other ghost memory pages remain allocated. While this may have originally been intended, it really should accept an arbitrary size of ghost memory to free and free it.

The sva_mm_load_pgtable() intrinsic fails to flush TLB when swapping Ghost Memory

The sva_mm_load_pgtable() SVA-OS instruction fails to flush the TLB after modifying the PML4E entry that maps ghost memory for the currently running thread. This could cause a thread to access another thread's ghost memory if the processor somehow loads a TLB entry with the old PML4E entry before sva_mm_load_pgtable() finishes execution.

setoperand() causes error in LLVM4.0.1 or newer version

SVA/llvm/lib/Transforms/Scalar/SFI.cpp

Line 523 in 38e6832

SI.setOperand (1, newPtr);

For a LoadInst or StoreInst I, we get its pointer by I->getPointerOperand(). After some bit-masking operations on the pointer we updated the pointer in the Instruction by I->setoperand().

In the current commit, it's done by I->set(0, newPtr); however, this would cause error for LLVM 4.0.1 or newer versions. There is no compilation error, but when we use opt to optimize programs, this would throw errors like i32Stored value type does not match pointer operand type!.

Current implementation of getPointerOperand() shows that the pointer operand is the second one (see http://llvm.org/doxygen/Instructions_8h_source.html#l00402). So we should use I->setoperand(1, newPtr) to update the bit-masked address in new LLVM.

CFI Checks Permit User-Space Targets

The CFI instrumentation does not add code to set the higher-order bits of an indirect branch target if the branch instruction reads its operand from memory. The purpose of setting these higher-order bits is to ensure that the target of the indirect branch resides in the kernel's code segment (as opposed to a user-space code segment).

Configure script requires that many options be explicitly specified

The Autoconf configure script for SVA, being written at the last minute and at great expense, requires that many of the --enable and --disable options be specified explicitly on the configure command line instead of having defaults. This is annoying and should be fixed.

Problem with allocPTPage() in SVA/lib/mmu.c

1. Here https://github.com/XiaowanDong/SVA/blob/master/SVA/lib/mmu.c#L1006, p is a pointer, so it should be compared against NULL instead of 0.
2. This line https://github.com/XiaowanDong/SVA/blob/master/SVA/lib/mmu.c#L1030 should be moved into the if statement above it.

Is this the KCoFI version of SVA or the memory safety version?

Hi @jtcriswell is this the KCoFI source?

If so we should mention that in the read me.

randomNumber should properly loop after RDRAND call if CF Flag is unset

Per the Intel Software Developer's Manual, RDRAND will set the CF flag to indicate success or failure (e.g.: in case of insufficient entropy) of the instruction.

The inline assembly in randomNumber() does check this flag (via JAE), but the resulting jump is to location 0x1, rather than label "1". Perhaps this could inspire a relevant fix.

Also, note sample code that uses a fixed number of retries.

SVA fails to deallocate ghost memory upon process termination

When a process exits, the sva_release_stack() intrinsic does not deallocate the physical memory used for ghost memory. Instead, it merely unmaps it and releases page table pages that are no longer needed for mapping the now unused ghost memory.

Must recompile libc with TLS disabled

The libc library must be recompiled with thread-local storage (TLS) disabled in order to work on SVA. This requires recompiling init and possibly mount and sh as well since they are statically linked.

Freeing Ghost Memory Causes Kernel Panic

An attempt to deallocate ghost memory (either on process exit or explicitly via a hyper call) causes a panic. The problem is that the unmapSecureMemory() function within SVA (Virtual Ghost) assumes that the page tables it should use are from the current process. However, in some cases, the ghost memory to free belongs to another process.

Configuration macros not used in handlers.S

The handlers.S file does not include config.h, preventing Virtual Ghost configuration options from enabling code within the assembly code. Virtual Ghost features in C code including config.h are conditionally compiled properly.

Incorrect bound checking in secmem.c

The function freeSecureMemory() in secmem.c tries to check if a value is within a range by chaining the comparisons together (SECMEMSTART <= pint < SECMEMEND).