mirage / ocaml-solo5

Freestanding OCaml runtime

ocaml-solo5 -- OCaml compiler with Solo5 backend

This package provides a OCaml compiler suitable for linking with a Solo5 base layer.

License and contributions

All original contributions to this package are licensed under the standard MIT license.

This package incorporates components derived or copied from musl libc, OpenBSD, OpenLibm and other third parties. For full details of the licenses of these third party components refer to the included LICENSE file.

The OCaml runtime ("OCaml Core System") built by this package is distributed under the terms of the GNU LGPL version 2.1 with a special exception for static or dynamic linking to produce an executable file. For details refer to the LICENSE file included in the version of the ocaml-src OPAM package installed on your system as a dependency when you build this package.

Components

The following components are built and installed:

In PREFIX/solo5-sysroot/bin:

• ocamlc: a bytecode OCaml compiler configured for the chosen target. Please note that the bytecode runtime is not supported.
• ocamlopt: a native OCaml compiler configured for the chosen target.

In PREFIX/solo5-sysroot/lib/ocaml:

• libasmrun.a, libotherlibs.a: OCaml native code runtime.
• Compiler libraries.
• In caml/: Header files for the OCaml runtime.

In PREFIX/solo5-sysroot/lib/nolibc:

• libnolibc.a: libc interfaces required by the OCaml runtime.
• libopenlibm.a: libm required by the OCaml runtime.

In PREFIX/solo5-sysroot/include/nolibc:

• Header files for nolibc and openlibm.

In PREFIX/lib/findlib.conf.d:

• solo5.conf: ocamlfind definition of the cross-compilation switch.

Usage

The installed compiler is able to build solo5 executables. The solo5 bindings (xen, hvt, spt, ...) is chosen at link time, using the solo5-specific -z solo5-abi=XXX compiler/linker option. Linking an executable with no bindings results in a dummy executable.

To build with the Solo5 compiler toolchain, it has to be selected using ocamlfind or dune:

• ocamlfind: ocamlfind -toolchain solo5 ...
• dune: dune build -x solo5, or add the toolchain in a build context in the dune workspace file.

Example

The example describes the minimal structure needed to build an ocaml-solo5 executable with dune, linked with the hvt bindings by default. It requires an application manifest and a startup file to initialize the libc.

Build: dune build -x solo5 Run: solo5-hvt _build/solo5/main.exe

Supported compiler versions

Tested against OCaml 4.12.1 through 4.14.1. Other versions may require changing configure.sh.

Porting to a different (uni)kernel base layer

Assuming your unikernel base layer is packaged for OPAM in a similar fashion to Solo5 this should be as simple as:

1. Adding the appropriate clauses to determine the OPAM packages required and MAKECONF_CFLAGS for compilation to configure.sh.
2. Implementing a nolibc/sysdeps_yourkernel.c.

Note that the nolibc code is intentionally strict about namespacing of APIs and header files. If your base layer exports symbols or defines types which conflict with nolibc then the recommended course of action is to fix your base layer to not export anything defined by "POSIX" or "standard C".

Updating the vendored copy of OpenLibm

OpenLibm is "vendored" into this repository using git subtree:

git subtree add --prefix openlibm https://github.com/JuliaLang/openlibm.git v0.5.4 --squash

To update the vendored copy of OpenLibm to the newer upstream version TAG, use the following command on a branch and then file a PR:

git subtree pull --prefix openlibm https://github.com/JuliaLang/openlibm.git TAG --squash

ocaml-solo5's Issues

Compilation fail into a Docker container (bubblewrap)

Currently, with Ubuntu 18.06 as the initial Docker container and OPAM > 2.0.0 with bubblewrap, it's not possible to install properly ocaml-freestanding. I did not find a solution to fix that yet (I tried OPAM_USER_PATH_RO, only available with OPAM > 2.1.0) than a first opam init --disable-sandboxing command.

`ocaml-freestanding.0.6.5` does not work on `alpine`

See this error when we want to install it:

C99 float ops unavailable, enable replacements with --enable-imprecise-c99-float-ops

Use `ocaml-ci` on this repository

Free memory estimation failed (again) ?

With qubes-mirage-fw, even with the latest release I still have a negative free memory value. The free memory is computed at https://github.com/mirage/mirage-xen/blob/d1ee224fbdaa79bd4c1136a80c4594c454d12c27/lib/memory.ml#L44 as heap words minus live words minus stafk words.

Before saying something is wrong again in malloc_memory_footprint I'll try to double check with the classical malloc_footprint :)

Fix support for OCaml 4.04.1+

OCaml 4.04.2 was released with a fix for CVE-2017-9772.

The CVE itself is not relevant for consumers of ocaml-freestanding since there are no environment variables in a freestanding runtime configuration, however the fix introduces additional dependencies on geteuid(), getuid(), getegid() and getgid(), for which ocaml-freestanding has no stubs.

MirageOS unikernels using ocaml-freestanding built with 4.04.2 will fail at link time due to this change.

It's possible that we should modify the runtime configuration to build as if -no-cplugins had been specified rather than add more stubs.

dlmalloc: Enable heap canaries

We should enable support for heap canaries in dlmalloc. This can be done by compiling with FOOTERS defined to 1, and adding code to initialise the magic value using solo5_clock_monotonic().

Provide for OCaml 4.07.1

This is a bugfix release, so hopefully the changes are limited.

Add some basic link tests to the build

To facilitate better detection of issues such as #21, we should consider adding some basic link tests to the build. These would verify that a full link of a (non-runnable) freestanding "Hello, World" works.

Clean a bit the `Makefile` and delete some deprecated usage of the `./configure`

From a great discussion with @shindere, we invoke the ./configure in the wrong way:

1. we should not define our own OC_CFLAGS
2. we should call ./configure with envrionment variables like: ./configure LD=... instead of LD=... ./configure

From a certain perspective, this work is not really needed (because ocaml-solo5 already work for 4.14) but it can helps us to move forward about the OCaml 5 support.

openbsd int64 formatter

https://github.com/mirage/ocaml-freestanding/blob/master/config.in/m.x86_64.h#L13

I believe this should be included in the ifdef guard and be set to ll on openbsd?
see https://man.openbsd.org/printf.3

ping @adamsteen

Add an easy way to set-up the dev environment for this package

Right now, dune build or make will fail after a fresh clone. As ./configure command is quite involved, it'd be nice to have a quick way to setup an environment where we could start testing.

gmp-freestanding, zarith-freestanding: Update to build with SSP, disable any use of -fPIC

Reporting this here as the OPAM gmp-freestanding and zarith-freestanding packages don't have an associated issue tracker, and I don't want to spam Solo5/solo5#294 with this.

The changes in Solo5/solo5#294 break gmp-freestanding due to issues with its cross-compiling hacks. While fixing this, I also found that -fPIC was not being disabled correctly, so I fixed that as it will come in handy with other upcoming Solo5 changes that will alter the build toolchain used for ocaml-freestanding. This in turn broke zarith-freestanding, which has even more cross-compiling hacks, so I added some more hacks and forcibly eradicated any use of dynlink/-fPIC there as well.

The fixed packages are in Solo5/opam-solo5@d132213.

@hannesm When you have a moment, could you verify that pinning Solo5/solo5#294 and the versions of gmp-freestanding, zarith-freestanding above gets you a set of packages that successfully builds a unikernel that uses them? (i.e. one using tls) I have tested this on Debian, but would like a 2nd pass on FreeBSD.

I'm not sure what to do for conflicts/depends on these packages:

Once a version of Solo5 with SSP is released, gmp-freestanding needs to be at least 6.1-2. Does that mean I should add a conflicts: on all older versions at that point? The zarith-freestanding update is fine as it explicitly depends on the newer gmp-freestanding, however, technically, older versions of zarith-freestanding will fail with the newer gmp-freestanding due to the PIC-disabling going on.

The new `ocaml-freestanding` layout does not handle frame-pointers/nnp-checkers

This feature from the OCaml distribution expects that the toolchain used to compile OCaml must be gcc or clang and it recognizes it by the name of the toolchain. In our case, our toolchain is: {x86_64,arm64}-solo5-none-static. So we failed on this specific test at the configure time.

ARM64 support

This issue is for discussing how ARM64 should be implemented in ocaml-freestanding. It is a counterpart to the Solo5 issue at Solo5/solo5#151.

For ocaml-freestanding, the changes should be much simpler. Roughly:

• configure.sh needs to be taught about ARM64, and the corresponding OCaml configure products need to be added to config/.
• nolibc needs ARM64 support. This is probably best done by pulling in the existing infrastructure from musl, i.e. adding the appropriate subset of arch/arm/bits/*, arch/x86_64/bits/* and integrating these into the build system. For our purposes, this is just a few extra #defines added to include/.

Floating point issue

I'm not really sure if this issue is related to ocaml-freestanding or solo5 or xen or qubes, sorry for the disturbance if I misplaced the issue here.
Actually, I have #MF (floating point exception) error from solo5 with this unikernel (compiled with mirage 4.0~beta3):

$ cat unikernel.ml 
open Lwt.Infix

module Hello (Time : Mirage_time.S) = struct
  let start _time =
    Logs.info (fun f -> f "total runtime 1 %.6f s" 0.5);
    Logs.info (fun f -> f "total runtime 2 %.6f s" 1.0);
    Logs.info (fun f -> f "total runtime 3 %.6f s" 0.75);
    Logs.info (fun f -> f "total runtime 4 %.6e s" 0.5);
    Logs.info (fun f -> f "total runtime 5 %.6e s" 0.25);
    Logs.info (fun f -> f "total runtime 6 %.6e s" 0.875);
    Logs.info (fun f -> f "total runtime 7 %.6e s" 0.6);
    Logs.info (fun f -> f "total runtime 8 %.6e s" 0.4);
    Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
    Lwt.return_unit
end
$ cat config.ml 
open Mirage

let main =
  foreign
    "Unikernel.Hello" (time @->job)

let () =
  register "hello" [main $ default_time]

It works well with the x*1/2^y prints but fails on the first 0.6 print. The faulting address is on a fldz instruction ("add 0 to the FPU stack") which is due to something around the compilation of https://github.com/mirage/ocaml-freestanding/blob/7e85ff780deba6f526ff53445b5a0bd4ac6fd86e/nolibc/vfprintf.c#L236
I was not able to recompile ocaml-freestanding with -msoft-float.
If needed the kernel can be found there : https://we.tl/t-DEeIN7TW80 (md5sum: 982d75122f9346b0101510c4a5644252)
One strange thing is that it works fine with Xen 4.8 (Qubes 4.0.4) but not with Xen 4.14 (Qubes 4.1), I'm not sure if something is broken between these two versions.

string_of_float returns `-NaN` for most numbers formatted by printf

I found this issue where yojson objects with timestamps (floats) were being serialized OK, but when deserialized they would came back as -NaN.

To test, I ran the following code:

    let float_test f : unit =
      let s = string_of_float f in
      let s_printf = Printf.sprintf "%f" f in
      Printf.printf "float=%f -> string_of_float=%s -> parsed=%f\n" f s (float_of_string s);
      Printf.printf "float=%f -> printf-string  =%s -> parsed=%f\n" f s_printf (float_of_string s_printf)
    in

    float_test 2.4;
    float_test 1493895831.871153;
    let attempts = ref 500 in
    while !attempts > 0 do
      decr attempts;
      let f = Random.float 999999999999999.0 in
      float_test f;
      float_test f
    done;

I got the following output in mirage-ukvm, with ocaml-freestanding 0.2.1:

float=2.400000 -> string_of_float=2.4 -> parsed=2.400000
float=2.400000 -> printf-string  =2.400000 -> parsed=2.400000
float=1493895831.871153 -> string_of_float=1493895831.87 -> parsed=1493895831.870000
float=1493895831.871153 -> printf-string  =1493895831.871153 -> parsed=-nan
float=481705019586434.312500 -> string_of_float=4.81705019586e+14 -> parsed=481705019586000.000000
float=481705019586434.312500 -> printf-string  =481705019586434.312500 -> parsed=-nan
float=481705019586434.312500 -> string_of_float=4.81705019586e+14 -> parsed=481705019586000.000000
float=481705019586434.312500 -> printf-string  =481705019586434.312500 -> parsed=-nan
float=554380010410619.812500 -> string_of_float=5.54380010411e+14 -> parsed=554380010411000.000000
float=554380010410619.812500 -> printf-string  =554380010410619.812500 -> parsed=-nan
float=554380010410619.812500 -> string_of_float=5.54380010411e+14 -> parsed=554380010411000.000000
float=554380010410619.812500 -> printf-string  =554380010410619.812500 -> parsed=-nan

i.e. when using string_of_float the number correctly roundtrips to a string and back, but using printf's float formatting it consistently gets parsed as -NaN.
2.4 was a random simple case I tried, and funnily enough it works both ways - perhaps the bug only affects numbers with a large number of decimal places?

I remember float_of_string used to not work at all in xen because strtod was missing(mirage/mirage-platform#118), so I'm guessing this is probably a bug in ocaml-freestanding's strtod?

How to add support for <inttypes.h>

Hello,

I'm trying to build a library that uses <inttypes.h>, more specifically it's in hacl - the PRIu64 macro is used - it should expand to a format string specifier. This C99 header are unfortunately not supported in ocaml-freestanding.

In this particular case I might be able to patch my way around this problem (this debug code can be stripped), but this is not ideal as this code is extracted, so I'd rather not touch it too much.

How would I go about adding support for <inttypes.h>? can I just add inttypes.h in nolibc/include/?

(same would apply to bswap64 - I recall adding it to a couple libraries because it was missing in freestanding)

Thanks!

Next release (0.6.0)

Several recent PRs provide new interfaces to C code downstream of ocaml-freestanding:

• #74 adds the __ocaml_freestanding__ preprocessor macro
• #74 adds an <endian.h>
• #76 adds an <inttypes.h>

With this in mind, I think that the next release should be 0.6.0 to reflect "more than minor changes/additions", and downstream packages that wish to use these interfaces should depend on ocaml-freestanding >= 0.6.0.

@hannesm (mirage-crypto and related), @dinosaure (hacl), are you OK with this?

On hvt, I sometimes get a negative reading from malloc_memory_usage

I use ocaml-solo5 at its latest release (0.4.1), and one of my unikernels has a negative malloc_memory_usage (called from mirage-solo5,

CAMLprim value
mirage_memory_get_fast_live_words(value v_unit)
{
    return Val_long(malloc_memory_usage() / sizeof(value));
}

Please see the evolution of the "memory live bytes" in the diagram below. I'm not sure what the peak on Oct 16th is about. Any chance there could be an overflow somewhere? In dlmalloc.i, I noticed dispose_chunk does unconditionally subtract psize...

static void dispose_chunk(mstate m, mchunkptr p, size_t psize) {
  gm->internal_memory_usage -= psize;

But I'm not too familiar with the dlmalloc code.

ocaml-freestanding fails to build on OpenBSD

Hi All

I am not sure if anyone else uses OpenBSD, but it currently fails, I will eventually get back to fix this, but wanted it known by a larger audience.

Sorry
Adam

[ERROR] The compilation of ocaml-freestanding failed at "/usr/local/bin/gmake".

#=== ERROR while compiling ocaml-freestanding.0.4.7 ===========================#
# context     2.0.5 | openbsd/x86_64 | ocaml-system.4.09.0 | https://opam.ocaml.org#c2df68ef
# path        ~/.opam/default/.opam-switch/build/ocaml-freestanding.0.4.7
# command     /usr/local/bin/gmake
# exit-code   2
# env-file    ~/.opam/log/ocaml-freestanding-42466-a4623a.env
# output-file ~/.opam/log/ocaml-freestanding-42466-a4623a.out
### output ###
# [...]
# In file included from ./caml/io.h:33:
# In file included from /home/asteen/.opam/default/.opam-switch/build/ocaml-freestanding.0.4.7/nolibc/include/sys/types.h:5:
# /home/asteen/.opam/default/lib/pkgconfig/../../include/solo5-bindings-hvt/crt/stdint.h:67:21: error: typedef redefinition with different types ('__uint64_t' (aka 'unsigned long long') vs 'unsigned long')
# typedef __uint64_t              uint64_t;
#                                 ^
# ./caml/config.h:114:26: note: previous definition is here
# typedef ARCH_UINT64_TYPE uint64_t;
#                          ^
# 2 warnings and 2 errors generated.
# gmake[1]: *** [Makefile:343: startup_nat_n.o] Error 1
# gmake[1]: Leaving directory '/home/asteen/.opam/default/.opam-switch/build/ocaml-freestanding.0.4.7/build/ocaml/runtime'
# gmake: *** [Makefile:76: build/ocaml/runtime/libasmrun.a] Error 2



<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
+- The following actions failed
| - build ocaml-freestanding 0.4.7
+-

could endian.h be enhanced with htobe64 and be64toh

these primitives are needed for mirage-crypto

(e.g. from dune-universe/ocaml-nocrypto@3d7bdd5):

#if (BYTE_ORDER == LITTLE_ENDIAN)
#define htobe64(x) (_bswap64(x))
#define be64toh(x) (_bswap64(x))
#elif (BYTE_ORDER == BIG_ENDIAN)
#define htobe64(x) (x)
#define be64toh(x) (x)
#else
#error "Cannot determine byte order"
#endif

Fails to install with `cp: cannot create regular file 'boot/ocamlrun': Permission denied`

[ERROR] The installation of ocaml-solo5 failed at "make install".

#=== ERROR while installing ocaml-solo5.0.8.1 =================================#
# context     2.1.2 | linux/x86_64 | ocaml.4.13.1 | https://opam.ocaml.org#e1c20992
# path        ~/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/ocaml-solo5.0.8.1
# command     ~/.opam/opam-init/hooks/sandbox.sh install make install
# exit-code   2
# env-file    ~/.opam/log/ocaml-solo5-351273-499edc.env
# output-file ~/.opam/log/ocaml-solo5-351273-499edc.out
### output ###
# [...]
# make[2]: Leaving directory '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/ocaml-solo5.0.8.1/ocaml/runtime'
# make -C stdlib \
#   OCAMLRUN='$(ROOTDIR)/runtime/ocamlrun' \
#   CAMLC='$(BOOT_OCAMLC) -use-prims ../runtime/primitives' all
# make[2]: Entering directory '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/ocaml-solo5.0.8.1/ocaml/stdlib'
# make[2]: Nothing to be done for 'all'.
# make[2]: Leaving directory '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/ocaml-solo5.0.8.1/ocaml/stdlib'
# cp runtime/ocamlrun boot/ocamlrun
# cp: cannot create regular file 'boot/ocamlrun': Permission denied
# make[1]: *** [Makefile:165: coldstart] Error 1
# make[1]: Leaving directory '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/ocaml-solo5.0.8.1/ocaml'
# make: *** [Makefile:112: ocaml] Error 2

rm: cannot remove '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/solo5.0.7.3/toolchain/include/x86_64-solo5-none-static/ssp/string.h': Permission denied
rm: cannot remove '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/solo5.0.7.3/toolchain/include/x86_64-solo5-none-static/ssp/unistd.h': Permission denied
rm: cannot remove '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/solo5.0.7.3/toolchain/include/x86_64-solo5-none-static/ssp/stdio.h': Permission denied
rm: cannot remove '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/solo5.0.7.3/toolchain/include/x86_64-solo5-none-static/ssp/ssp.h': Permission denied
rm: cannot remove '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/solo5.0.7.3/toolchain/include/x86_64-solo5-none-static/sanitizer/hwasan_interface.h': Permission denied
rm: cannot remove '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/solo5.0.7.3/toolchain/include/x86_64-solo5-none-static/sanitizer/tsan_interface.h': Permission denied
rm: cannot remove '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/solo5.0.7.3/toolchain/include/x86_64-solo5-none-static/sanitizer/lsan_interface.h': Permission denied
rm: cannot remove '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/solo5.0.7.3/toolchain/include/x86_64-solo5-none-static/sanitizer/common_interface_defs.h': Permission denied
rm: cannot remove '/home/ryan/projects/mirage-hello-solo5-dune-2/_opam/.opam-switch/build/solo5.0.7.3/toolchain/include/x86_64-solo5-none-static/sanitizer/asan_interface.h': Permission denied

`memcpy` does not work on RPi4

I think the issue is really complex but I took the nolibc implementation here for [gilbraltar][] and I was stuck at one point where memcpy seems to fail. Indeed, with my experiment, a simpler memcpy such as:

void *memcpy(void *v_dst, const void *v_src, size_t len)
{
  char *dst = v_dst;
  const char *src = v_src;

  for (; len; len--) *dst++ = *src++;
  return dst;
}

seems to work when the current one seems to do something illegal for the Raspberry Pi 4. I don't really know where is the error (if it's about Aarch64 or the layout of my ELF or just because the code is may be too complex for what it should do). But I'm sure that something is wrong only on Raspberry Pi 4.

Due to the fact that we want to support the cross-compilation and specially about the Aarch64 architecture, I wrote this issue to keep in our mind that may be something is wrong about the current implementation - but again, it's not clear when I can not dump why the Raspberry Pi 4 wants to do.

Note that the current nolibc works with qemu - so you can figure out how painful it was to debug.

ocaml-freestading 0.6.6 failing to compile on OpenBSD (7.0 Current)

checking for copysign... yes
configure: error: C99 float ops unavailable, enable replacements with --enable-imprecise-c99-float-ops
gmake: *** [Makefile:43: ocaml/Makefile.config] Error 1

$ ocaml --version
The OCaml toplevel, version 4.13.1
$ opam --version
2.0.8

if there is anything else you need or i can help with please let me know