mirage / mirage

MirageOS is a library operating system that constructs unikernels

Build Unikernels in OCaml

---

MirageOS is a library operating system that constructs secure, performant and resource-efficient unikernels.

About

MirageOS is a library operating system that constructs unikernels for secure, high-performance network applications across various cloud computing and mobile platforms. Developers can write code on a traditional OS such as Linux or macOS. They can then compile their code into a fully-standalone, specialised unikernel that runs under the Xen or KVM hypervisors and lightweight hypervisors like FreeBSD's BHyve, OpenBSD's VMM. These unikernels can deploy on public clouds, like Amazon's Elastic Compute Cloud and Google Compute Engine, or private deployments.

The most up-to-date documentation can be found at the homepage. The site is a self-hosted unikernel. Simpler skeleton applications are also available online. MirageOS unikernels repositories are also available here or there.

This repository

This repository contains the mirage command-line tool to create and deploy applications with MirageOS. This tool wraps the specialised configuration and build steps required to build MirageOS on all the supported targets.

Local install

You will need the following:

• a working OCaml compiler (4.08.0 or higher).
• the Opam source package manager (2.1.0 or higher).
• an x86_64 or armel Linux host to compile Xen kernels, or FreeBSD, OpenBSD or MacOS X for the solo5 and userlevel versions.

Then run:

$ opam install mirage
$ mirage --version

This should display at least version 4.0.0.

Using mirage

There are multiple stages to using mirage:

• write config.ml to describe the components of your applications;
• call mirage configure to generate the necessary code and metadata;
• optionally call make depends to install external dependencies and download Opam packages in the current dune workspace.
• call dune build to build a unikernel.

You can find documentation, walkthroughs and tutorials over on the MirageOS website. The install instructions are a good place to begin!

mirage's Issues

SSL support

more .annot issues

@avsm @samoht generation now ok in the first pass. but .annot files don't seem to be updated when source .ml changes?

Migrate mailing list archives

Mailing list has moved to Xen Project but we still need to migrate the archives at some point.

Links to both sets of archives can be found at: http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel

Non-dev prep work for Mirage release

This is a checklist to track the non-dev activities around the upcoming Mirage release (i.e running parallel to #111). Please do edit and tidy this up (it's a first draft at the moment)

Mailing list

• Announcement to current list about migration - @amirmc
• @larskurth to create new list [email protected] (in moderation mode)
• Copy over subscribers to new list @larskurth

On 6th Dec, put old list into moderation and unmoderate new list @avsm
@avsm will create the mbox archive from old list. @larskurth will get dev time at Xen to merge in archives.
Mailing list move captured in #132

Linux article on libOS

• Lars would like to write a Linux.com article on Library OSs (e.g OSV, Mirage etc). This would provide some groundwork and background to the Mirage Blog 1.0 post and PR.
• Lars will get in touch with Libby to see when such an article could be done by (might be after the 9th).
  @amirmc/@avsm need to send Lars a briefing note on terminology (use of 'unikernel' etc).

Blog posts for 1.0 release

Need blog post for both openmirage.org and Xen Project sites.

• @amirmc to draft posts and circulate by Wed
• Posts to be ready in final form on Friday (@amirmc).

Blog post should:

• Have a section on things for the future.
• Outline real use-cases where it's valuable (need @djs55/@avsm for this)
• Ref other OCaml sources where sensible (e.g ocaml.org, RWO)

Docs on xen project

• Include clear instructions on how to get the 1.0 release (@amirmc will draft - @djs55/@avsm to approve)
• Review Mirage portal on Xen site and update where needed. (@larskurth)
• Include links to other OCaml sources where sensible.

Look at OSV video

Site redesign

• Replace image on front page with Mirage Video [1]. WHO?
• @amirmc to send Lars sketch for button for Xen Project (need sizes)
• @amirmc to ask Lars for image for Mirage Logo/Text on top of page (to update what's currently there)
• Add the new Xen Project logo to the front page somewhere (@amirmc)
• Update mailing list details on Friday 6th Dec

License audit This is ongoing and mentioned in #110

News release

• @avsm to get confirmation from Neuman about quote.

random self init

The build system could add a Random.self_init to the generated foo__.ml which invokes the Lwt main runtime. A number of the Lwt examples use this, so it may be useful to compile the program with a deterministic seed for test purposes and a truly random seed for production use.

Pa_cstruct: map Bigarray as a C-like struct

Bitstring is excellent for pattern-matching, but the write path is a serious bottleneck: each BITSTRING invocation allocates a temporary buffer and writes into it, before blitting to the target.

A quick modification I did (in my iteratee-io branch) was to modify pa_bitstring to accept a target bitstring to write into (which is a very simple patch). However, the problem of sub-bitstrings not being reference counted remains a problem, and Bitstrings also operate on OCaml strings.

Therefore, this ticket will look into adapting the (excellent) Bitstring code into a new pa_cstruct which will operate only on Bigarrays, and have an attach operation to map the array as a C-like structure. Comments will contain design updates as they happen.

link through .annot files for emacs tuareg mode

there may be better ways to do this (tweak tuareg-mode for example) but for the time being- i find myself frequently executing the following sequence in order to have C-cC-t type highlighting in emacs tuareg-mode work

$ ocamlbuild backend/foo.bin
$ cd _build && for n backend/*.annot ; do ln -s $n ; done

...which is ugly but more-or-less works.

could the build system do this for me please? i'm assuming that, being an application synthesis framework an' all, if it works for one backend it'll work for them all, so would only require the most recently built backend to have the annot files pulled through.

am happy to take a stab at this if someone can point me at the relevant bit in (presumably) myocamlbuild.ml...

less clear for me- what's supposed to happen re the different backends for building mirage itself? some things seem straightforward to link through, but eg., net seems to have a more complex directory structure?

wire up mirage-net-xen

The mirage/mirage-net-xen repo needs to be wired up to complete the chain for the self-hosting mirage-www.

@djs55 has committed all the relevant fixes to the block layer, so we should now be in good shape to do the same for networking.

@samoht, is HTTP wired up now with cohttp-trunk? I can cut a release of 0.9.13 today if so.

ERROR: Already connected to xenstore: cannot reconnect

mirage-www crashes on startup with:

Top level exception: Xs.IO.Already_connected

uninstall error

mirage-unix.0.9.3 Downloading https://github.com/mirage/mirage-platform/archive/v0.9.3.tar.gz
  make unix-uninstall PREFIX=/Users/tgazagna/.opam/system
[WARNING] failure in package uninstall script, some files may remain:
# opam-version    1.1.0 (b4f46b61d3b28c56797cd3c0356b08746fa340ab)
# os              darwin
# command         make unix-uninstall PREFIX=/Users/tgazagna/.opam/system
# path            /Users/tgazagna/.opam/system/build/mirage-unix.0.9.3
# compiler        system (4.00.1)
# exit-code       2
# env-file        /Users/tgazagna/.opam/system/build/mirage-unix.0.9.3/mirage-unix-42943-bb845d.env
# stdout-file     /Users/tgazagna/.opam/system/build/mirage-unix.0.9.3/mirage-unix-42943-bb845d.out
# stderr-file     /Users/tgazagna/.opam/system/build/mirage-unix.0.9.3/mirage-unix-42943-bb845d.err
### stdout ###
make OS=unix PREFIX=/Users/tgazagna/.opam/system uninstall
cd unix && make uninstall
### stderr ###
make[2]: *** No rule to make target `uninstall'.  Stop.
make[1]: *** [uninstall] Error 2
make: *** [unix-uninstall] Error 2

Hello world failure.

I'm trying to follow the hello world tutorial. It fails during make basic-configure (see below), dllunixrun.so is indeed already installed and owned by mirage, should I remove it ?

> make basic-configure
mirage configure basic/config.ml  --unix
MIRAGE      Using the specified config file: basic/config.ml
MIRAGE      Compiling and dynlinkg /Users/dbuenzli/tmp/mirage-skeleton/basic/config.ml
MIRAGE      + Executing: rm -rf /Users/dbuenzli/tmp/mirage-skeleton/basic/_build/config.*
Finished, 4 targets (0 cached) in 00:00:00.

MIRAGE      1 JOBS: job1 | 0 DRIVERS: 
MIRAGE      Installing OPAM packages.
MIRAGE      + Executing: opam install --yes mirage-console-unix mirage-unix[WARNING] unknown package "mirage-xen"
[WARNING] unknown package "mirage-xen"
[WARNING] unknown package "mirage-xen"
[WARNING] unknown package "mirage-xen"
[WARNING] unknown package "mirage-xen"
[WARNING] unknown package "mirage-xen"
[WARNING] unknown package "mirage-xen"
[WARNING] unknown package "mirage-xen"
The following actions will be performed:
 - install mirage-unix.0.9.9 [required by mirage-console-unix]
 - install mirage-console-unix.1.0.0
2 to install | 0 to reinstall | 0 to upgrade | 0 to downgrade | 0 to remove

  Installing mirage-unix.0.9.9 
Building mirage-unix.0.9.9:
  make unix-build
  make unix-install PREFIX=/Users/dbuenzli/.opam/4.00.1
[ERROR] The compilation of mirage-unix.0.9.9 failed.
Removing mirage-unix.0.9.9.
  make unix-uninstall PREFIX=/Users/dbuenzli/.opam/4.00.1

[ERROR] Due to some errors while processing mirage-unix.0.9.9, the following actions will NOT proceed:
 - install mirage-console-unix.1.0.0

===== ERROR while installing mirage-unix.0.9.9 =====
# opam-version 1.1.0
# os           darwin
# command      make unix-install PREFIX=/Users/dbuenzli/.opam/4.00.1
# path         /Users/dbuenzli/.opam/4.00.1/build/mirage-unix.0.9.9
# compiler     4.00.1
# exit-code    2
# env-file     /Users/dbuenzli/.opam/4.00.1/build/mirage-unix.0.9.9/mirage-unix-84574-5f1023.env
# stdout-file  /Users/dbuenzli/.opam/4.00.1/build/mirage-unix.0.9.9/mirage-unix-84574-5f1023.out
# stderr-file  /Users/dbuenzli/.opam/4.00.1/build/mirage-unix.0.9.9/mirage-unix-84574-5f1023.err
### stdout ###
# ...[truncated]
# /Users/dbuenzli/.opam/4.00.1/bin/ocamlfind ocamlopt -c -I /Users/dbuenzli/.opam/4.00.1/lib/ocaml -I /Users/dbuenzli/.opam/4.00.1/lib/ocaml -I /Users/dbuenzli/.opam/4.00.1/lib/ocplib-endian -I /Users/dbuenzli/.opam/4.00.1/lib/ocplib-endian -I /Users/dbuenzli/.opam/4.00.1/lib/cstruct -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/io-page-unix -I /Users/dbuenzli/.opam/4.00.1/lib/mirage-clock-unix -package lwt -package cstruct -for-pack OS -pp 'camlp4o -I /Users/dbuenzli/.opam/4.00.1/lib/ocaml -I /Users/dbuenzli/.opam/4.00.1/lib/ocaml -I /Users/dbuenzli/.opam/4.00.1/lib/ocplib-endian -I /Users/dbuenzli/.opam/4.00.1/lib/ocplib-endian -I /Users/dbuenzli/.opam/4.00.1/lib/cstruct -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/io-page-unix -I /Users/dbuenzli/.opam/4.00.1/lib/mirage-clock-unix -I /Users/dbuenzli/.opam/4.00.1/lib/ocaml/camlp4 -I /Users/dbuenzli/.opam/4.00.1/lib/cstruct cstruct-syntax.cma -I /Users/dbuenzli/.opam/4.00.1/lib/lwt lwt-syntax-options.cma -I /Users/dbuenzli/.opam/4.00.1/lib/lwt lwt-syntax.cma -I /Users/dbuenzli/.opam/4.00.1/lib/ocaml str.cma' -I lib -o lib/main.cmx lib/main.ml
# touch lib/oS.mli  ; if  /Users/dbuenzli/.opam/4.00.1/bin/ocamlfind ocamlopt -pack -I lib lib/env.cmx lib/time.cmx lib/main.cmx -o lib/oS.cmx  ; then  rm -f lib/oS.mli  ; else  rm -f lib/oS.mli  ; exit 1; fi
# /Users/dbuenzli/.opam/4.00.1/bin/ocamlfind ocamlopt -a -I /Users/dbuenzli/.opam/4.00.1/lib/ocaml -I /Users/dbuenzli/.opam/4.00.1/lib/ocaml -I /Users/dbuenzli/.opam/4.00.1/lib/ocplib-endian -I /Users/dbuenzli/.opam/4.00.1/lib/ocplib-endian -I /Users/dbuenzli/.opam/4.00.1/lib/cstruct -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/io-page-unix -I /Users/dbuenzli/.opam/4.00.1/lib/mirage-clock-unix -cclib -lunixrun lib/oS.cmx -o lib/oS.cmxa
# cc -I/Users/dbuenzli/.opam/4.00.1/build/mirage-unix.0.9.9/unix/lib -c -Wall -O3 -fPIC -I/Users/dbuenzli/.opam/4.00.1/lib/ocaml -o lib/checksum_stubs.o lib/checksum_stubs.c
# /Users/dbuenzli/.opam/4.00.1/bin/ocamlmklib -o lib/unixrun lib/checksum_stubs.o
# /Users/dbuenzli/.opam/4.00.1/bin/ocamlfind ocamlc -a -I /Users/dbuenzli/.opam/4.00.1/lib/ocaml -I /Users/dbuenzli/.opam/4.00.1/lib/ocaml -I /Users/dbuenzli/.opam/4.00.1/lib/ocplib-endian -I /Users/dbuenzli/.opam/4.00.1/lib/ocplib-endian -I /Users/dbuenzli/.opam/4.00.1/lib/cstruct -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/lwt -I /Users/dbuenzli/.opam/4.00.1/lib/io-page-unix -I /Users/dbuenzli/.opam/4.00.1/lib/mirage-clock-unix -dllib -lunixrun lib/oS.cmo -o lib/oS.cma
# # Parallel statistics: { count(total): 2(19), max: 3, min: 3, average(total): 3.000(1.211) }
# make OS=unix PREFIX=/Users/dbuenzli/.opam/4.00.1 install
# cd unix && make install MIRAGE_OS=unix
# ./cmd install
### stderr ###
# ocamlfind: [WARNING] No such directory: /Users/dbuenzli/.opam/4.00.1/lib/mirage-unix
# ocamlfind: Conflict with another package: Library dllunixrun.so is already installed
# make[3]: *** [install] Error 2
# make[2]: *** [install] Error 2
# make[1]: *** [unix-install] Error 2

'opam install --yes mirage-console-unix mirage-unix' failed.
[ERROR]      The command "opam install --yes mirage-console-unix mirage-unix" exited with code 4.

make: *** [basic-configure] Error 1

generating .annot files for lib/

my ocamlbuild chops are not good enough - i don't seem to be getting .annot files generated for things under lib (eg., lib/openflow) even though it looks to me like myocamlbuild applies the -annot flag.

any ideas? is this another thing that's busted in my tree, or is it actually broken?

Break out boot libraries from mirage-platform

have a way to easily send ping packets using Mirage

Improve the lib_test/net testcase to send a real ping packet

It would be good if it wrote a valid Ethernet frame to make testing easier, rather than a random string.

Remove the linker warnings for Xen dietlibc

Being informed that "sprintf adds several kilobytes" really isn't very useful information

unspecified license

It's unclear from the quick glance at the repo and related website what's the license for the code in this and related repositories.

TCP hangs in direct stack

After clicking a few times on various links, the browser connection hangs after a seemingly indeterminate amount of time.

@balrajsingh narrowed it down to the Net.Channel code:

  let read_until t ch =
    lwt () = if Bitstring.bitstring_length t.ibuf = 0 then
      ibuf_refill t else return () in
    try_lwt
      let buf,off,len = t.ibuf in
      let idx = (String.index_from buf (off/8) ch) * 8 in
      let rlen = idx - off in
      (bitmatch t.ibuf with
      | { _:8; rest:-1:bitstring } when rlen = 0 ->
          t.ibuf <- rest;
          return (true, Bitstring.empty_bitstring)
      | { r:rlen:bitstring; _:8; rest:-1:bitstring } ->
          t.ibuf <- rest;
          return (true, r)
      | { _ } ->
        (* Printf.printf " *****>>>  Oh no .....\n%!"; *)
        let r = t.ibuf in
        t.ibuf <- "",0,0; 
        return (false, r))
    with Not_found -> begin
      let r = t.ibuf in
      t.ibuf <- "",0,0; 
      return (false,r)
    end

where the bad path being taken is the "Oh no" branch

improve DHCP client to be more robust

• Retry more than once
• try_lwt to ignore failures and plough on!

http server request body not handled

this was commented out in February 2011, probably because it would take too long to fix at the time (just guessing).
It also doesn't support PUT (e.g. for typical RESTful/CRUD interface).
I'm having a go at fixing this in my fork of mirage on the httprequestfix branch.

Schedule some posts over the coming days/weeks

Now that Mirage 1.0 is out, we need to put together a bunch of posts to describe the system and help people (and each-other) get to grips with it.

This will effectively form some of the documentation so it's more than just blog posts and wiki pages -- getting this content in written form is important and we can come back to each one to properly cross-reference. :)

Below are some ideas for posts we think are needed and also some people who could write them. If you're ok with the list as is, please say so in the comments. The idea is that once we've agreed on what posts we need, we'll create a separate issue for each post, where we can discuss what the contents need to include.

I'd also like us to agree on a schedule of posts so that this doesn't slip and we can (1) keep the cadence up while (2) spreading the work.

Homepages

• mirage-www (crunch) -- @avsm (12 Dec)
• mirage-decks -- @mor1 (13 Dec)
• Deployment on Amazon -- @djs55 and @avsm (16 Dec)
• Cowabloga -- @mor1 (18 Dec)
• Unix Raspberry Pi -- @amirmc (20 Dec)

Xen and the art of motorcycle VM Maintenence

• How boot works, MiniOS etc -- @avsm
• Device drivers -- @djs55
• Suspend/Resume -- @jonludlam
• Building a new backend/frontend driver pair (Randomfront/Randomback) -- @avsm
• Streaming unikernels directly to XenServer -- @djs55

Lwt

• Stuff on co-operative threading/Lwt -- @avsm
• Xen Event system -- @djs55

New backend target (Javascript)

• Things to port (clock, console, platform) -- @avsm

Filesystems and middlebox stuff

• crunch -- @avsm
• FUSE -- @dsheets
• FAT -- @djs55
• Xen-Disk -- @djs55

Networking

• TCP/IP -- @balrajsingh and @avsm
• unix-networking -- @avsm
• HTTP -- WHO?

Architecture/Types

• ctypes? -- @yallop

Update the DNS performance tests page on the wiki

The DNS performance test page on the Wiki is rather out of date, and being able to run these tests on the latest code would be useful (if only as a performance regression test, but also as a way to reproduce our ASPLOS 2013 results).
http://www.openmirage.org/wiki/performance

See comment: http://www.openmirage.org/wiki/performance#comment-1157202534

merge ocaml-fat

Merge the djs55/ocaml-fat repository into Mirage. A very basic read-only version can be modelled after the block/direct/SimpleKV.ml which depends on a block device.

A read-write version will require defining a new device type in os/unix/devices.ml (kv_rw) and plumbing that through, but that can be a second phase.

Port mirage-perf to new regression system

The new regression system should be enough to port the old mirage-perf repository over

[xen] sharing of the xenstore channel is broken

In userspace we normally open fresh xenstore connections over the unix domain socket. In kernel space we need to share the single shared request/response ring.

The current code will give errors like:

Unexpected_packet("Expecting DIRECTORY received READ")

when (eg) initialising two providers in parallel. This is really obvious now that I've made the blkif work better.

Clearly we need to make the Xs code able to demux responses to the correct caller based on the request id....

port Core Bigstring

After much to-and-fro, @djs55 and I decided that we really should have out-of-heap strings that are separate from OCaml strings. This is due to a lot of mixups with Io_page and normal strings (which, in Xen, absolutely must never ever ever be mixed up).

I am going to look at porting the Bigstring module from Core over, which uses Bigarray under the hood. A first step to Core integration perhaps!

mirage-www: remove Disqus

We should remove the Disqus and point people clearly to the issues URL instead. Right now there are too many comments and places to find...

unhygenic build

i seem to consistently get the following droppings left if i do "make all install":

syntax/pa_mirage.cma
syntax/pa_mirage.cmxs
syntax/str.cmxs

...meaning that subsequent "make all install" gets a hygiene warning requesting invocation of a sanitize script

is this intentional?!

"make bootstrap" always tries to install findlib in /usr/lib/ocaml

I tried make PREFIX=/home/celtic/mir-inst bootstrap:

mkdir -p "/home/celtic/mir-inst/lib/ocaml/site-lib/findlib"
mkdir -p "/home/celtic/mir-inst/bin"
test 1 -eq 0 || cp topfind "/usr/lib/ocaml"
cp: cannot create regular file `/usr/lib/ocaml/topfind': Permission denied

It still tries installing there!

plumb through camlp4 debug options

There are a number of camlp4 debug options which should be plumbed through. e.g. Lwt and Bitstring both define some, and we need a Log module too.

One way to do this is to have a %.debug.ml which represents a debug p4 output, and can be linked in instead of %.ml

What have happened to MPL?

I see MPL being removed from tools directory - is it just gone and not available anywhere except old project commits?

unix-direct netif broken

From @mor1 :

i'm porting the dns perf suite over to the new regress/ setup.
with the unix-socket build, i don't seem to be able to contact it at all -
and netstat doesn't show anything listening on 53/UDP as i'd expect.
with unix-direct then i see the usual startup but then start seeing many failed writes:

: mort@greyjay:net$; sudo _build/unix-direct/deens.bin 
Password:
Main: startup
[00000] 2011/08/17T22:06:39Z  info                Deens: starting server, port 53 
Manager: create
opendev: /dev/tap0
Netif: plug tap0
Manager: plug tap0
Manager: plug done, to listener
Manager: init done
Devices: starting provider Unix.Blkif
Main: entering runloop
ARP: who-has 10.0.0.1?
IPv4: dropping proto 2
IPv4: dropping proto 2
IPv4: dropping proto 2
ARP: transmitting probe -> 10.0.0.1
EXN: Failure("tap: partial write")
ARP: updating 10.0.0.1 -> 1e:81:8c:9d:18:0c
EXN: Failure("tap: partial write")
EXN: Failure("tap: partial write")

via "tcpdump -i tap0", pinging 10.0.0.1 from a terminal gets me nothing; pinging 10.0.0.255 shows echo requests from 10.0.0.1 -> 10.0.0.255, and responses from 0.0.0.0 -> 10.0.0.1 which seems odd.

similar behaviour if i build the tutorial slideset with unix-direct, although with unix-socket it works just fine.

configuration: expose a convenience function for DHCP

In module IP, there's a local function for a default IP address. a IP.dhcp <net> would also be useful.

Packages not available for my compiler or OS? (Linux x86-64)

I'm running on Fedora 18 (64bit x86). When I tried to build the Hello World example, I got:

$

[phil@quad mirage-skeleton]$ make basic-configure

mirage configure basic/config.ml --unix

MIRAGE Using the specified config file: basic/config.ml

MIRAGE Compiling and dynlinkg /mnt/mint/phil/development/OCaml_stuff/mirage/mirage-skeleton/basic/config.ml

MIRAGE + Executing: rm -rf /mnt/mint/phil/development/OCaml_stuff/mirage/mirage-skeleton/basic/_build/config.*

MIRAGE + Executing: cd /mnt/mint/phil/development/OCaml_stuff/mirage/mirage-skeleton/basic && ocamFinished, 4 targets (0 cached) in 00:00:01.

MIRAGE 1 JOBS (job1)

MIRAGE Installing OPAM packages.

MIRAGE + Executing: opam install --yes mirage-console-unix mirage-unix[NOTE] Package mirage-unix is already installed (current version is 0.9.7).

The dependency cudf of package opam-lib.1.0.0 is not available for your compiler or your OS.

'opam install --yes mirage-console-unix mirage-unix' failed.

[ERROR] The command "opam install --yes mirage-console-unix mirage-unix" exited with code 3.

That was with OCaml 4.01.00, so then I did:

$ opam switch 4.00.1

$ eval opam config env

$ opam install mirage

The dependency ospec>=0.3.1 of package uint.1.1.0 is not available for your compiler or your OS.

'opam install mirage' failed.
•Edit•Reply•

Libraries for 1.0 checklist

(extracted from #102, running parallel to #109)

Core Unix/Xen libraries:

• shared-memory-ring: Builds shared memory-based point-to-point communication protocols. Is reasonably mature and can be released. Can be used outside of Xen, need to benchmark vs c. 4.01.0dev ocaml improves perf with bigarray builtins. Implements the Xen ABI, but needs examples in-tree. (@djs55). For 1.0.
• ocaml-xenstore: Repo layout is confusing. Client and server in here, as well as Lwt bindings. Does build as stub domain and unix kernel. How can this stay in sync with xen-unstable? (@djs55). For 1.0.
• ocaml-fd-send-recv: UNIX only and trivial. Released and stable. For 1.0. (@djs55/@vbmithr)
• ocaml-xen-block-driver: Successfully moved out, and works in Xen. It once worked in userspace. (@djs55 / @jonludlam). For 1.0.
• ocaml-cstruct:

POST 1.0:

• ocaml-vchan: Vchan communication channel between VMs. Provides a socket-like interface between VMs. For 1.0 (Owner: @djs55). Post 2.0: vsock?
• ocaml-qmp: QEMU message-passing protocol to command a running qemu. Not for 1.0, really for advanced use (@djs55 ).
• ocaml-gntshare: Interface to mapping pages across Xen VMs. In ocaml-xen-lowlevel-libs, in good state, part of upstream xen-unstable and needs synching. (@djs55)
• ocaml-xen-net-driver: Can be moved out, and works in Xen. It once worked in userspace. (@djs55 / @jonludlam). For 1.1.

Core Mirage libraries and tools:

• mirage-platform:
• mirari:
• opam:

POST 1.0:

• dyntype:
• ocaml-re: stable, done
• cross-ref ocamldoc: leo

Storage libraries:

• ocaml-fat:
• mirage-fs: Just there for crunch. Probably not needed.
• ocaml-crunch: Works.

POST 1.0:

• libvhd: ?? Jon, suitable for block driver use?
• ocaml-iscsi: Doesnt exist, but djs really wants it.
• cagit: Needs a file system (Owner: tg).
• irminsule: tg, not for 1.0
• arakoon: In-memory-only patch, Arakoon already functorised across the storage layer. For 1.0? to investigate
• orm: Post 1.0, needs sqlite bindings

Network libraries

• ocaml-tuntap
• mirage-net - with a ping test?

POST 1.0:

• better abstraction for mirage-net
• ocaml-dns:
• ocaml-pcap:
• ocaml-openflow:
• ocaml-xmpp: ermine repos, lwt, need porting to mirage-net

RPC/coordination:

POST 1.0:

• rpc-light: Works well, but slow and no wire protocol defined. Vb mentioned Thrift. Also have bin_io? No versioning story. Needs obuild help. (Owner: tg)
• message-switch:
• ocamlmq: to investigate.
• logger: need a vchan-based logger and CLI to access from other domain

Security:

POST 1.0:

• mirage-cryptokit:
• ocaml-crypto-keys:
• stud: need ssl binding for fable
• ocaml-ssh: still in mpl-land, not too hard to port

Webby libraries:

• ocaml-cohttp:
• ocaml-cow:
• ocaml-atom:
• ocaml-cowabloga:
• ocaml-uri:

POST 1.0:

• ocaml-spdy: out of date, not for 1.0

Cloud interfaces:

• aws: ec2 bindings, just a shell script should be sufficient.
• xref to OSv docs on this?

POST 1.0:

• ocaml-libvirt: dave upstreaming
• xenopsd: convenient single-host daemon for ubuntu-unstable.

Testing:

POST 1.0:

• Ocamlot
• Pathos testing?

Tutorials and examples:

• mirage-skeleton: important for regress tests.
• mirage-www

Misc

• Integrate ocaml-tuntap into mirage-platform to remove tun hacks [vincent]

POST 1.0:

• cohttp/ssl releases [anil]

node.js build is broken

(there may be other issues but) the problems i saw were due to the openflow module's use of Net facilities which don't yet have dummies in the node version of the networking, e.g. OS.Netif, Net.Manager.interface.
The build is only attempted when js_of_ocaml is installed (which i only just installed) so I'm not sure when it stopped building.

mir-build broken on i386

The regress/myocamlbuild.ml checks for the wrong output from uname -m, it is i386 instead of x86_32

Docs: clarify that the minimum version 4.00.1, but 4.01.0 works fine too.

See http://openmirage.org/wiki/install#comment-1161246611 for some confusion on this.

pa_cow requires Cow to be opened

The <:html< >> and related syntax extensions all assume Cow is opened, and give a slightly obscure error message if it isn't.

Probably best to assume that no modules have been opened by the user when a syntax extension is opened.

`mirage clean` too aggressive

eg., it removes the mirage package itself (for me at least).

suggestion on irc was to make --no-opam the default for clean.

Repository disaggregation

A very nice feature of the existing Mirage tree is that everything is editable from a single repository. However, it's getting hard to manage all of these disparate protocols, and they should be released independently so they work on conventional Lwt_unix as well.

The first useful step here would be remove our OS.Unix bindings and switch to Lwt conventions where possible. This would mean having a Lwt_mirage module with a sub-set of the Lwt_unix functions, and using Lwt_unix directly from our socket bindings.

Secondly, we need to choose from all the sub-tree mechanisms in git to find the least painful one to split out the individual protocols in their own repositories, but still be able to handle them from a single tree if required. One advantage we have is that the Mirage ocamlbuild could be configured to auto-clone repositories from a specification file, thus letting us 'pin' protocols on release branches quite easily.

review uses of String.explode/implode

There are convenient but expensive functions String.explode/implode. Should review the use of these in the codebase and perhaps switch to Regexp if it's better.

See discussion in djs55@80632db#comments
and also Regexp benchmarks in http://www.openmirage.org/wiki/ocaml-regexp

make doc is broken with ocaml 3.13.0

The ocamldoc internals seems to have changed between 3.12 and 3.13.

Here is a tentative patch.

diff --git a/docs/odoc_json.ml b/docs/odoc_json.ml
index eaadfe0..30df22a 100644
--- a/docs/odoc_json.ml
+++ b/docs/odoc_json.ml
@@ -543,12 +543,12 @@ class gen () =

        method generate (modules_list : t_module list) =
                let write_module_json (name, json, _) =
-                       let oc = open_out (!Odoc_args.target_dir ^ "/" ^ name ^ ".json") in
+                       let oc = open_out (!Odoc_global.target_dir ^ "/" ^ name ^ ".json") in
                        output_string oc (json_to_string 0 json);
                        close_out oc
                in
                let write_index_json ml =
-                       let oc = open_out (!Odoc_args.target_dir ^ "/index.json") in
+                       let oc = open_out (!Odoc_global.target_dir ^ "/index.json") in
                        let make_record = function
                        | (name, Object ["module", Object m], (dc, cdc)) ->
                                let info = List.assoc "info" m in
@@ -585,7 +585,11 @@ class gen () =
                write_index_json modules
 end

-let generator = ((new gen ()) :> Odoc_args.doc_generator)
+module Generator = struct
+  class generator = object
+    method generate = (new gen ())#generate
+  end
+end

-let _ = Odoc_args.set_doc_generator (Some generator)
+let _ = Odoc_args.set_generator (Odoc_gen.Other (module Generator))

LWT tutorial examples in regress

Add LWT tutorial examples to regress/, so I can copy them into the mirage-tutorial slide presentation also.

mirage-cohttp: doesn't close connections on error

If you run lib_test/http and send garbage down the wire, the connection isn't closed by the server.

Switch OS to map Io_pages as arrays

We've tried a variety of strategies to map the OCaml string type to zero-copy I/O pages, but it ultimately needs reliable reference counted views onto a page. The easiest way to do this is via uint8 Bigarray, so this will port the OS.{Xen/UNIX} modules to use Bigarray.

This will require touching pretty much all of Net and the upstream libraries in order to move away from string parsing, but it should hopefully be hidden away behind Lwt_stream and possibly an Lwt_io variant for OS.Xen.

Fix upstream support for Sys.os_type = xen

See mirage/mirage-platform#78

We need to file an upstream bug report to not assert false if there's an unknown os_type, as this happens if we just link Filename in

mirage-platform unix shouldnt need dllunixrun.so

This is only there for checksum_stubs now, which we should be able to remove and move into mirage-net.

1.0 Release checklist

Usecases:

• Decks.OpenMirage.org
• WWW.OpenMirage.org (Xen mode, content served from a block device)
• WWW.OpenMirage.org (Crunched)
• Personal home pages for: @samoht @mor1 @djs55 @amirmc @jonludlam
• Move repo to mirage org to keep history (@avsm)
• Merge mirage repo with mirage-platform master (@avsm)
• Blog post about OSCon (@mor1)
• Fix mirage-skeleton and make root Makefile build everything (@djs55)
• Populate xen.org wiki (@djs55)
• Quick RT script (@avsm)
• Vchan blog post (@vbmithr)
• Link all the videos and blogs to mirage-www (@avsm)
• Configure Bytemark Xen machines (@djs55)
• Sanitize installation instructions (@mor1) -- will be checked by Prashanth and Chris.
• fix event race in activations.wait (@djs55 / @avsm )

DP2

• Try the new ocamldoc to build a trove (@avsm)
• OPAM 1.1 testing (@samoht) the week after
• XenSummit storage submission (@djs55) august end
• Multiple website + load balancer when it works
• Breaking up mirage-net
• Github trove? (@avsm)
• Load testing
• Openflow status
• Fix Mirari on Unix (@avsm)
• Add combinator interface to Mirari instead of concrete syntax (@avsm)
• Travis tests
• Self-updating website via Travis
• Port mirage-www to cowabloga (@avsm)

regression suite tool/lib paths

as you suggested, "need to sort out ./configure nonsense" :)

specifically, regression suite tests are in-tree so should use relative paths for MIRAGELIB