mirage / ocaml-fat

Read and write FAT format filesystems from OCaml

Pure OCaml implementation of the FAT filesystem

This library has two purposes:

1. primary: to allow the easy preparation of bootable disk images containing Mirage kernels
2. secondary: to provide a simple key=value store for Mirage applications

Note that "filesystems" are inherently legacy systems which modern Mirage applications will not use directly. The most likely use for this library is in booting a Mirage application via some kind of disk image.

Known limitations

• can only make FAT16 filesystems
• no VFAT support
• no exFAT support (see #78)
• corrupts under concurrent writes

PRs welcome to address any of these limitations!

ocaml-fat's Issues

Improve the `fat` command-line image

It's very hard currently to pick a file in the local filesystem and add it to the image. Parents directory are not implicitly created, and should be identical to the host filesystem. Would be nice to add more options there.

README should include information on what works and what doesn't

e.g., that format can only make FAT16 filesystems, whether VFAT is implemented, what the user can expect to work and not work when writing to the filesystem, etc.

Compilation and Tests Failing

Running dune runtest or dune build reports the following errors:

File "src/fat.ml", line 1:
Error: The implementation src/fat.pp.ml
       does not match the interface src/.fat_filesystem.objs/byte/fat.cmi:
       ... ...
       The value `get_partial' is required but not provided
       File "src/mirage_kv.mli", line 140, characters 2-98:
         Expected declaration
       In module KV_RO:
       The value `size' is required but not provided
       File "src/mirage_kv.mli", line 175, characters 2-60:
         Expected declaration

Blocks/Pages read/write

The mirage-block-solo5 only accepts reading and writing one block at a time in https://github.com/Solo5/solo5/blob/0eb8cb8f57943e7872a94e19695911a1e0b8aef6/bindings/hvt/block.c#L39-L42. In

ocaml-fat/src/fat.ml

Lines 134 to 136 in 9171848

	let page = alloc 4096 in
	let block_number = Int64.(div sector_number (of_int sectors_per_block)) in
	B.read device block_number [ page ] >>= function

, we ask to read 8 blocks of 512 bytes which fail when running with hvt as target.

The mirage-block-xen only accepts to read and write 1 page at a time in https://github.com/mirage/mirage-block-xen/blob/cf6d97c1f48a73baeedc57136028191db36d892f/lib/front/blkfront.ml#L461. In

ocaml-fat/src/fat.ml

Lines 96 to 104 in 9171848

	let page = alloc bps in
	B.get_info device >>= fun {sector_size; _} ->
	let rec loop = function
	| [] -> Lwt.return (Ok ())
	| (sector, buffer) :: xs ->
	let offset = sector * bps in
	let sector' = offset / sector_size in
	Cstruct.memset page 0;
	B.read device (Int64.of_int sector') [ page ] >>= function

, it reads a bytes_per_sector bytes (with a default value of 512 bytes) which raises the Buffer_not_exactly_one_page exception.

I tested changing the page allocation from 4096 to 512 and from bps to 4096 respectively in https://github.com/palainp/ocaml-fat. Now this works with my small tests, I have a "Test Successful" when I run the dune test bench, but, as I don't have a clear vision of what could be impacted, it may trigger new bugs elsewhere. Does anyone have any idea what these changes may entail?

Fat tool: add an MBR, partition table?

If we routinely add an MBR, partition table then

1. Our images will look more normal
2. We can use the MBR disk signature as a unique name in mirage Block.connect

To make Block.connect nice, we may need to add a virtual block device abstraction somewhere. Perhaps ocaml-MBR needs to implement Block?

'fat': adding a 8.3 filename, fails a sanity check

djs@debian:~/djs55/ocaml-fat$ ./main.native create foo
djs@debian:~/djs55/ocaml-fat$ ./main.native add foo lib
copyin lib/fat_format.mli to /lib/fat_format.mli
copyin lib/META to /lib/META
fat: listdir(/lib) = [ 'META.   '(8), 'fat_format.mli'(14),
     '.memoryIO.ml.swp'(16), 'fs.mli'(6), 'fat.mllib'(9),
     '.name.mli.swp'(13), '.fs.ml.swp'(10), 'fat_format.ml'(13),
     'name.mli'(8), 's.ml'(4), '.sectorMap.ml.swp'(17), 'path.ml'(7),
     'path.mli'(8), 'sectorMap.mli'(13), 'boot_sector.mli'(15),
     'memoryIO.mli'(12), 'KV_RO.ml'(8), 'result.ml'(9), 'memoryIO.ml'(11),
     'update.ml'(9), 'result.mli'(10), 'name.ml'(7), 'fs.ml'(5),
     'update.mli'(10), 'entry.ml'(8), 'entry.mli'(9), 's.ml.rej'(8) ],
     doesn't include 'META'(4)
djs@debian:~/djs55/ocaml-fat$ echo $?

can't create / connect to fat file on `tmpfs`

Doing a simple:

strace -f fat create /tmp/out.fat 5MiB

Gives me the following relevant sequence:

[pid 10083] open("/tmp/out.fat", O_RDWR|O_CREAT, 0644) = 4
(...)
[pid 10083] write(4, "All work and no play makes Dave "..., 512) = 512
( ... )
[pid 10083] close(4)                    = 0
( ... )
[pid 10082] open("/tmp/out.fat", O_RDWR|O_DIRECT) = -1 EINVAL (Invalid argument)
[pid 10082] open("/tmp/out.fat", O_RDONLY|O_DIRECT) = -1 EINVAL (Invalid argument)
[pid 10082] write(2, "fat: connect /tmp/out.fat: faile"..., 47fat: connect /tmp/out.fat: failed to open file
) = 47

(i.e it does the initial write, but then the connect() fails)

Seems like perhaps O_DIRECT won't work on:

tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel)

Compared to a file in the current working directory, which works:

[pid 10611] open("out.fat", O_RDWR|O_DIRECT) = 4

Multiple errors with image files

I'm using the mirage-dev OPAM repository.

If I run a unikernel that uses a fat image (fat_of_files ~regexp:".gitignore" () in config.ml):

2016-10-14 15:06:45 +02:00: ERR [mirage-block-unix] read: Invalid argument in read '' at file /path/to/unikernel/fat_block1.img offset 0 with buffer of length 512
2016-10-14 15:06:45 +02:00: ERR [application] main: Fs.Make(B)(M).Block_device_error(_)
Raised at file "format.ml", line 185, characters 41-52
Called from file "format.ml", line 427, characters 6-24

See https://github.com/g2p/irc-unikernel/tree/fat-test-case for a full test case. Use the unix target.

The image isn't properly formatted and doesn't contain .gitignore (only the message about Dave being a dull boy).
If I try to recreate the image:

$ fat create fat_block1.img $((16*2**20)) || echo FAIL
fat: fat_block1.img already exists
FAIL
$ rm -f fat_block1.img
$ fat create fat_block1.img $((16*2**20)) || echo FAIL
fat: connect buffered:fat_block1.img: failed to open file
FAIL

The image did appear, despite the exit status.
But it's still badly formatted and the unikernel can't use it.

Using mkfs.vfat, the unikernel fails with the same error.

needs to be ported to mirage 2.3.0 types

Should be a small patch, and if we dont it risks rolling back libs from mirage

ocaml/opam-repository#3777

Writing to a new path leads to "Assertion failured"

Test-case:

open Lwt

module F = Fat.Fs.Make(Block)(Io_page)

let (>>|=) x f = x >>= function
  | `Error e -> failwith (Fat.Fs.string_of_filesystem_error e)
  | `Ok v -> f v

let main =
  Block.connect "disk.img" >>= function
  | `Error (`Unknown msg) -> failwith msg
  | `Error _ -> failwith "Block error"
  | `Ok block ->
  F.connect block >>|= fun fs ->
  F.format fs 0x100000L >>|= fun () ->
  let data = Cstruct.of_string "hi" in
(*   F.create fs "/data" >>|= fun () -> *)
  F.write fs "/data" 0 data >>|= fun () ->
  F.listdir fs "/" >>|= fun items ->
  List.iter (Printf.printf "Item: %s\n") items;
  print_endline "Done listing.";
  return ()

let () = Lwt_unix.run main

Results:

Fatal error: exception Failure("Unknown error: File "lib/fs.ml", line 263, characters 22-28: Assertion failed")

The comment and/or code in chain_of_file looks wrong. It says it returns None if the path doesn't exist, but in fact it asserts false in that case:

(** [chain_of_file device fs path] returns [Some chain] where [chain] is the chain
    corresponding to [path] or [None] if [path] cannot be found or if it
    is / and hasn't got a chain. *)
let chain_of_file device fs path =
  if Path.is_root path then return None
  else
    let parent_path = Path.directory path in
    find device fs parent_path >>= fun entry ->
    match entry with
    | `Ok (Dir ds) ->
      begin match Name.find (Path.filename path) ds with
        | None -> assert false
        | Some f ->
          let start_cluster = (snd f.Name.dos).Name.start_cluster in
          return (Some(Entry.Chain.follow fs.format fs.fat start_cluster))
      end
    | _ -> return None

I tried modifying it to return None instead, but that causes corruption because the of_file function treats None as indicating the root directory, causing the write to corrupt that.

Do not define the Fs_error exception inside the functor

Because of this. It is impossible to pattern match outside the functor. Or at least, provide a pretty printer for the error to make debugging easier.

listdir shows items that don't exist

Test-case (creates "/data", deletes it, then lists "/"):

open Lwt

module F = Fat.Fs.Make(Block)(Io_page)

let (>>|=) x f = x >>= function
  | `Error e -> failwith (Fat.Fs.string_of_filesystem_error e)
  | `Ok v -> f v

let main =
  Block.connect "disk.img" >>= function
  | `Error (`Unknown msg) -> failwith msg
  | `Error _ -> failwith "Block error"
  | `Ok block ->
  F.connect block >>|= fun fs ->
  F.format fs 0x100000L >>|= fun () ->
  F.create fs "/data" >>|= fun () ->
  F.destroy fs "/data" >>|= fun () ->
  F.listdir fs "/" >>|= fun items ->
  List.iter (Printf.printf "Item: %s\n") items;
  print_endline "Done listing.";
  return ()

let () = Lwt_unix.run main

This outputs:

Item:��K�
Done listing.

The CLI should use buffered I/O

... because unbuffered I/O is very slow.

`fat create` tries to open a file named "buffered:<...>"

When creating a fat file it tries to literally open buffered:<...> when it comes time to connect:

$ strace -f fat create out.fat 5MiB
( ... )
[pid 10743] open("out.fat", O_RDWR|O_CREAT, 0644) = 4
( ... )
[pid 10743] write(4, "All work and no play makes Dave "..., 512) = 512
( ... )
[pid 10742] open("buffered:out.fat", O_RDWR) = -1 ENOENT (No such file or directory)
[pid 10742] open("buffered:out.fat", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 10742] write(2, "fat: connect buffered:out.fat: f"..., 51fat: connect buffered:out.fat: failed to open file
) = 51

Adding --unbuffered is a simple workaround, but it's a bit odd.

I feel like this could well be a dependency version issue, as the code looks like it should handle this. I'm building under opam2nix, which should produce the same solve results as opam does (unless there's bugs, which there definitely could be), so here are all my dependency versions in case that's relevant:

https://gist.github.com/timbertson/0598e79a43a0483a033ad5a0447942a6

Error: The implementation lib/KV_RO.m does not match the interface lib/KV_RO.cmi

Got an error trying to build mirage-fat, (I am trying to get the fat binary built), I get this error message:

# W: Cannot find source file matching module 'fat' in library fat
# E: Failure("Command ''/usr/bin/ocamlbuild' lib/fat.cma lib/fat.cmxa lib/fat.a lib/fat.cmxs fat/main.native shell/main.native lib_test/test.native -tag debug -tag tests -j 4' terminated with error code 10")
# make: *** [build] Error 1

I used this Dockerfile to build mirage-fat:

FROM ubuntu:16.04

RUN DEBIAN_FRONTEND=noninteractive apt-get update -y && \
    apt-get install -y  ocaml-nox-4.02.3 ocaml-native-compilers opam m4 pkg-config && \
    apt-get clean -y && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*


RUN opam init -y && cd /tmp/ && \
    git clone https://github.com/mirage/ocaml-fat && \
    cd /tmp/ocaml-fat && \
    opam pin add ocaml-fat . -n -y && \
    opam install ocaml-fat --verbose

(place it in an empty folder and do docker build .)

I tried checking out and building old versions, going back to early September, all failed..

Help is appreciated!
Yuval

Error while compiling the dev version

I've got this message after adding the mirage opam-repo-dev remote:

+ /Users/thomas/.opam/system/bin/ocamlfind ocamlopt -g -linkpkg -package cmdliner -package cstruct -package io-page.unix -package lwt -package mirage-block-unix -package mirage-types -package re -package re.str fat/common.cmx lib/fat.cmxa fat/impl.cmx fat/main.cmx -o fat/main.native
# File "_none_", line 1:
# Error: No implementations provided for the following modules:
#          Io_page referenced from fat/impl.cmx

Support exFAT filesystem