bazelbuild / sandboxfs

A virtual file system for sandboxing

sandboxfs: A virtual file system for sandboxing

sandboxfs is a FUSE file system that exposes a combination of multiple files and directories from the host's file system in the form of a virtual tree with an arbitrary layout. You can think of a sandbox as an arbitrary view into the host's file system with different access privileges per directory.

sandboxfs is designed to allow running commands with limited access to the file system by using the virtual tree as their new root, and to do so consistently across a variety of platforms.

sandboxfs is licensed under the Apache 2.0 license and is not an official Google product.

Releases

The latest version of sandboxfs is 0.2.0 and was released on 2020-04-20.

See the installation instructions for details on how to build and install sandboxfs.

See the release notes file for more details.

Usage

sandboxfs is fully documented in the sandboxfs(1) manual page, which is located in the man/sandboxfs.1 file. You can view a rendered version of this manual page using the following command after cloning the tree:

man ./man/sandboxfs.1

Contributing

If you'd like to contribute to sandboxfs, there is plenty of work to be done! Please make sure to read our contribution guidelines to learn about some important prerequisite steps.

sandboxfs's Issues

EOPNOTSUPP not exported by nix crate on FreeBSD

On FreeBSD the libc crate defines ENOTSUP as an alias for EOPNOTSUPP. The nix crate however only exports ENOTSUP. Would it be possible to switch to using ENOTSUP in sandboxfs instead for FreeBSD? It's likely the same way on the other BSDs.

sandboxfs/src/nodes/mod.rs

Line 117 in 6d8108f

return Err(nix::Error::from_errno(Errno::EOPNOTSUPP));

sandboxfs/src/nodes/mod.rs

Line 155 in 6d8108f

return Err(nix::Error::from_errno(Errno::EOPNOTSUPP));

Upgrade CI to modern macOS

Our Travis CI builds for macOS currently rely on the image that provides El Capitan because there is no known mechanism to load the FUSE kext without interacting with the UI (travis-ci/travis-ci#10017).

This is problematic because we cannot use modern APIs in our code (things like futimensat), which are necessary to implement and/or test certain semantics.

sandboxfs performance impact on an iOS project

I just wanted to provide some numbers for our iOS project when using sandboxfs. If this isn't useful for you, feel free to close it.

Note: I ran clean --expunge between each build:

Default sandboxing:

% USE_BAZEL_VERSION=last_green bazelisk build --config=debug Modules/PassengerApp:Lyft
2019/03/02 12:22:49 Using unreleased version at commit 1bc7d14739e1c2359f4b91d7aff422f550a53c27
Starting local Bazel server and connecting to it...
INFO: Analysed target //Modules/PassengerApp:Lyft (320 packages loaded, 11106 targets configured).
INFO: Found 1 target...
Target //Modules/PassengerApp:Lyft up-to-date:
  bazel-bin/Modules/PassengerApp/Lyft.ipa
INFO: Elapsed time: 478.154s, Critical Path: 368.44s
INFO: 1658 processes: 1146 darwin-sandbox, 512 local.
INFO: Build completed successfully, 2172 total actions

Sandboxfs:

% USE_BAZEL_VERSION=last_green bazelisk build --config=debug Modules/PassengerApp:Lyft --experimental_use_sandboxfs --experimental_sandboxfs_path=`which sandboxfs`
22019/03/02 12:31:23 Us019/03/02 12:31:23 Using unreleased version at commit 1bc7d14739e1c2359f4b91d7aff422f550a53c27
Starting local Bazel server and connecting to it...
INFO: Analysed target //Modules/PassengerApp:Lyft (320 packages loaded, 11106 targets configured).
INFO: Found 1 target...
Target //Modules/PassengerApp:Lyft up-to-date:
  bazel-bin/Modules/PassengerApp/Lyft.ipa
INFO: Elapsed time: 662.932s, Critical Path: 592.93s
INFO: 1658 processes: 1146 darwin-sandbox, 512 local.
INFO: Build completed successfully, 2172 total actions

Local:

% USE_BAZEL_VERSION=last_green bazelisk build --config=debug Modules/PassengerApp:Lyft --config=nosandbox
2019/03/02 12:43:33 Using unreleased version at commit 1bc7d14739e1c2359f4b91d7aff422f550a53c27
Starting local Bazel server and connecting to it...
INFO: Analysed target //Modules/PassengerApp:Lyft (320 packages loaded, 11106 targets configured).
INFO: Found 1 target...
Target //Modules/PassengerApp:Lyft up-to-date:
  bazel-bin/Modules/PassengerApp/Lyft.ipa
INFO: Elapsed time: 265.130s, Critical Path: 207.45s
INFO: 1658 processes: 1658 local.
INFO: Build completed successfully, 2172 total actions

I can't provide the full project to reproduce this, but if there's any other specific questions I can answer I can try to provide other useful info

Directory becomes temporarily inaccessible after it has been moved

When moving directories inside a sandbox, it takes a few seconds for the directories to become accessible. Directories are listed, but ls cannot stat() the directory entries.

For reproducing the issue, Bash auto completion has to be used when typing the directory names. By copy-pasting the commands as is, the directory that gets moved is immediately accessible.

Terminal 1:

$ mkdir -p ./out ./new_sandbox
$ sandboxfs --version
sandboxfs 0.1.1
$ sandboxfs --mapping=ro:/:/ --mapping=rw:/out:$PWD/out ./new_sandbox
...

Terminal 2:

$ cd new_sandbox/out/
$ mkdir target to_be_moved
$ mv to_be_moved/ target/  # <- Use TAB here for auto completion
$ while sleep 1; do date && ls -l target; done
Wed Feb  5 13:26:07 EET 2020
ls: cannot access 'target/to_be_moved': No such file or directory
total 0
d????????? ? ? ? ?            ? to_be_moved
Wed Feb  5 13:26:08 EET 2020
ls: cannot access 'target/to_be_moved': No such file or directory
total 0
d????????? ? ? ? ?            ? to_be_moved

...

Wed Feb  5 13:26:52 EET 2020
total 0
drwxrwxr-x 2 tkuosmanen tkuosmanen 2 Feb  5 13:25 to_be_moved

In the example above, it took roughly 45 seconds for the directory to become accessible.

This was run on Ubuntu 16.04 workstation with 4.4.0-133-generic kernel and ext4 root file system. Sandboxfs is pre-built release 0.1.1 from sandboxfs-0.1.1-20191024-linux-x86_64.tgz.

The issue has been reproduced also on Ubuntu 18.04 workstation with 4.15.0-72-generic kernel.

Tar: file changed as we read it

Issue

When using tar to create/modify an archive inside a sandboxfs filesystem, sometimes tar will fail with the file changed as we read it error message.

When using sandboxfs this issue is observed regularly but not consistent. Without sandboxfs we never see the issue.

Minimal example

./BUILD
genrule(
    name = "test",
    tools = ["script.sh"],
    outs = ["test.tar"],
    cmd = "$(location script.sh) $@",
)

./script.sh
#!/bin/sh -e

for x in {0..1000}; do
    mkdir -p workdir
    touch workdir/$x.txt
done

tar -C workdir -cf $1 .

for x in {0..100}; do
    rm -rf workdir/*
    tar -C workdir -xf $1 .
    tar -C workdir -cf $1 .
done

Command: bazel clean && bazel build :test --experimental_use_sandboxfs --genrule_strategy=sandboxed --spawn_strategy=sandboxed

hanwen/go-fuse

Just out of curiosity, have you considered to use https://github.com/hanwen/go-fuse for FUSE go-binding? We are using that extensively for a similar use case and didn't notice any significant performance issue.

Fails to build on FreeBSD/i386

error[E0308]: mismatched types
  --> src/nodes/conv.rs:72:19
   |
72 |     Timespec::new(val.tv_sec() as sys::time::time_t, usec)
   |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected i64, found i32
help: you can cast an `i32` to `i64`, which will sign-extend the source value
   |
72 |     Timespec::new((val.tv_sec() as sys::time::time_t).into(), usec)
   |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

Running 'sandboxfs' with no arguments should show the usage message

Instead of just telling users to run sandboxfs --help to see the usage message

Device files are not supported

It appears that reading from and writing to device files is not supported via sandboxfs and osxfuse on macOS. Given the example command below:

sandboxfs --mapping ro:/:/ --mapping rw:/dev:/dev ./mnt

The following attempts to write to any device file in the mounted /dev subdirectory fail with the following error:

$ cd mnt
$ echo hi > ./dev/null
-bash: echo: write error: Operation not permitted
$ head -1 < ./dev/random
head: Error reading stdin

I have not verified if the same limitation exists on Linux as well at the time of writing, but it nevertheless would be quite nice if sandboxfs supported binaries or scripts which wrote to /dev/null if the user requested that the device file be mounted, for example.

Passing -h to sandboxfs returns an error but shows no message

Running sandboxfs with a lone -h flag results in the program returning with an error code but causes nothing to be printed to standard error. -h should be implemented to behave like --help or should be made to fail like any other unsupported option.

Track ctimes internally

sandboxfs currently does not track ctimes for the nodes it manages. Instead, it relies on the ctimes returned by the underlying file systems and propagates those to the caller.

This has unfortunate consequences: because we don't track ctimes, we cannot synthesize what the attributes of a file should be e.g. immediately after creation, which means we end up issuing extra stat calls in cases where we shouldn't.

So: track ctimes (and possibly also creation times) and avoid these unnecessary operations.

container_image file not found when used in genrule

Description of the problem / feature request:

Not sure if this issue belongs here or not.
Depending on a container_image in a genrule tools errors when sandboxfs is used

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

WORKSPACE

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "io_bazel_rules_docker",
    # HEAD at 2018/08/26
    urls = ["https://github.com/bazelbuild/rules_docker/archive/d6046608f4979ad891a8c0574dd07ca7bf5f870e.tar.gz"],
    sha256 = "6872b8b0b509c6c3fcc3876f9b009b1984d9c5de80f9234dd6bfb8d8c504f314",
    strip_prefix = "rules_docker-d6046608f4979ad891a8c0574dd07ca7bf5f870e",
)

load("@io_bazel_rules_docker//container:container.bzl", container_repositories = "repositories")
container_repositories()

BUILD.bazel

load("@io_bazel_rules_docker//container:container.bzl", "container_image")

container_image(
    name = "image"
)

genrule(
    name = "use-tool",
    tools = [":image"],
    outs = ["out.txt"],
    # works when run standalone
    # local = True,
    cmd = "cat $(location :image) > $@",
)

bazel build --sandbox_debug --verbose_failures //:use-tool

Loading: 
Loading: 0 packages loaded
INFO: Analysed target //:use-tool (0 packages loaded).
INFO: Found 1 target...
[0 / 2] [-----] BazelWorkspaceStatusAction stable-status.txt
ERROR: /home/aymanrady/dev/sandbox/bazel-docker-py/BUILD.bazel:7:1: Executing genrule //:use-tool failed (Exit 1): linux-sandbox failed: error executing command 
  (cd /home/aymanrady/.cache/bazel/_bazel_aymanrady/3e53b67470e506c3592f875ac1f72484/execroot/__main__ && \
  exec env - \
    PATH=/home/aymanrady/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    TMPDIR=/tmp \
  /home/aymanrady/.cache/bazel/_bazel_aymanrady/3e53b67470e506c3592f875ac1f72484/execroot/__main__/_bin/linux-sandbox -t 15 -w /home/aymanrady/.cache/bazel/_bazel_aymanrady/3e53b67470e506c3592f875ac1f72484/sandbox/linux-sandbox/1/execroot/__main__ -w /tmp -w /dev/shm -D -- /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; cat bazel-out/host/bin/image > bazel-out/k8-fastbuild/genfiles/out.txt')
src/main/tools/linux-sandbox.cc:154: linux-sandbox-pid1 has PID 6230
src/main/tools/linux-sandbox-pid1.cc:175: working dir: /home/aymanrady/.cache/bazel/_bazel_aymanrady/3e53b67470e506c3592f875ac1f72484/sandbox/linux-sandbox/1/execroot/__main__
src/main/tools/linux-sandbox-pid1.cc:194: writable: /home/aymanrady/.cache/bazel/_bazel_aymanrady/3e53b67470e506c3592f875ac1f72484/sandbox/linux-sandbox/1/execroot/__main__
src/main/tools/linux-sandbox-pid1.cc:194: writable: /tmp
src/main/tools/linux-sandbox-pid1.cc:194: writable: /dev/shm
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /dev
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /dev/pts
src/main/tools/linux-sandbox-pid1.cc:265: remount rw: /dev/shm
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /dev/mqueue
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /dev/hugepages
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /run
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /run/lock
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /run/rpc_pipefs
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /run/cgmanager/fs
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /run/user/1000
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /run/user/1000/gvfs
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /run/snapd/ns
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /run/docker/netns/default
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/kernel/security
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/systemd
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/cpuset
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/net_cls,net_prio
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/memory
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/freezer
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/blkio
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/devices
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/hugetlb
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/perf_event
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/cpu,cpuacct
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/rdma
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/pids
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/pstore
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/kernel/debug
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/kernel/config
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/fuse/connections
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /proc
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /proc/sys/fs/binfmt_misc
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /proc/sys/fs/binfmt_misc
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /proc/fs/nfsd
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/vlc/190
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/ubuntu-make/471
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/spotify/19
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/tldr/162
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/jq/6
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/core/5145
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/ubuntu-make/479
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/vlc/365
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/core/4917
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/ubuntu-make/492
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/http/21
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/vlc/158
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /snap/core/4830
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /home
src/main/tools/linux-sandbox-pid1.cc:265: remount rw: /home/aymanrady/.cache/bazel/_bazel_aymanrady/3e53b67470e506c3592f875ac1f72484/sandbox/linux-sandbox/1/execroot/__main__
src/main/tools/linux-sandbox-pid1.cc:265: remount rw: /home/aymanrady/.cache/bazel/_bazel_aymanrady/3e53b67470e506c3592f875ac1f72484/sandbox/linux-sandbox/1/execroot/__main__
src/main/tools/linux-sandbox-pid1.cc:265: remount rw: /tmp
src/main/tools/linux-sandbox-pid1.cc:265: remount rw: /dev/shm
src/main/tools/process-tools.cc:118: sigaction(32, &sa, nullptr) failed
src/main/tools/process-tools.cc:118: sigaction(33, &sa, nullptr) failed
cat: bazel-out/host/bin/image: No such file or directory
src/main/tools/linux-sandbox-pid1.cc:437: waitpid returned 2
src/main/tools/linux-sandbox-pid1.cc:457: child exited with code 1
src/main/tools/linux-sandbox.cc:204: child exited normally with exitcode 1
Target //:use-tool failed to build
INFO: Elapsed time: 0.385s, Critical Path: 0.09s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Any other information, logs, or outputs that you want to share?

Bazel 0.16.1
sandboxfs 625ac9a

Windows support?

Hi! Please forgive a rather uninformed question. Is support for Windows possible? Is it in scope?

lgetxattr returns ENOENT instead of ENODATA

There is an issue with fetching extended attributes on Ubuntu 16.04 platform. Using a build from
the latest top of tree (54362b7), a simple ls -la of the sandbox root gives No such file or directory error message. Commands like cp -a will return non-zero exit code, because they are unable to get the extended attributes of the source directories.

Terminal 1:

$ mkdir dummy_dir
$ cd dummy_dir/
$ echo dummy > regular_file.txt
$ mkdir regular_dir
$ mkdir new_sandbox
$ sandboxfs $PWD/new_sandbox

Terminal 2:

$ cd dummy_dir/
$ ls -la
ls: new_sandbox: No such file or directory  # <- This is caused by failing lgetxattr call.
total 17
drwxr-xr-x  4 tkuosmanen tkuosmanen 4096 Feb 19 10:50 .
drwxr-xr-x 14 tkuosmanen tkuosmanen 4096 Feb 19 10:49 ..
dr-xr-xr-x  2 tkuosmanen tkuosmanen    2 Feb 19 10:58 new_sandbox
drwxr-xr-x  2 tkuosmanen tkuosmanen 4096 Feb 19 10:49 regular_dir
-rw-r--r--  1 tkuosmanen tkuosmanen    6 Feb 19 10:49 regular_file.txt

$ sudo strace -u tkuosmanen ls -la
...
lstat("new_sandbox", {st_mode=S_IFDIR|0555, st_size=2, ...}) = 0
lgetxattr("new_sandbox", "security.selinux", 0x19b4d90, 255) = -1 ENOENT (No such file or directory)
write(2, "ls: ", 4ls: )                     = 4
write(2, "new_sandbox", 11new_sandbox)             = 11
write(2, ": No such file or directory", 27: No such file or directory) = 27
write(2, "\n", 1)                       = 1
...
lstat("regular_dir", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lgetxattr("regular_dir", "security.selinux", 0x19b7e30, 255) = -1 ENODATA (No data available)
getxattr("regular_dir", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available)
getxattr("regular_dir", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available)
...
lstat("regular_file.txt", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
lgetxattr("regular_file.txt", "security.selinux", 0x19b7e90, 255) = -1 ENODATA (No data available)
getxattr("regular_file.txt", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available)

I'm able to repro the issue on 4.4.0-133-generic and 4.4.0-171-generic kernels.

avoid format! in the `map` hot path

My product team is finally porting some C++ projects to Bazel, and we're using sandboxfs to reduce Bazel sandbox overhead of C++ compile actions which pull in thousands of headers from Boost. We were surprised to find out that tmpfs (--sandbox_base=/dev/shm) performed better than sandboxfs (thread). Profling sandboxfs, a flame graph revealed >50% of CPU time spent in create_sandbox, with the bulk of that in apply_mapping.

A surprising amount of time in apply_mapping was in Rust's string formatting code, which I tracked to here: https://github.com/bazelbuild/sandboxfs/blob/sandboxfs-0.2.0/src/nodes/dir.rs#L537-L543

By replacing the format!() macro with a static string, string formatting dropped out of apply_mapping's call graph and reduced create_sandbox's share of CPU time to 37%.

The tradeoff is that the error message, if ever reported to the user, no longer includes the path that we failed to stat. I'm a total Rust noob, so I don't yet know if there's a way to either turn this into a simple/cheap string concatenation or to lazily evaluate the format operation only upon failure.

Example usage of sandboxfs in the man page doesn't work

For example, the man page gives this as an example usage:

    sandboxfs \
         --read_only_mapping=/:/ \
         --read_write_mapping=/tmp:/tmp/fresh-tmp \
         /mnt

But this is what happens when I try it (on a Mac):

$ /usr/local/bin/sandboxfs --read_only_mapping=/:/ --read_write_mapping=/tmp:/tmp/fresh-tmp /mnt
flag provided but not defined: -read_only_mapping
Type 'sandboxfs --help' for details

broken result files when linking with _ld.gold_ is interrupted

Description of the problem

When I interrupt a link step of a C++ project, I get a result of in bazel-bin that has got the correct size but contains only nul bytes.

How To Reproduce The Problem

• set up a C++ project with lots of cc_binary rules
• make sure your toolchain uses ld.gold (-fuse-ld=gold)
• start the build with bazel build --experimental_use_sandboxfs
• press Ctrl-C
• look for files filled with nul bytes
  find bazel-bin/ -xtype f -size +0 -exec grep --text --files-without-match -P '[^\0]' {} + -ls

The problem also occurs when a build error occurs and Bazel interrupts currently running linking steps on its own (if you don't use --keep_going).

Environment

RHEL 7.4: Linux 3.10.0, CPU with four cores + HT, using devtool-10 (devtoolset-10-binutils-2.35-3.el7.x86_64).

SW Version

bazel 4.1.0
sandboxfs 0.2.0

Further Observations

ld.gold does not seem to make any effort to prevent broken output files if it gets killed: https://sourceware.org/bugzilla/show_bug.cgi?id=27972

The problem does not occur with GNU ld. GNU ld deletes unfinished output filse in an atexit handler.

Does sandboxfs only check for the existance of the output files and ignore the exit value of ld.gold?

Reach outside of sandboxfs

I am just debugging a nodejs rule that seems to behave in a non-hermetic way (see bazelbuild/rules_nodejs#612 if you are interested in background) and I thought sandboxfs can help me there with greater hermeticity, but the rule still behaves exactly the same way. So my question, given an absolute path is it still possible to access the filesystem outside of sandboxfs? And if so is there some way of blocking this?

Parsing of static flags differs from parsing of dynamic config

I noticed today that the way mapping paths are handled in static vs. dynamic configs differ: for the static case, we correctly sanitize the paths and require them to be absolute. For dynamic, no such validation occurs and non-absolute paths are silently accepted.

This is bogus. Both configuration approaches should match in behavior, which mean both must use the same data parsing code.

(And as reconfiguration_test.go says, we need many more tests for the data parsing.)

Files become inaccessible when their parent directory is moved

This issue affects the latest top of tree (855834e). Files become inaccessible when their parent directory is moved. Below is an error case that can be reliably reproduced with a build from 855834e:

Terminal 1:

mkdir -p ./out ./new_sandbox
sandboxfs --mapping=ro:/:/ --mapping=rw:/out:$PWD/out ./new_sandbox
...

Terminal 2:

$ cd new_sandbox/out/
$ mkdir target to_be_moved
$ echo dummy > to_be_moved/dummy.txt
$ mv to_be_moved/ target/
$ echo "contents of target:" && ls -l target && echo "contents of target/to_be_moved:" && ls -l target/to_be_moved
contents of target:
total 0
drwxrwxr-x 2 tkuosmanen tkuosmanen 2 Feb  7 14:18 to_be_moved
contents of target/to_be_moved:
ls: cannot access 'target/to_be_moved/dummy.txt': No such file or directory
total 0
-????????? ? ? ? ?            ? dummy.txt

File target/to_be_moved/dummy.txt is stuck in a bad state.

This was run on Ubuntu 16.04 workstation with 4.4.0-133-generic kernel and ext4 root file system.

Removing a file causes sandboxfs to crash

With a build of the latest top of tree (ef7ff49), sandboxfs crashes when removing a file. The parent directory of the file needs to be moved first and then removing the file hits some sort of cache corruption.

Terminal 1:

$ mkdir -p ./out ./new_sandbox
$ RUST_LOG=debug sandboxfs --mapping=ro:/:/ --mapping=rw:/out:$PWD/out ./new_sandbox

...

DEBUG 2020-02-11T08:20:06Z: fuse::request: UNLINK(24) parent 0x0000000000000004, name "dummy.txt"
thread 'main' panicked at 'Tried to delete unknown path from the cache', src/libcore/option.rs:1188:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
 WARN 2020-02-11T08:20:06Z: fuse::reply: Reply not sent for operation 24, replying with I/O error
DEBUG 2020-02-11T08:20:06Z: sandboxfs::concurrent: Closing ShareableFile with fd 3
 INFO 2020-02-11T08:20:06Z: fuse::session: Unmounted /home/tkuosmanen/new_sandbox
 INFO 2020-02-11T08:20:06Z: sandboxfs::reconfig: Reached end of reconfiguration input; file system mappings are now frozen

Terminal 2:

$ cd new_sandbox/out/
$ mkdir target to_be_moved
$ echo dummy > to_be_moved/dummy.txt
$ mv to_be_moved target
$ rm target/to_be_moved/dummy.txt
rm: cannot remove 'target/to_be_moved/dummy.txt': Input/output error

This was run on Ubuntu 16.04 workstation with 4.4.0-133-generic kernel and ext4 root file system.

Add a few words about building it and the rationale to combine go and rust within a single project

sandboxfs hangs when trying to mount

Hello ,

I am trying to build a hello-world c++ program using the sandboxfs.
My OS is RHEL 7.3 with {kernel 3.10 , libfuse.so.2.9.2} and the user has sudo access.
When I build with the command line "bazel build //main:hello-world --experimental_sandboxfs_path=/home/egechir/install_dir/sandboxfs/bin/sandboxfs --experimental_use_sandboxfs" , I get these log messages. In this case, Bazel invokes sandboxfs to mount on a certain mount point.
When I execute sandboxfs with the command "/home/egechir/install_dir/sandboxfs/bin/sandboxfs --allow=self ." , I get these log messages.

I found these two log messages interesting
DEBUG 2019-08-29T15:48:30Z: fuse::request: INIT(1) kernel: ABI 7.22, flags 0xf7fb, max readahead 131072
DEBUG 2019-08-29T15:48:30Z: fuse::request: INIT(1) response: ABI 7.8, flags 0x1, max readahead 131072, max write 16777216

My interpretation of above is that the rust-fuse receives the FUSE_INIT (kernel ABI 7.22) messaget from kernel driver and rust-fuse replies with FUSE_INIT( ABI 7.8). Then the kernel driver has either replied with a FUSE protocol message that it has not yet implemented in rust-fuse or the kernel replied nothing. In both those logs, one can see that when the sandboxfs process is either terminated or interrupted, the rust-fuse receives the FUSE_FORGET message from the kernel.

Does anyone know why the communication between sandboxfs and kernel stops when trying to mount ?
Are the ABIs of rust-fuse and kernel incompatible ?
Any advice on how to go forward ?

rules_nodejs tests failing

Clone https://github.com/bazelbuild/rules_nodejs and run bazel test --experimental_use_sandboxfs //... on latest master (I ran on commit 4d0d9ab457fc9eddc30571ac26aa1479f2afa57a) and you will see all tests failing with:

src/main/tools/linux-sandbox-pid1.cc:198: "mount(/home/markus/.cache/bazel/_bazel_markus/b5e90fb44c2dddce81f37d15a927885e/sandbox/linux-sandbox/84/execroot/build_bazel_rules_nodejs/_tmp/ac725c8d6b5fe71de53c076523af737f, /home/markus/.cache/bazel/_bazel_markus/b5e90fb44c2dddce81f37d15a927885e/sandbox/linux-sandbox/84/execroot/build_bazel_rules_nodejs/_tmp/ac725c8d6b5fe71de53c076523af737f, nullptr, MS_BIND | MS_REC, nullptr)": No such file or directory

I am running on Linux elementary os 5 (Ubuntu 18.04) and bazel 0.23.0rc3. This might very well be an issue with rules_nodejs and not sandboxfs but it is strange that the error comes directly from linux-sandbox-pid1.cc. If you prefer I am also happy to open up an issue on rules_nodejs. Just let me know.

building rules_go fails

I was curios to test out the new sandboxfs version but it seems there is an issue building (certain) go applications. As an easy way to reproduce clone bazelbuild/rules_go@6a85f5d and run bazel build //... --experimental_use_sandboxfs, building without sandboxfs works.

The error I am seeing:

INFO: Mounting sandboxfs instance on /home/markus/.cache/bazel/_bazel_markus/1ae6fc80da3e13e6eb67d84f874de1f0/sandbox/sandboxfs
ERROR: /home/markus/.cache/bazel/_bazel_markus/1ae6fc80da3e13e6eb67d84f874de1f0/external/io_bazel_rules_go/BUILD.bazel:9:1: GoStdlib external/io_bazel_rules_go/plan9_amd64_pure_stripped/stdlib%/pkg failed (Exit 1) stdlib failed: error executing command bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/stdlib -sdk external/go_sdk -installsuffix plan9_amd64 -out ... (remaining 3 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
can't load package: package internal/syscall/unix: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/1ae6fc80da3e13e6eb67d84f874de1f0/sandbox/sandboxfs/101/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/plan9_amd64_pure_stripped/stdlib%/src/internal/syscall/unix
can't load package: package os/signal/internal/pty: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/1ae6fc80da3e13e6eb67d84f874de1f0/sandbox/sandboxfs/101/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/plan9_amd64_pure_stripped/stdlib%/src/os/signal/internal/pty
can't load package: package runtime/msan: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/1ae6fc80da3e13e6eb67d84f874de1f0/sandbox/sandboxfs/101/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/plan9_amd64_pure_stripped/stdlib%/src/runtime/msan
can't load package: package syscall/js: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/1ae6fc80da3e13e6eb67d84f874de1f0/sandbox/sandboxfs/101/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/plan9_amd64_pure_stripped/stdlib%/src/syscall/js
can't load package: package vendor/golang_org/x/net/lif: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/1ae6fc80da3e13e6eb67d84f874de1f0/sandbox/sandboxfs/101/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/plan9_amd64_pure_stripped/stdlib%/src/vendor/golang_org/x/net/lif
can't load package: package vendor/golang_org/x/net/route: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/1ae6fc80da3e13e6eb67d84f874de1f0/sandbox/sandboxfs/101/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/plan9_amd64_pure_stripped/stdlib%/src/vendor/golang_org/x/net/route
GoStdlib: error running subcommand: exit status 1
INFO: Build complete; unmounting sandboxfs...

I built sandboxfs from latest master here and I am using bazel version 0.22.0rc1