Secrets, config maps, and other minimal volume support about telepresence

[itamarst]

It seems bindfs doesn't support nested mounts, which means it will impact full system... not ideal.

[itamarst]

More OS X resources:

• http://unix.stackexchange.com/questions/198590/what-is-a-bind-mount
• http://osxbook.com/book/bonus/ancient/whatismacosx/arch_fs.html (some parts are no longer available, e.g. synthfs/nullfs).

[itamarst]

The following scheme appears to work on OS X (requires brew cask install osxfuse and brew install homebrew/fuse/bindfs):

Create a RAM disk:

$ hdid -nomount ram://1024
/dev/disk2
$ newfs_hfs /dev/disk2
$ mount -t hfs /dev/disk2 ~/temp
$ echo hello > ~/temp/a
$ umount ~/temp

Now bind-mount / and then union-mount the RAM disk:

$ mkdir ~/staging
$ bindfs / ~/staging
$ mount -t hfs -o union /dev/disk2 ~/staging

~/staging is a now combination of / plus extra files from the RAM disk. You can now do:

$ chroot ~/staging

And resulting process now has files available to it both from normal filesystem and additional files from RAM disk. These additional files can be secrets/configmaps/etc. copied over from Kubernetes by parsing the Deployment volumes.

[itamarst]

Simpler alternative, without RAM disk:

$ mkdir ~/staging
$ echo hello > ~/staging/secret
$ bindfs -o union / ~/staging
$ chroot ~/staging

This will have problems though if / has stuff that conflicts with paths we want to inject, since it will come last. So possibly more complex scheme is necessary.

[itamarst]

Rather than copying files from Kubernetes, even nicer would be to use sshfs to just make remote volumes on pod locally accessible.

On Linux bind mounts of root and volumes will do the trick.

On OS X it's tricker. Ideally it would look like this:

$ bindfs / ~/staging  # or mount --bind on linux, say
$ sshfs -o union root@pod:/ ~/staging/

The problem is that on Mac only one osxfuse filesystem can be mounted at same time, so this will not actually work... But there is a potential workaround: you can union mount to a parent directory.

On remote pod:

$ cd /
$ mkdir staging
$ mount --bind / /staging

Now, on Mac:

$ mkdir -p ~/mount/staging
$ bindfs / ~/mount/staging
$ sshfs -o union root@pod:/ ~/mount

Now ~/mount/staging is pod:/staging, which has same contents as pod:/, and so we can do the chroot. TADA! 🙈

In practice rather than mount --bind / on pod, we want to do one bind mount per volume specified in the Kubernetes Deployment so that only actual volumes are made visible in the local process.

[itamarst]

On Linux proot seeminly works... but isn't packaged for Fedora. Another option:

1. unshare --mount --map-root-user, which gives new user and mount namespaces.
2. Create fake root directory, with directories where we want to do bind mounts and then overlayfs / on top of that. That gives us normal / + a few extra directories.
3. Bind mount remote volumes on to that.
4. chroot on to that.

Downsides: assuming it works, the unshare --map-root-user makes the user look like root, which is confusing. Also it has... problems with /dev/null/ and the like.

Probably better to go with proot.

[itamarst]

Status report: I have working transparent volume support on both OS X and Linux, with bindfs+chroot and proot respectively.

Unfortunately bindfs has ... issues. The result is that networking doesn't work on OS X anymore, even if volumes do, because bindfs breaks certain things.

The goals, to make them clear, were:

1. Transparency: volumes on remote container appear at same path on local machine. This means code can just hard code paths.
2. Isolation: local access to remote volumes is limited only to the process being wrapped by --run-shell.

The current solution achieves these both, at the unacceptable cost of breaking networking.

Some options that would preserve transparent and isolated volumes:

• Try to fix bindfs.
• Figure out if fakechroot works/can be made to work on OS X (https://github.com/dex4er/fakechroot/), and somehow combine with sshfs union mounts. In particular it allows symlinks out of the fake chroot, so building up a parallel structure of symlinks, combined with union mount might work. maybe?
• Switch to something like https://github.com/probonopd/user-union, i.e. custom LD_PRELOAD code.

The other option is to give up on transparency. Instead of volumes appearing at same path, they appear inside temporary directory whose address is accessible at an environment variable, e.g. TELEPRESENCE_ROOT. Users of Telepresence then need to modify their code. E.g. this:

secret_file = open("/app/secrets")

will become:

volume_root = os.environ.get("TELEPRESENCE_ROOT", "/")
secret_file = open(os.path.join(volume_root, "app/secrets"))