akhramov / knast

[discontinued] Experimental OCI & CRI-compatible container runtimes for FreeBSD

KNAST

Knast is FreeBSD experimental toolkit for building a modern container infrastructure. This is a research project project not intended to be used in production. See Goals.

Status

Currently, knast provides an OCI-compatible container runtime, which diverges from the etalon realization in some places. Namely,

• runc start doesn’t detach process.
• runc kill cannot be applied to created process.
• Non-standard extensions, such as runc run are not implemented.

Kernel requirements

Knast runs on amd64 FreeBSD 13.0-STABLE Kernel and newer. Support for aarch64 is planned.

Knast runs on GENERIC kernel, but requires several kernel modules to be loaded (or compiled in kernel). These are

• if_bridge is required for setting up VNET networking for jail.
• if_epair is required for setting up VNET networking for jail
• linux64 (optional) is required for linux jails
• pf firewall is required for networking, use pf service to load it.

Following sysctl variables need to be set:

• net.inet.ip.forwarding -> 1 for jail networking.

Usage

Building

cargo build release

Creating a bundle

To run a container OCI runtimes need a runtime bundle. Runtime bundles are built from OCI images. Knast is a mere runtime and not responsible for creating these files for you. You can use third party tools to create a bundle, or create it manually.

For convenience, we provide a util to fetch and unpack OCI images from docker registry. For instance, following command will create a runtime bundle from oldoldstable debian:

fetch_image debian:oldoldstable-20201209-slim

fetch_image will create a bundle somewhere in a $HOME directory the exact location will be printed.

In this example we fetched the oldoldstable debian, whose binaries still rely on older kernel ABI which is likely will be covered by Linuxulator.

OCI lifecycle

Once the bundle is created, you can create a container.

Navigate to runc folder and build the project using cargo build.

Following will create a VNET jail, configure network for it, mount all necessary devices and so on.

runc create debian /home/akhramov/containers/d19a2ab9-af67-4d04-8aef-9c364686c4fb

Then you will be able to start the container

runc start debian

Finally, you can delete the stopped container

runc delete debian

Apart from that, you can query containers state using state command and send signals to the container using kill. Please note that kill diverges from the etalone runc realization in sense that it only support signal numbers, not names.

Project structure

This project consists of several libraries, namely

• netzwerk contains network-related routines. Setting up interfaces, NATs, etc
• registratur is a client library for docker registry. It’s just a convenience library containing types & HTTP client and does not directly serve project goals. This functionality is to be handled by other tools.
• baustelle builds containers from images. It even supports a limited subsets of dockerfiles, though, just like with registratur this functionality is to be handled by other tools.
• storage provides storage-agnostic embedded db. Is used by runc to store containers state and other metadata.
• runc provides an OCI compatible runc binary.

Goals

Knast is a research project conducted (in order of priority) to

• examine capabilities of Rust in niche of OS-level virtualization, in vein of Oracle’s railcar. Long term goal is to implement modules for the popular orchestration / containerization solutions, like a CNI plugin.
• Examine feasibility to implement a container runtime without userspace dependencies: a self-sufficient binary which doesn’t rely on presence of any binaries in system.

  That implies that we will need to rewrite bindings to mount, devfs, route, partially implement functionality provided by ifconfig. Again, such a good test for Rust :)

• Build a runtime for containerd, a CRI, etc. Which has lower priority.

State

Not ready, please don’t use it anywhere.

Other efforts.

• https://github.com/samuelkarp/runj this project will likely achieve production-readiness sooner and reuses stable, reliable tooling which is already there. Please check it out.

Contributing

If you would like to contribute – please do. Check the list of open issues and tackle any task you want in regards to project priorities.

Testers, bug reporters, ocasional users – I love you all.

knast's Issues

containerd-shim

For containerd integration, we need to provide a containerd shim.

containerd shim is a ttrpc (gRPC) service (.proto file, documentation)

Conventionally, shims are wrappers around runtime binaries. However, we can experiment a bit and implement container routines in the shim itself.

Continious integration

Set up pull requests pipelines using GH Action runners.

Since GitHub doesn't provide FreeBSD environments (actions/runner-images#2529), we need to set up Vagrant boxes for

• latest release version (13.0 as of time of writing)
• current development version (14.0-CURRENT as of time of writing)

The CI should have following capabilities

• Runs tests for all crates in this repo (need to set up a Cargo Workspace)
• Collects and reports coverage (https://github.com/marketplace/codecov)
• Uses caching techniques to minimize build time (https://github.com/mozilla/sccache)

Deprecate and remove baustelle module.

It will be more productive to focus on porting & improving buildkit.

Relicense

MIT or BSD 2-Clause

Documentation

• Project description & vision
• Contributing guide
• Roadmap

rlimits support

Quoting runtime-spec

For systems that support POSIX rlimits (for example Linux and Solaris), the process object supports the following process-specific properties...

FreeBSD definitely supports rlimits, see getrlimit(2). According to rctl(8), it's possible to set rlimits on a jail base, e.g. following will limit the amount of memory jail knast can allocate to 1g.

rctl -a jail:knast:vmemoryuse:deny=1g

The scope of this task

• write bindings to programmatically mimic the behavior of rctl(8)
• support runtime config's process.limits option
• figure out the sane defaults. E.g. podman generates runtime spec like this

  {
    "type": "RLIMIT_NOFILE",
    "hard": 524288,
    "soft": 524288
  },
  {
    "type": "RLIMIT_NPROC",
    "hard": 32768,
    "soft": 32768
  }

• Contribute these defaults to containerd

Research & document containerd integration

In order to provide support for containerd, our implementation of runc needs to

• implement not only standard operations, but also additional, such as
  • exec
  • running in detached mode
• implement "containerd-shim", an interface for containerd.

The scope of this task is to

• document in free-form what has to be done in order to integrate with containerd
• document in free-form how to build containerd on FreeBSD
• document in free-form how containerd allocates network addresses for containers. Currently we set up networking right within runtime, but probably it should be done somehow else
• (optional) create a proof of concept of containerd-shim

Research

• containerd (how to integrate? https://github.com/samuelkarp/runj)
• OCI & CNI