cri-o / ocicni

CRI-O - OCI-based implementation of Kubernetes Container Runtime Interface

Compatibility matrix: CRI-O ⬄ Kubernetes

CRI-O follows the Kubernetes release cycles with respect to its minor versions (1.x.y). Patch releases (1.x.z) for Kubernetes are not in sync with those from CRI-O, because they are scheduled for each month, whereas CRI-O provides them only if necessary. If a Kubernetes release goes End of Life, then the corresponding CRI-O version can be considered in the same way.

This means that CRI-O also follows the Kubernetes n-2 release version skew policy when it comes to feature graduation, deprecation or removal. This also applies to features which are independent from Kubernetes. Nevertheless, feature backports to supported release branches, which are independent from Kubernetes or other tools like cri-tools, are still possible. This allows CRI-O to decouple from the Kubernetes release cycle and have enough flexibility when it comes to implement new features. Every feature to be backported will be a case by case decision of the community while the overall compatibility matrix should not be compromised.

For more information visit the Kubernetes Version Skew Policy.

CRI-O	Kubernetes	Maintenance status
main branch	master branch	Features from the main Kubernetes repository are actively implemented
release-1.x branch (v1.x.y)	release-1.x branch (v1.x.z)	Maintenance is manual, only bugfixes will be backported.

The release notes for CRI-O are hand-crafted and can be continuously retrieved from our GitHub pages website.

What is the scope of this project?

CRI-O is meant to provide an integration path between OCI conformant runtimes and the Kubelet. Specifically, it implements the Kubelet Container Runtime Interface (CRI) using OCI conformant runtimes. The scope of CRI-O is tied to the scope of the CRI.

At a high level, we expect the scope of CRI-O to be restricted to the following functionalities:

• Support multiple image formats including the existing Docker image format
• Support for multiple means to download images including trust & image verification
• Container image management (managing image layers, overlay filesystems, etc)
• Container process lifecycle management
• Monitoring and logging required to satisfy the CRI
• Resource isolation as required by the CRI

What is not in the scope of this project?

• Building, signing and pushing images to various image storages
• A CLI utility for interacting with CRI-O. Any CLIs built as part of this project are only meant for testing this project and there will be no guarantees on the backward compatibility with it.

CRI-O is an implementation of the Kubernetes Container Runtime Interface (CRI) that will allow Kubernetes to directly launch and manage Open Container Initiative (OCI) containers.

The plan is to use OCI projects and best of breed libraries for different aspects:

• Runtime: runc (or any OCI runtime-spec implementation) and oci runtime tools
• Images: Image management using containers/image
• Storage: Storage and management of image layers using containers/storage
• Networking: Networking support through the use of CNI

It is currently in active development in the Kubernetes community through the design proposal. Questions and issues should be raised in the Kubernetes sig-node Slack channel.

Roadmap

A roadmap that describes the direction of CRI-O can be found here. The project is tracking all ongoing efforts as part of the Feature Roadmap GitHub project.

CI images and jobs

CRI-O's CI is split-up between GitHub actions and OpenShift CI (Prow). Relevant virtual machine images used for the prow jobs are built periodically in the jobs:

• periodic-ci-cri-o-cri-o-main-periodics-setup-periodic
• periodic-ci-cri-o-cri-o-main-periodics-setup-fedora-periodic
• periodic-ci-cri-o-cri-o-main-periodics-evented-pleg-periodic

The jobs are maintained from the openshift/release repository and define workflows used for the particular jobs. The actual job definitions can be found in the same repository under ci-operator/jobs/cri-o/cri-o/cri-o-cri-o-main-presubmits.yaml for the main branch as well as the corresponding files for the release branches. The base image configuration for those jobs is available in the same repository under ci-operator/config/cri-o/cri-o.

Commands

Command	Description
crio(8)	OCI Kubernetes Container Runtime daemon

Examples of commandline tools to interact with CRI-O (or other CRI compatible runtimes) are Crictl and Podman.

Configuration

File	Description
crio.conf(5)	CRI-O Configuration file
policy.json(5)	Signature Verification Policy File(s)
registries.conf(5)	Registries Configuration file
storage.conf(5)	Storage Configuration file

Security

The security process for reporting vulnerabilities is described in SECURITY.md.

OCI Hooks Support

You can configure CRI-O to inject OCI Hooks when creating containers.

CRI-O Usage Transfer

We provide useful information for operations and development transfer as it relates to infrastructure that utilizes CRI-O.

Communication

For async communication and long running discussions please use issues and pull requests on the GitHub repo. This will be the best place to discuss design and implementation.

For chat communication, we have a channel on the Kubernetes slack that everyone is welcome to join and chat about development.

Awesome CRI-O

We maintain a curated list of links related to CRI-O. Did you find something interesting on the web about the project? Awesome, feel free to open up a PR and add it to the list.

Getting started

Installing CRI-O

To install CRI-O, you can follow our installation guide. Alternatively, if you'd rather build CRI-O from source, checkout our setup guide. We also provide a way in building static binaries of CRI-O via nix as part of the cri-o/packaging repository. Those binaries are available for every successfully built commit on our Google Cloud Storage Bucket. This means that the latest commit can be installed via our convenience script:

> curl https://raw.githubusercontent.com/cri-o/packaging/main/get | bash

The script automatically verifies the uploaded sigstore signatures as well, if the local system has cosign available in its $PATH. The same applies to the SPDX based bill of materials (SBOM), which gets automatically verified if the bom tool is in $PATH.

Besides amd64, we also support the arm64, ppc64le and s390x bit architectures. This can be selected via the script, too:

curl https://raw.githubusercontent.com/cri-o/packaging/main/get | bash -s -- -a arm64

It is also possible to select a specific git SHA or tag by:

curl https://raw.githubusercontent.com/cri-o/packaging/main/get | bash -s -- -t v1.21.0

The above script resolves to the download URL of the static binary bundle tarball matching the format:

https://storage.googleapis.com/cri-o/artifacts/cri-o.$ARCH.$REV.tar.gz

Where $ARCH can be amd64,arm64,ppc64le or s390x and $REV can be any git SHA or tag. Please be aware that using the latest main SHA might cause a race, because the CI has not finished publishing the artifacts yet or failed.

We also provide a Software Bill of Materials (SBOM) in the SPDX format for each bundle. The SBOM is available at the same URL like the bundle itself, but suffixed with .spdx:

https://storage.googleapis.com/cri-o/artifacts/cri-o.$ARCH.$REV.tar.gz.spdx

Running Kubernetes with CRI-O

Before you begin, you'll need to start CRI-O

You can run a local version of Kubernetes with CRI-O using local-up-cluster.sh:

1. Clone the Kubernetes repository
2. From the Kubernetes project directory, run:

CGROUP_DRIVER=systemd \
CONTAINER_RUNTIME=remote \
CONTAINER_RUNTIME_ENDPOINT='unix:///var/run/crio/crio.sock' \
./hack/local-up-cluster.sh

For more guidance in running CRI-O, visit our tutorial page

The HTTP status API

CRI-O exposes per default the gRPC API to fulfill the Container Runtime Interface (CRI) of Kubernetes. Besides this, there exists an additional HTTP API to retrieve further runtime status information about CRI-O. Please be aware that this API is not considered to be stable and production use-cases should not rely on it.

On a running CRI-O instance, we can access the API via an HTTP transfer tool like curl:

$ sudo curl -v --unix-socket /var/run/crio/crio.sock http://localhost/info | jq
{
  "storage_driver": "btrfs",
  "storage_root": "/var/lib/containers/storage",
  "cgroup_driver": "systemd",
  "default_id_mappings": { ... }
}

The following API entry points are currently supported:

Path	Content-Type	Description
/info	application/json	General information about the runtime, like storage_driver and storage_root.
/containers/:id	application/json	Dedicated container information, like name, pid and image.
/config	application/toml	The complete TOML configuration (defaults to /etc/crio/crio.conf) used by CRI-O.
/pause/:id	application/json	Pause a running container.
/unpause/:id	application/json	Unpause a paused container.

The subcommand crio status can be used to access the API with a dedicated command line tool. It supports all API endpoints via the dedicated subcommands config, info and containers, for example:

$ sudo crio status info
cgroup driver: systemd
storage driver: btrfs
storage root: /var/lib/containers/storage
default GID mappings (format <container>:<host>:<size>):
  0:0:4294967295
default UID mappings (format <container>:<host>:<size>):
  0:0:4294967295

Metrics

Please refer to the CRI-O Metrics guide.

Tracing

Please refer to the CRI-O Tracing guide.

Container Runtime Interface special cases

Some aspects of the Container Runtime are worth some additional explanation. These details are summarized in a dedicated guide.

Debugging tips

Having an issue? There are some tips and tricks for debugging located in our debugging guide

Adopters

An incomplete list of adopters of CRI-O in production environments can be found here. If you're a user, please help us complete it by submitting a pull-request!

Weekly Meeting

A weekly meeting is held to discuss CRI-O development. It is open to everyone. The details to join the meeting are on the wiki.

Governance

For more information on how CRI-O is goverened, take a look at the governance file

License Scan

ocicni's Issues

Kubernetes-specific CNI_ARGS always added

The code constructing the CNI_ARGS value seems to be initialized with specific variables: K8S_POD_NAMESPACE, K8S_POD_NAME and K8S_POD_INFRA_CONTAINER_ID.

ocicni/pkg/ocicni/ocicni.go

Line 869 in 513ef78

{"K8S_POD_NAMESPACE", podNetwork.Namespace},

My understanding is that these are Kubernetes-specific, and not part of CNI specification.

At least one CNI-plugin (and probably others) checks for the presence of these variables in CNI_ARGS to decide whether being in K8S context or "generic" CNI.

For example code from Calico CNI Plugin: https://github.com/projectcalico/cni-plugin/blob/eb74819f8618478746606378fe3b64c76f54c909/internal/pkg/utils/utils.go#L435

My specific use-case is trying to use Calico as CNI-plugin via Podman which (to my understanding) uses this project (OCICNI) somewhere in the stack to call the CNI-plugin. That doesn't work for me because OCICNI adds the K8S_POD_NAMESPACE arg to CNI_ARGS so Calico behaves as it were inside K8S Cluster instead of as generic CNI.

So what I'm trying to understand is whether it is intended that the OCICNI project contains K8S specific things like the above K8S-specific arguments? Or is Podman not supposed to use this project as "generic" CNI-library?
The relationship between Podman, libpod, CRI-O and OCICNI is a bit confusing to me (and what parts are supposed to be K8S-specific).

TL;DR - Would it be possible to not have K8S_xxx args send to the CNI-plugin somehow?

[feature request] need support more NetworkPlugin interface

Now, NetworkPlugin interface has been updated as follows :

// Plugin is an interface to network plugins for the kubelet
type NetworkPlugin interface {
	// Init initializes the plugin.  This will be called exactly once
	// before any other methods are called.
	Init(host Host, hairpinMode kubeletconfig.HairpinMode, nonMasqueradeCIDR string, mtu int) error

	// Called on various events like:
	// NET_PLUGIN_EVENT_POD_CIDR_CHANGE
	Event(name string, details map[string]interface{})

	// Name returns the plugin's name. This will be used when searching
	// for a plugin by name, e.g.
	Name() string

	// Returns a set of NET_PLUGIN_CAPABILITY_*
	Capabilities() utilsets.Int

	// SetUpPod is the method called after the infra container of
	// the pod has been created but before the other containers of the
	// pod are launched.
	SetUpPod(namespace string, name string, podSandboxID kubecontainer.ContainerID, annotations map[string]string) error

	// TearDownPod is the method called before a pod's infra container will be deleted
	TearDownPod(namespace string, name string, podSandboxID kubecontainer.ContainerID) error

	// GetPodNetworkStatus is the method called to obtain the ipv4 or ipv6 addresses of the container
	GetPodNetworkStatus(namespace string, name string, podSandboxID kubecontainer.ContainerID) (*PodNetworkStatus, error)

	// Status returns error if the network plugin is in error state
	Status() error
}

However，what we have supported as follows :

// CNIPlugin is the interface that needs to be implemented by a plugin
type CNIPlugin interface {
	// Name returns the plugin's name. This will be used when searching
	// for a plugin by name, e.g.
	Name() string

	// SetUpPod is the method called after the sandbox container of
	// the pod has been created but before the other containers of the
	// pod are launched.
	SetUpPod(network PodNetwork) (types.Result, error)

	// TearDownPod is the method called before a pod's sandbox container will be deleted
	TearDownPod(network PodNetwork) error

	// Status is the method called to obtain the ipv4 or ipv6 addresses of the pod sandbox
	GetPodNetworkStatus(network PodNetwork) (string, error)

	// NetworkStatus returns error if the network plugin is in error state
	Status() error
}

We should update it to to keep in sync.

not enough arguments in ocicni.go

Hi,

When i test your project, it fails xith

command: go test -buildmode pie -compiler gc -ldflags "-X github.com/cri-o/ocicni/version=0.1.0 -extldflags '-Wl,-z,relro '"
testing: github.com/cri-o/ocicni
github.com/cri-o/ocicni/pkg/ocicni

github.com/cri-o/ocicni/pkg/ocicni [github.com/cri-o/ocicni/pkg/ocicni.test]

./ocicni.go:503:35: not enough arguments in call to cninet.AddNetworkList
have (*libcni.NetworkConfigList, *libcni.RuntimeConf)
want (context.Context, *libcni.NetworkConfigList, *libcni.RuntimeConf)
./ocicni.go:521:29: not enough arguments in call to cninet.DelNetworkList
have (*libcni.NetworkConfigList, *libcni.RuntimeConf)
want (context.Context, *libcni.NetworkConfigList, *libcni.RuntimeConf)
./ocicni_test.go:327:25: cannot use fake (type *fakeExec) as type invoke.Exec in argument to initCNI:
*fakeExec does not implement invoke.Exec (wrong type for ExecPlugin method)
have ExecPlugin(string, []byte, []string) ([]byte, error)
want ExecPlugin(context.Context, string, []byte, []string) ([]byte, error)
./ocicni_test.go:400:25: cannot use fake (type *fakeExec) as type invoke.Exec in argument to initCNI:
*fakeExec does not implement invoke.Exec (wrong type for ExecPlugin method)
have ExecPlugin(string, []byte, []string) ([]byte, error)
want ExecPlugin(context.Context, string, []byte, []string) ([]byte, error)
FAIL github.com/cri-o/ocicni/pkg/ocicni [build failed]

Thanks

ocicni should support podCIDR

CNI support some capabilities including ip ranges for dynamically configuring the IP range(s) for address allocation. Runtimes that manage IP pools, but not individual IP addresses, can pass these to plugins.
CRI also support UpdateRuntimeConfig function to let kubernetes notify itself when node have valid PodCIDR.
So I think ocicni should support to receive and store podCIDR, and pass it to CNI plugins by CNI-runtime-capabilities.

Tag a release

I think we could tag a release (maybe 0.1?).
@dcbw @rajatchopra wdyt?

tag a new release

I am currently going through Podman's dependencies (updating the vendor.conf) trying to pin them to releases when possible and available. Some distribution, Debian in particular, are very conservative regarding go packaging as many libraries don't have any release (or no semantic versioning).

Can we tag a new release?

ocicni: Always start monitorNetDir thread?

It seems the CNI config directory is only monitored via inotify if we do not find a default config? This is fine for the initial setup, but once we have a persistent config during restart it no longer responds to changes.

Is there any harm in always starting this?

ocicni/pkg/ocicni/ocicni.go

Lines 170 to 175 in 17d9dd2

	if err := plugin.syncNetworkConfig(); err != nil {
	// We do not have a valid default network, so start the
	// monitoring thread. Network setup/teardown requests
	// will fail until we have a valid default network.
	go plugin.monitorNetDir()
	}

First start:

# started kubelet
Jan 29 09:44:05 kubelet20 crio: time="2018-01-29 09:44:05.933403577-06:00" level=error msg="error updating cni config: Missing CNI default network"
Jan 29 09:44:05 kubelet20 crio: time="2018-01-29 09:44:05.933408554-06:00" level=error msg="CNI setting failed, continue monitoring: Missing CNI default network"
Jan 29 09:44:05 kubelet20 crio: time="2018-01-29 09:44:05.933694464-06:00" level=info msg="CNI network kubernetes (type=bridge) is used from /etc/cni/net.d/10-bridge.conf"
Jan 29 09:44:05 kubelet20 crio: time="2018-01-29 09:44:05.933709239-06:00" level=info msg="CNI asynchronous setting succeeded"
Jan 29 09:44:15 kubelet20 crio: time="2018-01-29 09:44:15.430039091-06:00" level=info msg="About to add CNI network cni-loopback (type=loopback)"
Jan 29 09:44:15 kubelet20 crio: time="2018-01-29 09:44:15.431892205-06:00" level=info msg="About to add CNI network kubernetes (type=bridge)"
Jan 29 09:44:18 kubelet20 crio: time="2018-01-29 09:44:18.983438288-06:00" level=info msg="CNI network kubernetes (type=bridge) is used from /etc/cni/net.d/10-bridge.conf"
Jan 29 09:44:18 kubelet20 crio: time="2018-01-29 09:44:18.983452983-06:00" level=info msg="CNI asynchronous setting succeeded"

Subsequent crio (re)starts:

Jan 29 09:45:02 kubelet20 crio: time="2018-01-29 09:45:02.849241747-06:00" level=info msg="CNI network kubernetes (type=bridge) is used from /etc/cni/net.d/10-bridge.conf"
Jan 29 09:45:02 kubelet20 crio: time="2018-01-29 09:45:02.849352934-06:00" level=info msg="CNI network kubernetes (type=bridge) is used from /etc/cni/net.d/10-bridge.conf"
# no monitoring...

The default network is immutable

According to https://github.com/cri-o/ocicni/blob/master/pkg/ocicni/ocicni.go#L338, it seems like the defaultNetName, once initialized, is immutable. That doesn't seem right.

It is especially problematic if that network is deleted, which is the case I've personally seen.

Consider moving this project into CRI-O

I'm not sure if this makes sense but do we want to move this project into CRI-O? We would benefit by a lots of other things, like outdated dependency checks, go modules, linters, unit tests and versioning. Is the project used somewhere else which we would break up when moving it into CRI-O? WDYT?

Validation is needed before using a CNI config

ref to kubernetes/kubernetes#80482

a fsnotify-related risk

Ocicni now is using fsnotify to load cni config files real-time, but fsnotify has a risk, if we remove cni config path /etc/cni/net.d and recreate it, any changes of this directory will not be detected by fsnotify any more. In other words, ocicni will not work properly after cni config path was removed.

cri-o + kubelet can't start containers due to veth mac mismatch

Description

When running the kubelet + cri-o in "standalone" mode, aka no control plane, I see what looks like a race condition on Pod startup, with it frequently complaining that the veth mac address doesn't match the expected value.

remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx-combined-127.0.0.1_default_19b0558958fc9f867987740a6eb88042_2(f898a0f1feaa8e63f45eaec61a38d4f8f9cf2298cde3879c3726d2b8b0696cfe): Interface vetha38a080a Mac doesn't match: ee:7b:4d:57:3a:d9 not found

The kubelet will back off several times as it reattempts to start the Pod, and typically will eventually succeed. Other times, I will remove the Pod manifest from /etc/kubernetes/manifests and recreate it.

Steps to reproduce the issue:
This is verbose, but exactly how I set up the kubelet:

1. On Fedora Core 31

CRI-O is available as a DNF module stream. On FC31 be sure to install the latest version to circumvent this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1754170

2. dnf -y module enable cri-o:1.16
dnf -y install crio
3. Delete any existing CNI rm /etc/cni/net.d/*.conf
4. Add /etc/cni/net.d/100-crio-bridge.conflist

{
  "cniVersion": "0.4.0",
  "name": "bridge-firewalld",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isDefaultGateway": true,
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "subnet": "10.88.0.0/16",
        "routes": [
          {
            "dst": "0.0.0.0/0"
          }
        ]
      }
    },
    { 
      "type": "portmap",
      "capabilities": {
	"portMappings": true
     }
    },
    {
      "type": "firewall"
    }
  ]
}

5. dnf -y install kubernetes-node
6. Disable docker which is brought in automatically
   systemctl mask docker
cp /usr/lib/systemd/system/kubelet.service /etc/systemd/system/kubelet.service

Replace /etc/systemd/system/kubelet.service with crio:

 [Unit]
 Description=Kubernetes Kubelet Server
 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
-After=docker.service
-Requires=docker.service
+After=crio.service
+Requires=crio.service
 
 [Service]
 WorkingDirectory=/var/lib/kubelet

7. systemctl daemon-reload
8. Remove the now obsolete flag --allow-privileged from /etc/kubernetes/config that would prevent the service from starting:

 KUBE_LOG_LEVEL="--v=0"
 
 # Should this cluster be allowed to run privileged docker containers
-KUBE_ALLOW_PRIV="--allow-privileged=true"
+KUBE_ALLOW_PRIV=""
 
 # How the controller-manager, scheduler, and proxy find the apiserver
 KUBE_MASTER="--master=http://127.0.0.1:8080"

9. Apply the below modifications to /etc/kubernetes/kubelet:

 KUBELET_HOSTNAME="--hostname-override=127.0.0.1"
 
 # Add your own!
-KUBELET_ARGS="--cgroup-driver=systemd --fail-swap-on=false"
+KUBELET_ARGS="--cgroup-driver=systemd --fail-swap-on=false --pod-manifest-path=/etc/kubernetes/manifests --container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=10m"

10. mkdir /etc/kubernetes/manifests
11. systemctl enable kubelet --now
12. Drop sample Pod in /etc/kubernetes/manifests

apiVersion: v1
kind: Pod
metadata:
  name: echoserver
spec:
  containers:
    - name: nginx-combined
      image: gcr.io/google-containers/echoserver:1.10
      ports:
        - name: web
          containerPort: 8080
          hostPort: 9091
          protocol: TCP
      resources:
        limits:
          cpu: "100m"
          memory: "50Mi"

Describe the results you received:

Mar 19 16:31:17 localhost.localdomain kubelet[1166]: E0319 16:31:17.672240    1166 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx-combined-127.0.0.1_default_19b0558958fc9f867987740a6eb88042_2(f898a0f1feaa8e63f45eaec61a38d4f8f9cf2298cde3879c3726d2b8b0696cfe): Interface vetha38a080a Mac doesn't match: ee:7b:4d:57:3a:d9 not found
Mar 19 16:31:17 localhost.localdomain kubelet[1166]: E0319 16:31:17.672295    1166 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "nginx-combined-127.0.0.1_default(19b0558958fc9f867987740a6eb88042)" failed: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx-combined-127.0.0.1_default_19b0558958fc9f867987740a6eb88042_2(f898a0f1feaa8e63f45eaec61a38d4f8f9cf2298cde3879c3726d2b8b0696cfe): Interface vetha38a080a Mac doesn't match: ee:7b:4d:57:3a:d9 not found
Mar 19 16:31:17 localhost.localdomain kubelet[1166]: E0319 16:31:17.672322    1166 kuberuntime_manager.go:692] createPodSandbox for pod "nginx-combined-127.0.0.1_default(19b0558958fc9f867987740a6eb88042)" failed: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx-combined-127.0.0.1_default_19b0558958fc9f867987740a6eb88042_2(f898a0f1feaa8e63f45eaec61a38d4f8f9cf2298cde3879c3726d2b8b0696cfe): Interface vetha38a080a Mac doesn't match: ee:7b:4d:57:3a:d9 not found
Mar 19 16:31:17 localhost.localdomain kubelet[1166]: E0319 16:31:17.672392    1166 pod_workers.go:190] Error syncing pod 19b0558958fc9f867987740a6eb88042 ("nginx-combined-127.0.0.1_default(19b0558958fc9f867987740a6eb88042)"), skipping: failed to "CreatePodSandbox" for "nginx-combined-127.0.0.1_default(19b0558958fc9f867987740a6eb88042)" with CreatePodSandboxError: "CreatePodSandbox for pod \"nginx-combined-127.0.0.1_default(19b0558958fc9f867987740a6eb88042)\" failed: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx-combined-127.0.0.1_default_19b0558958fc9f867987740a6eb88042_2(f898a0f1feaa8e63f45eaec61a38d4f8f9cf2298cde3879c3726d2b8b0696cfe): Interface vetha38a080a Mac doesn't match: ee:7b:4d:57:3a:d9 not found"

Describe the results you expected:
Pod starts the first time the kubelet attempts it.

Additional information you deem important (e.g. issue happens only occasionally):
This seems to be a race condition with the veth being created because it will eventually work most of the time. I don't know how to instrument it further.

Output of crio --version:

crio -v
crio version 1.16.1
commit: "53843bd41d4bb3385c3e035bbf082e629b46f3bd-dirty"

Additional environment details (AWS, VirtualBox, physical, etc.):
Fedora Core 31 on x86 workstation, no virtualization

$ /usr/bin/kubelet --version
Kubernetes v1.15.8-beta.0

In memory configuration and plugin

I'm not sure if this is a great idea so I want to put it under discussion: What about providing a default bridged network fallback if no further configuration was found? We would have to built-in the configuration as well as the binary code of the plugin here. I imagine that this could be a strong default if users want to try out podman without having all the CNI plugins and configs at hand.

New release compatible with CNI v1.0

Is there a new release planned that would include #90 for compatibility with CNI 1.0?

CRI-O's not using this library!

CRI-O's still using pkg/ocicni in CRI-O's repo. We need to port changes made there here and switch CRI-O to use this.

ocicni: Always start plugin.monitorNetDir thread?

It seems the CNI config directory is only monitored via inotify if we do not find a default config? This is fine for the initial setup, but once we have a persistent config during restart it no longer responds to changes.

Is there any harm in always starting this?

ocicni/pkg/ocicni/ocicni.go

Lines 170 to 175 in 17d9dd2

	if err := plugin.syncNetworkConfig(); err != nil {
	// We do not have a valid default network, so start the
	// monitoring thread. Network setup/teardown requests
	// will fail until we have a valid default network.
	go plugin.monitorNetDir()
	}

First start:

# started kubelet
Jan 29 09:44:05 kubelet20 crio: time="2018-01-29 09:44:05.933403577-06:00" level=error msg="error updating cni config: Missing CNI default network"
Jan 29 09:44:05 kubelet20 crio: time="2018-01-29 09:44:05.933408554-06:00" level=error msg="CNI setting failed, continue monitoring: Missing CNI default network"
Jan 29 09:44:05 kubelet20 crio: time="2018-01-29 09:44:05.933694464-06:00" level=info msg="CNI network kubernetes (type=bridge) is used from /etc/cni/net.d/10-bridge.conf"
Jan 29 09:44:05 kubelet20 crio: time="2018-01-29 09:44:05.933709239-06:00" level=info msg="CNI asynchronous setting succeeded"
Jan 29 09:44:15 kubelet20 crio: time="2018-01-29 09:44:15.430039091-06:00" level=info msg="About to add CNI network cni-loopback (type=loopback)"
Jan 29 09:44:15 kubelet20 crio: time="2018-01-29 09:44:15.431892205-06:00" level=info msg="About to add CNI network kubernetes (type=bridge)"
Jan 29 09:44:18 kubelet20 crio: time="2018-01-29 09:44:18.983438288-06:00" level=info msg="CNI network kubernetes (type=bridge) is used from /etc/cni/net.d/10-bridge.conf"
Jan 29 09:44:18 kubelet20 crio: time="2018-01-29 09:44:18.983452983-06:00" level=info msg="CNI asynchronous setting succeeded"

Subsequent crio (re)starts:

Jan 29 09:45:02 kubelet20 crio: time="2018-01-29 09:45:02.849241747-06:00" level=info msg="CNI network kubernetes (type=bridge) is used from /etc/cni/net.d/10-bridge.conf"
Jan 29 09:45:02 kubelet20 crio: time="2018-01-29 09:45:02.849352934-06:00" level=info msg="CNI network kubernetes (type=bridge) is used from /etc/cni/net.d/10-bridge.conf"
# no monitoring...

Use file-based locking to prevent concurrent DEL calls

For example, podman does not use a long-running process for handling container sandbox operations. This means that two 'podman stop' calls for the same container can run concurrently and call a CNI DEL concurrently, which is a violation of the CNI specification.

To handle this, ocicni should implement file-based locking with github.com/alexflint/go-filemutex (see CNI plugins' plugins/ipam/host-local/backend/disk/ for an examle) to ensure that multiple processes cannot call DEL on the same containerID concurrently.

ocicni is ignoring environment CNI_ARGS and synthesizing a new one

TL;DR Is there a way to invoke ocicni to "pass-through" CNI_ARGS from the environment instead of synthesizing k8s-type CNI_ARGS?

https://github.com/containers/libpod is a consumer of this package. However, the package is ignoring CNI_ARGS from the environment and synthesizing a k8s friendly(?) version. This prevents podman from influencing CNI plugins like host-local or static.

Presumably because ocicni was only expecting to be used in k8s environment - however standalone users like podman need to be able to pass-through CNI_ARGS to the plugin.

See containers/podman#4398

CNI_ARGS set by user when calling podman:

CNI_ARGS="IP:192.168.125.200;GATEWAY=192.168.125.1 podman ...

CNI_ARGS that the plugin actually sees:

IgnoreUnknown=1;K8S_POD_NAMESPACE=abcd_3;K8S_POD_NAME=abcd_3;K8S_POD_INFRA_CONTAINER_ID=4c7f1903383a9552973ba2deb5ceb2e8b65ab1e06309c8af948b217724718bc3

Additional checks for cni 0.4.0 failing (atleast with the bridge plugin)

Here is the config I am trying to use in CRI-O:

{
  "cniVersion": "0.4.0",
  "name": "bridge-firewall",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "routes": [
            { "dst": "0.0.0.0/0" }
        ],
        "ranges": [
            [
               {
                   "subnet": "10.85.0.0/16",
                   "gateway": "10.85.0.1"
               }
            ]
        ]
      }
    },
    {
      "type": "firewall"
    }
  ]
}

However, I see that the additional checks in ocicni for version 0.4.0 and higher start failing:

Jun 05 13:57:28 localhost.localdomain crio[25444]: time="2020-06-05 13:57:28.849936252-07:00" level=error msg="Error checking network: Interface veth6e529a55 Mac doesn't match: 66:43:79:e6:a9:17 not found"
Jun 05 13:57:28 localhost.localdomain crio[25444]: time="2020-06-05 13:57:28.849949659-07:00" level=error msg="Error while checking pod to CNI network \"bridge-firewall\": Interface veth6e529a55 Mac doesn't match: 66:43:79:e6:a9:17 not found"
Jun 05 13:57:42 localhost.localdomain crio[25444]: time="2020-06-05 13:57:42.845894713-07:00" level=error msg="Error checking network: Interface vethc824d8d9 Mac doesn't match: e6:0f:c8:73:c9:ad not found"
Jun 05 13:57:42 localhost.localdomain crio[25444]: time="2020-06-05 13:57:42.845901992-07:00" level=error msg="Error while checking pod to CNI network \"bridge-firewall\": Interface vethc824d8d9 Mac doesn't match: e6:0f:c8:73:c9:ad not found"

If I compile the check out, then pods come up fine for me:

diff --git a/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go b/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go
index 85dcdcfe2..82bf12bb6 100644
--- a/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go
+++ b/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go
@@ -778,24 +778,8 @@ func (network *cniNetwork) addToNetwork(ctx context.Context, rt *libcni.RuntimeC
 func (network *cniNetwork) checkNetwork(ctx context.Context, rt *libcni.RuntimeConf, cni *libcni.CNIConfig, nsManager *nsManager, netns string) (cnitypes.Result, error) {
        logrus.Infof("About to check CNI network %s (type=%v)", network.name, network.config.Plugins[0].Network.Type)
-       gtet, err := cniversion.GreaterThanOrEqualTo(network.config.CNIVersion, "0.4.0")
-       if err != nil {
-               return nil, err
-       }
-
        var result cnitypes.Result
-
-       // When CNIVersion supports Check, use it.  Otherwise fall back on what was done initially.
-       if gtet {
-               err = cni.CheckNetworkList(ctx, network.config, rt)
-               logrus.Infof("Checking CNI network %s (config version=%v)", network.name, network.config.CNIVersion)
-               if err != nil {
-                       logrus.Errorf("Error checking network: %v", err)
-                       return nil, err
-               }
-       }
-
-       result, err = cni.GetNetworkListCachedResult(network.config, rt)
+       result, err := cni.GetNetworkListCachedResult(network.config, rt)

Is this something we have to fix in ocicni or upstream in the bridge plugin?

Tearing down networks should allow an empty netns

If the runtime knows that the network namespace doesn't exist (e.g. because the node has rebooted), it should pass an empty or otherwise invalid netns.

ocicni should accept an empty netns on delete.

This was noticed in cri-o/cri-o#2849