k8snetworkplumbingwg / ovs-cni

Open vSwitch CNI plugin

Open vSwitch CNI plugin

This plugin allows user to define Kubernetes networks on top of Open vSwitch bridges available on nodes. Note that ovs-cni does not configure bridges, it's up to a user to create them and connect them to L2, L3 or an overlay network. This project also delivers OVS marker, which exposes available bridges as Node resources, that can be used to schedule pods on the right node via intel/network-resources-injector. Finally please note that Open vSwitch must be installed and running on the host.

In order to use this plugin, Multus must be installed on all hosts and NetworkAttachmentDefinition CRD created.

Overview

First create network attachment definition. This object specifies to which Open vSwitch bridge should the pod be attached and what VLAN ID should be set on the port. For more information, check plugin documentation.

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ovs-conf
  annotations:
    k8s.v1.cni.cncf.io/resourceName: ovs-cni.network.kubevirt.io/br1
spec:
  config: '{
      "cniVersion": "0.4.0",
      "type": "ovs",
      "bridge": "br1",
      "vlan": 100
    }'
EOF

Once the network definition is created and desired Open vSwitch bridges are available on nodes, a pod requesting the network can be created.

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: ovs-conf
spec:
  containers:
  - name: samplepod
    command: ["/bin/sh", "-c", "sleep 99999"]
    image: alpine
    resources:  # this may be omitted if intel/network-resources-injector is present on the cluster
      limits:
        ovs-cni.network.kubevirt.io/br1: 1
EOF

Such pod should contain default eth0 interface connected to default Kubernetes network and also net1 connected to the bridge.

$ kubectl exec samplepod ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether 0a:58:0a:f4:00:07 brd ff:ff:ff:ff:ff:ff
5: net1@if12: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether e6:f4:2e:b4:4b:6e brd ff:ff:ff:ff:ff:ff

Deployment and Usage

You can choose to deploy this plugin on local virtualized cluster or on your arbitrary cluster. After that you can follow demo that will guide you through preparation of Open vSwitch bridges, defining networks on Kubernetes and attaching pods to them.

Components

• CNI Plugin - Documentation of standalone Open vSwitch CNI plugin.
• Port Mirroring - Documentation of an OVS CNI extension, allowing for port mirroring.
• Hardware Offload - Documentation of hardware offload functionality, using SR-IOV.
• Marker - Documentation of daemon set exposing bridges as node resources.

Development

Development guide is a go-to reference point for development helper commands, building, testing, container images and local cluster.

ovs-cni's Issues

Failed to connect to ovsdb error: Invalid socket file

On creating pod with 2 ovs interface it gives error "ovs": error in getting result from AddNetwork: failed to connect to ovsdb error: Invalid socket file, failed to clean up sandbox container
How can I debug more to know the actual cause of issue.

The yaml definitions file works on other setup. Here are the yaml files used:
ovs-net.yml

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ovs-net-1
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "vsperf-br0"
    }'

pod.yml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-ovs
  annotations:
    k8s.v1.cni.cncf.io/networks: ovs-net-1, ovs-net-1
spec:
  containers:
  - name: nginx-ovs
    image: nginx:1.7.9
    ports:
      - containerPort: 80

kubectl describe pod nginx-ovs

Name:         nginx-ovs
Namespace:    default
Priority:     0
Node:         worker/10.10.120.21
Start Time:   Sat, 16 May 2020 13:48:19 -0700
Labels:       <none>
Annotations:  k8s.v1.cni.cncf.io/networks: ovs-net-1, ovs-net-1
              k8s.v1.cni.cncf.io/networks-status: 
Status:       Pending
IP:           
IPs:          <none>
Containers:
  nginx-ovs:
    Container ID:   
    Image:          nginx:1.7.9
    Image ID:       
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tw2c7 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-tw2c7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-tw2c7
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age              From               Message
  ----     ------                  ----             ----               -------
  Normal   Scheduled               17s              default-scheduler  Successfully assigned default/nginx-ovs to worker
  Warning  FailedCreatePodSandBox  9s               kubelet, worker    Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "cf902dc941fa85325b871bcb93caa0298a9a8c1281b707150249a2cae984db68" network for pod "nginx-ovs": networkPlugin cni failed to set up pod "nginx-ovs_default" network: Multus: [default/nginx-ovs]: error adding container to network "ovs-net-1": delegateAdd: error invoking DelegateAdd - "ovs": error in getting result from AddNetwork: failed to connect to ovsdb error: Invalid socket file, failed to clean up sandbox container "cf902dc941fa85325b871bcb93caa0298a9a8c1281b707150249a2cae984db68" network for pod "nginx-ovs": networkPlugin cni failed to teardown pod "nginx-ovs_default" network: delegateDel: error invoking DelegateDel - "ovs": error in getting result from DelNetwork: failed to connect to ovsdb error: Invalid socket file / delegateDel: error invoking DelegateDel - "ovs": error in getting result from DelNetwork: failed to connect to ovsdb error: Invalid socket file]
  Normal   SandboxChanged          8s (x2 over 9s)  kubelet, worker    Pod sandbox changed, it will be killed and re-created.

The name of the bridge is vsperf-br0
The setup has one worker node and one master node.
The worker node has ovs installed. And ovs-daemonset.yml is deployed in the cluster.

Research whether we should schedule OVS CNI DPs on master node

"BTW just a very small nit can you please remove this from all our yaml ?

• key: node-role.kubernetes.io/master

So the cni will run on all the nodes just line any other cni."

"So the problem appears on kubeadm version 1.12+
They added a new taint on the master node node.kubernetes.io/not-ready:NoSchedule
This taint should be removed after the main network cni is running but I don't want to have a uncover situation that a pod is schedule on a node because I as the toleration for this taint but our cni will not be installed there and the pod will fail"

#51 (comment)

mac address conflicts at high scale

We have a deployment of 100 worker nodes with 24 vlans and 1000 ovs-cni interfaces on each of these vlans.
On few of these vlans, the ovs-cni ends up allocating same mac address to different pods.
Is there any way to avoid duplicate mac address allocation with ovs-cni?

Design: Selective VLANs on the Root Device and Ease of Network Orchestration

Problem:

The OVS CNI in its present form supports two types of OVS network attachment definitions (NAD).

1. Single VLAN untagged interface

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata: 
  name: ovs_br-bond_1024
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br-bond",
      "vlan": 1024
}'

2. Transparent trunk

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata: 
  name: ovs_br-int_transparent_trunk
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br-bond",
}'

Note: The absence of the “vlan” field implies transparent access to all VLANs present on the specified bridge and underlying root device.

In case of Transparent trunk scenario, the root device is entirely given to application(s) running in the pod container(s) and can not used by infrastructure or primary networks. Hence there should be a mechanism to use a set of VLANs by secondary networks and remaining VLANs can be used by infrastructure.

There should also ease of network orchestration which allows pod can request for specific VLAN(s) in the allowed VLAN range. This would reduce number of NAD objects tremendously.

Proposal:

For selective trunk network attachments, enhance the OVS CNI with the following option:

3. Selective trunk

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata: 
  name: ovs_br-bond_trunk_1-100_200_300
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br-bond",
      “trunk”: "1-100,200,300"
}'

The new optional “trunk” attribute enumerates the list of allowed VLAN IDs on the trunk interface. Ranges of VLAN IDs can be specified.

During the ADD command the OVS CNI creates a veth pair, moves one endpoint into the pod and attaches the host endpoint to the specified OVS bridge as access port for the specified VLAN (1), as transparent trunk port (2) or as trunk port with list of allowed VLAN IDs (3).

The above three OVS NAD formats have a well-defined VLAN visibility. If there are many VLANs, the Kubernetes operator would have to create a correspondingly large number of NADs. Especially potential space of the selective trunk NADs is huge.

It would significantly simplify network orchestration if the operator provides generic NADs for a bridge that only specify the mode of the OVS port (untagged access or trunk) and a list of allowed VLAN IDs and leave it to the pod specification to select which of the allowed VLAN IDs to use.
To this end we suggest two more optional fields in the OVS NAD:

mode: 	         access | trunk
allowed-vlans:	 List of allowed VLAN IDs

The allowed_vlans parameter ensures that pods cannot access VLANs on the bridge that are used by shall be isolated (e.g. infrastructure or primary networks).

With this there will be two additional OVS NAD variants:

4. Pod-specified untagged single VLAN interface

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata: 
  name: ovs_br-bond_access
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br-bond",
      "mode": "access",
      "allowed-vlans": "1-100,200,300"
}'

5. Pod-specified selective trunk interface

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata: 
  name: ovs_br-bond_trunk
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br-bond",
      "mode": "trunk",
      "allowed-vlans": "1-100,200,300"
}'

The pod specification will then encode the desired VLAN ID(s) as cni-args in its network annotation, e.g.:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: |
      [
        {
          "name":" ovs_br-bond_access",
           "cni-args": {
              "wanted-vlans": "300"
          }
        }
      ]

It will configure the OVS ports for the wanted VLAN ID(s), rejecting or dropping VLAN IDs that are not in the allowed_vlans list of the NAD.

Failure while running 'make' command

Hello,

A failure have occured while I was running 'make' command in the main branch. Also same failures happen in tagged versions. (e.g v0.24.0)

Rerproduce:

1. clone the ovs-cni repo
2. run 'make'

Error:

hack/install-go.sh /root/apps/ovs-cni/build/_output/bin/
+ destination=/root/apps/ovs-cni/build/_output/bin/
+ version=1.16.8
+ tarball=go1.16.8.linux-amd64.tar.gz
+ url=https://dl.google.com/go/
+ mkdir -p /root/apps/ovs-cni/build/_output/bin/
+ curl -L https://dl.google.com/go//go1.16.8.linux-amd64.tar.gz -o /root/apps/ovs-cni/build/_output/bin//go1.16.8.linux-amd64.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 66 100 66 0 0 478 0 --:--:-- --:--:-- --:--:-- 474
100 123M 100 123M 0 0 33.6M 0 0:00:03 0:00:03 --:--:-- 38.8M
+ tar -xf /root/apps/ovs-cni/build/_output/bin//go1.16.8.linux-amd64.tar.gz -C /root/apps/ovs-cni/build/_output/bin/
setting GOPATH...
building golint...
go get: upgraded golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f => v0.0.0-20210508222113-6edffad5e616
go get: upgraded golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d => v0.0.0-20211015210444-4f30a5c0130f
go get: upgraded golang.org/x/tools v0.0.0-20210106214847-113979e3529a => v0.1.8
# golint installation modifies go.mod and it causes golint failure so run mod tidy here
/root/apps/ovs-cni/build/_output/bin//go//bin//go mod tidy
running golint...
go: inconsistent vendoring in /root/apps/ovs-cni/.gopath/src/github.com/k8snetworkplumbingwg/ovs-cni:
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod

To ignore the vendor directory, use -mod=readonly or -mod=mod.
To sync the vendor directory, run:
go mod vendor
/root/apps/ovs-cni/build/_output/bin//go//bin//go fmt ./pkg/... ./cmd/...
go: inconsistent vendoring in /root/apps/ovs-cni:
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod

To ignore the vendor directory, use -mod=readonly or -mod=mod.
To sync the vendor directory, run:
go mod vendor
Makefile:53: recipe for target 'format' failed
make: *** [format] Error 1

Rename all imports so they use the new organization

After migration from kubevirt GitHub org, all the imports are still pointing to the old org.

Drop all ovs-vsctl usage from plugin

Although most of the code uses socket connection to OVS DB, there is still some usage of ovs-vsctl. We should get rid of it, so we can drop the openvswitch RPM dependency.

Rename resource to reflect the new org

Resource exposed by OVS CNI marker are still ovs-cni.network.kubevirt.io, this should be changed (while maintaining backwards compatibility) to reflect the new organization.

IP addresses duplicate on two pods running on different nodes

Hi,
Need help, IP address found duplicate on two pods running on different nodes as below:

root@tgl02:/home/yockgenm/ovs-cni# kubectl exec -it test01-deployment-866cf959f5-n7nvq -- ip a
......
7: eth0@if56: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1480 qdisc noqueue state UP
link/ether 1a:ce:62:38:5a:f1 brd ff:ff:ff:ff:ff:ff
inet 192.168.231.196/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::18ce:62ff:fe38:5af1/64 scope link
valid_lft forever preferred_lft forever
8: net1@if57: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 0a:58:c0:a8:01:ca brd ff:ff:ff:ff:ff:ff
inet 192.168.1.202/24 brd 192.168.1.255 scope global net1 <-ERR: IP address duplicate
valid_lft forever preferred_lft forever
inet6 fe80::5c13:31ff:fef7:c7d4/64 scope link
valid_lft forever preferred_lft forever

root@tgl02:/home/yockgenm/ovs-cni# kubectl exec -it test01-deployment-866cf959f5-xcsb7 -- ip a
............
7: eth0@if220: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1480 qdisc noqueue state UP
link/ether a6:a8:2c:e1:01:dc brd ff:ff:ff:ff:ff:ff
inet 192.168.156.74/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a4a8:2cff:fee1:1dc/64 scope link
valid_lft forever preferred_lft forever
8: net1@if221: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 0a:58:c0:a8:01:ca brd ff:ff:ff:ff:ff:ff
inet 192.168.1.202/24 brd 192.168.1.255 scope global net1 <-ERR: IP address duplicate
valid_lft forever preferred_lft forever
inet6 fe80::9477:f0ff:feb9:be01/64 scope link
valid_lft forever preferred_lft forever

pods info

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test01-deployment-866cf959f5-n7nvq 1/1 Running 0 37m 192.168.231.196 tgl02
test01-deployment-866cf959f5-qhzvt 1/1 Running 0 37m 192.168.156.73 kbl01
test01-deployment-866cf959f5-xcsb7 1/1 Running 0 37m 192.168.156.74 kbl01

network attachment definition

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ovs-conf
  annotations:
    k8s.v1.cni.cncf.io/resourceName: ovs-cni.network.kubevirt.io/br1
spec:
  config: '{
      "cniVersion": "0.4.0",
      "type": "ovs",
      "bridge": "ovs-br1",
      "vlan": 100,
	  "ipam": {
        "type": "host-local",
        "subnet": "192.168.1.0/24",
        "rangeStart": "192.168.1.201",
        "rangeEnd": "192.168.1.250",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "192.168.1.1"
      }
    }'
EOF

pod deployment

cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test01-deployment
  labels:
    app: yockgen
spec:
  replicas: 3
  selector:
    matchLabels:
      app: test01pod
  template:
    metadata:      
      annotations:
        k8s.v1.cni.cncf.io/networks: ovs-conf
      labels:
        app: test01pod
    spec:
      containers:
      - name: test01pod
        command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
        image: alpine
        securityContext:
          capabilities:
            add: ["NET_ADMIN", "SYS_TIME","NET_RAW"]
EOF

OVS-CNI not working with sbr chained plugin due to interface index mismatch

Hi,

When using OVS-CNI and chaining the sbr plugin in a NetworkAttachmentDefinition, sbr fails to create a default route in the pod.

Steps to replicate:

1. Using the local cluster deployment with 1 master node and 2 worker nodes. Manually install sbr plugin in /opt/cni/bin/ on every node.

2. OVS setup on every node:

ovs-vsctl add-br br-data 
ovs-vsctl add-port br-data eth1

3. I used a per-worker configuration for the net-attach-def in order to be able to allocate IPs with host-local IPAM and avoid collisions. Each node has a /etc/cni/multus/net.d/signet-100.conflist file using a different allocation range:

[root@node01 ~]# mkdir -p /etc/cni/multus/net.d
[root@node01 ~]# cat << EOF > /etc/cni/multus/net.d/signet-100.conflist
{
  "cniVersion": "0.3.1",
  "name": "signet-100",
  "plugins": [
    {
      "type": "ovs",
      "bridge": "br-data",
      "vlan": 100,
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.33.0/24",
        "rangeStart": "192.168.33.150",
        "rangeEnd": "192.168.33.154",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ]
      }
    },
    {
      "type": "sbr"
    }
  ]
}
EOF

[root@node03 ~]# mkdir -p /etc/cni/multus/net.d
[root@node03 ~]# cat << EOF > /etc/cni/multus/net.d/signet-100.conflist
{
  "cniVersion": "0.3.1",
  "name": "signet-100",
  "plugins": [
    {
      "type": "ovs",
      "bridge": "br-data",
      "vlan": 100,
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.33.0/24",
        "rangeStart": "192.168.33.170",
        "rangeEnd": "192.168.33.174",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ]
      }
    },
    {
      "type": "sbr"
    }
  ]
}
EOF

4. Added the net-attach-def:

$ cat <<EOF | ./cluster/kubectl.sh apply -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: signet-100
  annotations:
    k8s.v1.cni.cncf.io/resourceName: ovs-cni.network.kubevirt.io/br-data
EOF

5. Create a pod:

$ cat <<EOF | ./cluster/kubectl.sh apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pod1-100
  annotations:
    k8s.v1.cni.cncf.io/networks: signet-100
spec:
  containers:
  - name: samplepod
    command: ["sh", "-c", "sleep 99999"]
    image: alpine
    securityContext:
      privileged: false
EOF

6. Inspect pod networking and routing:

$ ./cluster/kubectl.sh exec -it pod1-100 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1440 qdisc noqueue state UP 
    link/ether 82:1c:2e:9a:46:7a brd ff:ff:ff:ff:ff:ff
    inet 10.244.186.194/32 scope global eth0
       valid_lft forever preferred_lft forever
6: net1@if12: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:00:00:5e:23:80 brd ff:ff:ff:ff:ff:ff
    inet 192.168.33.170/24 brd 192.168.33.255 scope global net1
       valid_lft forever preferred_lft forever

$ ./cluster/kubectl.sh exec -it pod1-100 -- ip route show table 100
192.168.33.0/24 dev net1 scope link  src 192.168.33.170 

There is no default route created and logs show a mismatch between the interface index and the interface structure returned by OVS-CNI:

[root@node03 ~]# journalctl -xeu kubelet 
Oct 30 07:31:40 node03 kubelet[1688]: 2020/10/30 07:31:40 Incorrect interface for IP address 192.168.33.170
Oct 30 07:31:40 node03 kubelet[1688]: 2020/10/30 07:31:40 Network namespace to use and lock: /proc/4868/ns/net
Oct 30 07:31:40 node03 kubelet[1688]: 2020/10/30 07:31:40 First unreferenced table: 100
Oct 30 07:31:40 node03 kubelet[1688]: 2020/10/30 07:31:40 Moving route {Ifindex: 6 Dst: 192.168.33.0/24 Src: 192.168.33.170 Gw: <nil> Flags: [] Table: 254} from table 254 to 100
Oct 30 07:31:40 node03 kubelet[1688]: 2020-10-30T07:31:40Z [verbose] Add: default:pod1-100:c5276974-2997-480a-90d5-26e14e2a8b76:default/signet-100(signet-100):net1 {"cniVersion":"0.3.1","interfaces":[{"name":"veth30ce3682","mac":"22:49:b5:d4:32:bd"},{"name":"net1","mac":"02:00:00:5e:23:80","sandbox":"/proc/4868/ns/net"}],"ips":[{"version":"4","interface":0,"address":"192.168.33.170/24","gateway":"192.168.33.1"}],"routes":[{"dst":"0.0.0.0/0"}],"dns":{}}

The result structure returned by OVS-CNI would contain two interfaces, first the host veth interface, followed by the container interface: "interfaces":[{"name":"veth30ce3682","mac":"22:49:b5:d4:32:bd"},{"name":"net1","mac":"02:00:00:5e:23:80","sandbox":"/proc/4868/ns/net"}],.

The result structure as returned by IPAM references the interface at index 0: "ips":[{"version":"4","interface":0,"address":"192.168.33.170/24","gateway":"192.168.33.1"}].

As a result, sbr is not capable to match the IP with the interface, since the container interface net1 is actually at index 1: Incorrect interface for IP address 192.168.33.170.

Support multiple nodes and OCP in local cluster

As discussed in #1, it is desirable to test the code with multiple nodes and on different platforms too. In order to accomplish that, we should copy more of KubeVirt cluster helpers.

It is not clear to me yet how to do it, not pile unused and unrelated code from kubevirt here yet.

Mirror port

Is it possibile to inject a mirror (span) port into a pod? Can't find any reference in code or documentation. Thanks

Can I use k8s service with ovs-cni?

Hi guys,

k8s service is based on the kube-proxy and iptables . I don't know whether service is available while ovs-cni is enable in K8s ?

for example , I configure one service called myservice . Can it work normally with ovs-cni ?

apiVersion: v1
kind: Service
metadata:
  name: myservice
spec:
  selector:
    app: myapp
  ports:
  \- protocol: TCP
    port: 80
    targetPort: 80

Support for probes for ovs cni marker

We have a requirement where we want to monitor ovs-cni-marker for liveness so that it can restarted in case of process hangs/failures.

Migrate the image under NPWG Quay repository

The image is still hosted under the original kubevirt org.

Dockerized tests fail due to missing Fedora 34 image

The fedora:34 image no longer exists on quay.io, thus dockerized tests are failing.

make docker-test                                                                              
hack/test-dockerized.sh
[+] Building 0.4s (3/3) FINISHED                                                                                   
 => [internal] load build definition from Dockerfile                                                          0.0s
 => => transferring dockerfile: 999B                                                                          0.0s
 => [internal] load .dockerignore                                                                             0.0s
 => => transferring context: 2B                                                                               0.0s
 => ERROR [internal] load metadata for quay.io/fedora/fedora:34-x86_64                                        0.3s
------
 > [internal] load metadata for quay.io/fedora/fedora:34-x86_64:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: quay.io/fedora/fedora:34-x86_64: not found
make: *** [docker-test] Error 1

Implement CHECK command

Currently, we do not implement the CHECK command [1]. Study the requirements in CNI SPEC [2] and implement the functionality for OVS CNI. Reference implementation can be found in Linux bridge CNI [3].

[1] https://github.com/kubevirt/ovs-cni/blob/master/pkg/plugin/plugin.go#L398
[2] https://github.com/containernetworking/cni/blob/master/SPEC.md
[3] https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/bridge.go#L742

Mark functional tests as such and run them on local cluster

Currently, there are functional tests (calling ovs-vsctl binary) ran as unit tests. We should mark them as functional tests, run them only with make functest and do that on local cluster.

Automate OVS bridge deployment

Hi,

We are using ovs-cni inside our Kubernetes cluster and we were wondering if there was any known way to automate OVS bridges deployments as a pod get "attached" to an OVS bridge. As far as I know, it doesn't seem to be available with ovs-cni but will it be a next feature or not at all ?

Thanks.

[enhancement] CNI should mark nodes according to existence of bridges

Daemon on each node will monitor available OVS bridges and set them on respective node label, by appending the bridge name to the label key name.
For example, if the following OVS bridges exists on the node:

br-red
br-yellow
br-blue

The the following node labels will be added:

apiVersion: v1
kind: Node
metadata:
  labels:
    kubernetes.io/ovs-cni-br-red: 1
    kubernetes.io/ovs-cni-br-yellow: 1
    kubernetes.io/ovs-cni-br-blue: 1

The bridges will be added/deleted periodically from the list.
> - "fake" bridges (ports) should be filtered out.
> - There might be OVS bridges which are not intended for our use, adding them to the list does no harm unless they are mapped to a requested network in the NetworkAttachmentDefinition CRD. This should be a configuration error in the NetworkAttachmentDefinition CRD, and not an issue that should be detected here. A future enhancement may be to add a list of blacklisted bridges as configuration.

A controller will run a MutatingAdmissionWebhook server that will receive the pod creation events, will look for pods that has the following annotation: ```k8s.v1.cni.cncf.io/networks```
If such annotation exists, the daemon will parse the list of network names, and will lookup each name into the list of network attachment CRDs of type ```ovs``` to get the bridge name from the network name. Based on these bridges, the following nodeAffinity sections (beta feature), will be added to the pod.

For example, assuming that:  ```k8s.v1.cni.cncf.io/networks=red-net,blue-net```
And that the following CRDs are configured:
```yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: red-net
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br-red",
    }'
```
and:
```yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: blue-net
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br-blue",
    }'
```
The mutating web hook will add the following nodeAffinity:
```yaml
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubevirt.io/ovs-cni-br-red
            operator: Exist
          - key: kubevirt.io/ovs-cni-br-blue
            operator: Exists
```

As mentioned above, two processes should exist:
- distributed daemon that is doing the bridge discovery on the nodes
- centralized controller doing the admission as well as the translation between the network name and the bridge name (using the NetworkAttachmentDefinition CRD)

Not all supported platforms offer Go 1.10 or higher

In the current state, with using network namespaces from Go, we enforce Go 1.10, since this code cannot be compiled on Go 1.9 or lower [1]. However, at this moment we need to work with Go 1.9 to support all platforms.

We should create a temporary workaround to work in container network namespace using nsenter or ip netns.

This code should be reverted and issue closed as soon as Go 1.9 platforms support is dropped.

[1] https://www.weave.works/blog/linux-namespaces-and-go-don-t-mix

error adding container to network "work-network": failed to find bridge ovs-br1 on Microk8s

Using Microk8 for cluster setup, experiencing " error adding container to network "work-network": failed to find bridge ovs-br1" issue, ovs-br1 have been created on all nodes as below:

Kubernetes Distribution: Microk8s

Error Log

Events:
  Normal   AddedInterface          51s                 multus             Add eth0 [10.1.231.211/32]  
  Warning  FailedCreatePodSandBox  51s                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a2ffab9564353e1e97c6049d027ac72f01290ff4db1ccd07e0bfd87acd12d9e5": [default/multus-deployment-55c7b5f4fb-2gth5:work-network]: error adding container to network "work-network": failed to find bridge ovs-br1

ovs-cni installed

yockgenm@tgl02:~$ kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS              RESTARTS           AGE
kube-system   kube-multus-ds-amd64-wfvfb                 1/1     Running             9 (7d ago)         28d
kube-system   ovs-cni-amd64-jrsmv                        1/1     Running             0                  19h
kube-system   calico-node-vvcj8                          1/1     Running             5 (2d23h ago)      7d
kube-system   ovs-cni-amd64-qf4v4                        1/1     Running             0                  19h
kube-system   kube-multus-ds-amd64-w8sjn                 1/1     Running             5 (2d23h ago)      7d
kube-system   coredns-64c6478b6c-dt4t4                   1/1     Running             8 (7d ago)         26d
kube-system   calico-kube-controllers-59c6c9c94c-prhqb   1/1     Running             7 (7d ago)         24d
kube-system   calico-node-s6g77                          1/1     Running             7 (7d ago)         24d
kube-system   kube-multus-ds-fplck                       0/1     CrashLoopBackOff    1289 (55s ago)     7d
kube-system   kube-multus-ds-fl55x                       1/1     Running             2610 (5m47s ago)   24d
default       multus-deployment-55c7b5f4fb-wjq4d         0/1     ContainerCreating   0                  11m
default       multus-deployment-55c7b5f4fb-kqtkj         0/1     ContainerCreating   0                  11m
default       multus-deployment-55c7b5f4fb-2gth5         0/1     ContainerCreating   0                  11m

OvS Bridge

//node 1
yockgenm@tgl02:~$ sudo ovs-vsctl show
c8d62735-42a6-4326-823a-f4ed39ee4722
    Bridge ovs-br1
        Port ovs-br1
            Interface ovs-br1
                type: internal
    ovs_version: "2.16.2"

//node 2
root@yockgenm-tgl01:~# ovs-vsctl show
1d00c8e0-309c-4b80-aa96-498089c59a83
    Bridge ovs-br1
        datapath_type: netdev
        Port ovs-br1
            Interface ovs-br1
                type: internal
        Port dpdk-p0
            Interface dpdk-p0
                type: dpdk
                options: {dpdk-devargs="0000:58:00.0", flow-ctrl-autoneg="true"}
    ovs_version: "2.16.90"

Network Definition

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: work-network
  annotations:
    k8s.v1.cni.cncf.io/resourceName: ovs-cni.network.kubevirt.io/br1
spec:
  config: '{
      "cniVersion": "0.4.0",
      "type": "ovs",
      "bridge": "ovs-br1",
      "vlan": 100,
	  "ipam": {
        "type": "host-local",
        "subnet": "192.168.1.0/24",
        "rangeStart": "192.168.1.201",
        "rangeEnd": "192.168.1.250",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "192.168.1.1"
      }
    }'
EOF

Pod Deployment

cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: multus-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: multuspod
  template:
    metadata:
      annotations:
        k8s.v1.cni.cncf.io/networks: work-network
      labels:
        app: multuspod
    spec:
      containers:
      - name: multuspod
        command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
        image: alpine

EOF

Any help is really appreciate!

Dose it support IPAM now ?

The ovs-cni's version comes to v0.9.0, dose it support IPAM from this version？
If it still not own this feature, is there anothor tool to act as IPAM and used together?
or is there a plan to implement IPAM?

Host local IPAM configured with erroneous NAD results in stale ovs ports

When using ovs-cni with host-local IPAM, if there is any invalid configuration specific to IPAM in the NAD, ovs port is created but IPAM rejects the configuration and throws an error. The ovs port is not cleared. This results in multiple stale ports over a period of time.
How to reproduce:

1. Create a NAD with the following content.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ovs-green-bad-net
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br_vin",
      "mtu": 9000,
      "vlan": 1029,
      "ipam": {
          "type": "host-local",
          "ranges": [
              [ {
                   "subnet": "172.100.29.0/24",
                   "rangeStart": "172.100.29.100",
                   "rangeEnd": "172.100.100.254",
                   "gateway": "172.100.100.1"
              } ]
          ]
      }
  }'

2. create a sample deployment that uses this NAD.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ovs-hello-app-dep
  name: ovs-hello-app-dep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ovs-hello-app
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: ovs-hello-app
      annotations:
        k8s.v1.cni.cncf.io/networks: ovs-green-bad-net
    spec:
      containers:
      - image: gcr.io/google-samples/hello-app:1.0
        imagePullPolicy: IfNotPresent
        name: hello-app-container

We see error messages in the pod log and ports are created which are never removed.

Use Fedora for Docker images

KubeVirt projects use Fedora. In order to save some disk space, we should too.

Can ping but iperf fails

Setup everything following demo.md. Ping is working fine, but iperf gives 0 Mbps.
Also mentioned in #145 but no solution given

Document deployment and usage on arbitrary cluster

Currently, we document only usage on local kubevirtci cluster. We should add a description how to install multus and ovs-cni on a cluster and how to use it. This depends on #10

dial unix /run/multus/multus.sock: connect: connection refused

Hello,

I followed the README guide to install multus-cni:
cat ./deployments/multus-daemonset.yml | kubectl apply -f -

and i think multus-cni is successfully installed by check multus-pod status:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  43m   default-scheduler  Successfully assigned kube-system/kube-multus-ds-kvrh8 to master
  Normal  Pulled     43m   kubelet            Container image "ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot" already present on machine
  Normal  Created    43m   kubelet            Created container install-multus-binary
  Normal  Started    43m   kubelet            Started container install-multus-binary
  Normal  Pulled     43m   kubelet            Container image "ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot" already present on machine
  Normal  Created    43m   kubelet            Created container kube-multus
  Normal  Started    43m   kubelet            Started container kube-multus

Then i try to install a samplepod, but i get this error:
error adding container to network "multus-cni-network": CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": dial unix /run/multus/multus.sock: connect: connection refused

Could anyone know what's the problem?
Thank you

can't use ovs cni on ocp4 without workarounds

on ocp4, the ovs cni plugin suffers from two issues:

• the rhcos nodes miss the libunbound.so.2 library.
  as a workaround, i copied the library from the ovs pods to /root on all the nodes and added this path to ld

  echo /root > /etc/ld.so.conf.d/root.conf
  ldconfig

• the path for searching cni plugins is different so i had to manually copy the ovs plugin binary to the proper place

cp /opt/cni/bin/ovs /var/lib/cni/bin

Fail to deploy ovs example

Fail to deploy ovs on openshift with the follow error.

NAMESPACE                           NAME                              READY     STATUS    RESTARTS   AGE
kube-system                         ovs-cni-plugin-amd64-xjmdn        0/1       Error     3          24m

[sscheink@sscheink kubevirt]$ oc logs -f ovs-cni-plugin-amd64-xjmdn -n kube-system
cp: cannot create regular file '/host/opt/cni/bin/ovs': Permission denied

CNI failed to retrieve network namespace path

Hi, I think we found a bug maybe. Or I made a misstake with the configuration and would be thankful for help.

Generel the ovs-cni works great. But if I hard kill a Kubernetes Node (or do a reboot without draining the node), on which are pods running with the ovs-cni, the pods does not restart propably.

I startet the samplepod from the documentation and than reboot the ubuntu system which runs the kubernetes node (with reboot, no drain before).
Before the reboot, the pod is running well.
After the reboot kubectl get pods says:
samplepod-1 0/1 Error 0 8m1s <none>

When I describe the Pod I get:
Normal SandboxChanged 8s (x14 over 2m41s) kubelet, HOSTNAME Pod sandbox changed, it will be killed and re-created.

And the Kubelet Log says:

Feb 18 11:34:35 HOSTNAME kubelet[1219]: 2020/02/18 11:34:35 CNI DEL was called for container ID: bc79277db8b567a2864807a9d310eb240bffed01d840df0135ea34198ffd017b, network namespace , interface name
Feb 18 11:34:35 HOSTNAME kubelet[1219]: panic: This should never happen, if it does, it means caller does not pass container network namespace as a parameter and therefore OVS port cleanup will not
Feb 18 11:34:35 HOSTNAME kubelet[1219]: goroutine 1 [running, locked to thread]:
Feb 18 11:34:35 HOSTNAME kubelet[1219]: github.com/kubevirt/ovs-cni/pkg/plugin.CmdDel(0xc0000f2620, 0xc000014500, 0x5)
Feb 18 11:34:35 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/pkg/plugin/plugin.go:276 +0x792
Feb 18 11:34:35 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.(*dispatcher).checkVersionAndCall(0xc00009bed8, 0xc0000f2620, 0x71a080, 0xc0000cc4e0, 0x6d0d30, 0x0, 0xc00009bec0)
Feb 18 11:34:35 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:185 +0x259
Feb 18 11:34:35 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.(*dispatcher).pluginMain(0xc00009bed8, 0x6d0d20, 0x6d0d28, 0x6d0d30, 0x71a080, 0xc0000cc4e0, 0xc00001c060, 0x25, 0
Feb 18 11:34:35 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:251 +0x3d4
Feb 18 11:34:35 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.PluginMainWithError(...)
Feb 18 11:34:35 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:286
Feb 18 11:34:35 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.PluginMain(0x6d0d20, 0x6d0d28, 0x6d0d30, 0x71a080, 0xc0000cc4e0, 0xc00001c060, 0x25)
Feb 18 11:34:35 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:301 +0x129
Feb 18 11:34:35 HOSTNAME kubelet[1219]: main.main()
Feb 18 11:34:35 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/cmd/plugin/main.go:26 +0x11c
Feb 18 11:34:35 HOSTNAME kubelet[1219]: E0218 11:34:35.943468    1219 cni.go:379] Error deleting loadbalancer_samplepod-1/bc79277db8b567a2864807a9d310eb240bffed01d840df0135ea34198ffd017b from networ
Feb 18 11:34:35 HOSTNAME kubelet[1219]: E0218 11:34:35.944454    1219 remote_runtime.go:128] StopPodSandbox "bc79277db8b567a2864807a9d310eb240bffed01d840df0135ea34198ffd017b" from runtime service fa
Feb 18 11:34:35 HOSTNAME kubelet[1219]: E0218 11:34:35.944527    1219 kuberuntime_manager.go:878] Failed to stop sandbox {"docker" "bc79277db8b567a2864807a9d310eb240bffed01d840df0135ea34198ffd017b"}
Feb 18 11:34:35 HOSTNAME kubelet[1219]: E0218 11:34:35.944618    1219 kuberuntime_manager.go:658] killPodWithSyncResult failed: failed to "KillPodSandbox" for "08155d12-cdcd-4c2d-a7cc-ad8f8f14761e"
Feb 18 11:34:35 HOSTNAME kubelet[1219]: E0218 11:34:35.944663    1219 pod_workers.go:191] Error syncing pod 08155d12-cdcd-4c2d-a7cc-ad8f8f14761e ("samplepod-1_loadbalancer(08155d12-cdcd-4c2d-a7cc-ad
Feb 18 11:34:38 HOSTNAME kubelet[1219]: E0218 11:34:38.621370    1219 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stat
Feb 18 11:34:38 HOSTNAME kubelet[1219]: W0218 11:34:38.854130    1219 cni.go:328] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "c12faf953
Feb 18 11:34:38 HOSTNAME kubelet[1219]: 2020/02/18 11:34:38 CNI DEL was called for container ID: c12faf95367d396da18c3b90b1099ebce6b372bc073622bba1d1b88bb621e2c6, network namespace , interface name
Feb 18 11:34:38 HOSTNAME kubelet[1219]: panic: This should never happen, if it does, it means caller does not pass container network namespace as a parameter and therefore OVS port cleanup will not
Feb 18 11:34:38 HOSTNAME kubelet[1219]: goroutine 1 [running, locked to thread]:
Feb 18 11:34:38 HOSTNAME kubelet[1219]: github.com/kubevirt/ovs-cni/pkg/plugin.CmdDel(0xc000106690, 0xc0000ea4a8, 0x5)
Feb 18 11:34:38 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/pkg/plugin/plugin.go:276 +0x792
Feb 18 11:34:38 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.(*dispatcher).checkVersionAndCall(0xc0000cbed8, 0xc000106690, 0x71a080, 0xc0000d04e0, 0x6d0d30, 0x0, 0xc0000cbec0)
Feb 18 11:34:38 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:185 +0x259
Feb 18 11:34:38 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.(*dispatcher).pluginMain(0xc0000cbed8, 0x6d0d20, 0x6d0d28, 0x6d0d30, 0x71a080, 0xc0000d04e0, 0xc00012a000, 0x25, 0
Feb 18 11:34:38 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:251 +0x3d4
Feb 18 11:34:38 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.PluginMainWithError(...)
Feb 18 11:34:38 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:286
Feb 18 11:34:38 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.PluginMain(0x6d0d20, 0x6d0d28, 0x6d0d30, 0x71a080, 0xc0000d04e0, 0xc00012a000, 0x25)
Feb 18 11:34:38 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:301 +0x129
Feb 18 11:34:38 HOSTNAME kubelet[1219]: main.main()
Feb 18 11:34:38 HOSTNAME kubelet[1219]:         /home/travis/gopath/src/github.com/kubevirt/ovs-cni/cmd/plugin/main.go:26 +0x11c

The Openvswitch Log says after reboot:

...
     Port "vethb0617177"
            Interface "vethb0617177"
                error: "could not open network device vethb0617177 (No such device)"
        Port "veth5f25970b"
            Interface "veth5f25970b"
                error: "could not open network device veth5f25970b (No such device)"
        Port "veth62e7faab"
            Interface "veth62e7faab"
                error: "could not open network device veth62e7faab (No such device)"
        Port "vethc5110e9d"
            Interface "vethc5110e9d"
                error: "could not open network device vethc5110e9d (No such device)"
        Port "vethef8643f7"
            Interface "vethef8643f7"
                error: "could not open network device vethef8643f7 (No such device)"
        Port "veth12efe3d7"
            Interface "veth12efe3d7"
                error: "could not open network device veth12efe3d7 (No such device)"

Here some Informations, which could help to reproduce this behaviour:
Kubernetes on Bare Metal Server
Kubernetes Version: v1.16.3
Multus-CNI Version: v3.4

Here is the deployment yaml:

---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-network-addons
  labels:
    name: cluster-network-addons

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: networkaddonsconfigs.networkaddonsoperator.network.kubevirt.io
spec:
  group: networkaddonsoperator.network.kubevirt.io
  names:
    kind: NetworkAddonsConfig
    listKind: NetworkAddonsConfigList
    plural: networkaddonsconfigs
    singular: networkaddonsconfig
  scope: Cluster
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          type: string
        kind:
          type: string
        metadata:
          type: object
        spec:
          type: object
        status:
          type: object
  version: v1alpha1
  versions:
  - name: v1alpha1
    served: true
    storage: true
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    kubevirt.io: ""
  name: cluster-network-addons-operator
  namespace: cluster-network-addons
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    name: cluster-network-addons-operator
  name: cluster-network-addons-operator
rules:
- apiGroups:
  - security.openshift.io
  resourceNames:
  - privileged
  resources:
  - securitycontextconstraints
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - operator.openshift.io
  resources:
  - networks
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networkaddonsoperator.network.kubevirt.io
  resources:
  - networkaddonsconfigs
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    kubevirt.io: ""
  name: cluster-network-addons-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-network-addons-operator
subjects:
  - kind: ServiceAccount
    name: cluster-network-addons-operator
    namespace: cluster-network-addons
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    name: cluster-network-addons-operator
  name: cluster-network-addons-operator
  namespace: cluster-network-addons
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - configmaps
  verbs:
  - get
  - list
  - watch
  - create
  - patch
  - update
  - delete
- apiGroups:
  - apps
  resources:
  - deployments
  - replicasets
  verbs:
  - get
  - list
  - watch
  - create
  - patch
  - update
  - delete

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    kubevirt.io: ""
  name: cluster-network-addons-operator
  namespace: cluster-network-addons
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cluster-network-addons-operator
subjects:
  - kind: ServiceAccount
    name: cluster-network-addons-operator

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    networkaddonsoperator.network.kubevirt.io/version: 0.23.0
  name: cluster-network-addons-operator
  namespace: cluster-network-addons
spec:
  replicas: 1
  selector:
    matchLabels:
      name: cluster-network-addons-operator
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: cluster-network-addons-operator
    spec:
      containers:
      - env:
        - name: MULTUS_IMAGE
          value: quay.io/kubevirt/cluster-network-addon-multus:v3.2.0-1.gitbf61002
        - name: LINUX_BRIDGE_IMAGE
          value: quay.io/kubevirt/cni-default-plugins:v0.8.1
        - name: LINUX_BRIDGE_MARKER_IMAGE
          value: quay.io/kubevirt/bridge-marker:0.2.0
        - name: NMSTATE_HANDLER_IMAGE
          value: quay.io/nmstate/kubernetes-nmstate-handler:v0.12.0
        - name: OVS_CNI_IMAGE
          value: quay.io/kubevirt/ovs-cni-plugin:v0.8.0
        - name: OVS_MARKER_IMAGE
          value: quay.io/kubevirt/ovs-cni-marker:v0.8.0
        - name: KUBEMACPOOL_IMAGE
          value: quay.io/kubevirt/kubemacpool:v0.8.0
        - name: OPERATOR_IMAGE
          value: quay.io/kubevirt/cluster-network-addons-operator:0.23.0
        - name: OPERATOR_NAME
          value: cluster-network-addons-operator
        - name: OPERATOR_VERSION
          value: 0.23.0
        - name: OPERATOR_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: OPERAND_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: WATCH_NAMESPACE
        image: quay.io/kubevirt/cluster-network-addons-operator:0.23.0
        imagePullPolicy: Always
        name: cluster-network-addons-operator
        resources: {}
      serviceAccountName: cluster-network-addons-operator

Thanks for your help. And let me know, if you need more inforamtions.

Rename the key for ovs-cni.network.kubevirt.io/br1

We have problem with net-attach-def objects when it is required to have both device (for device plugin) and bridge (for ovs-cni) as resource names.
This is mainly because both are described with same k8s.v1.cni.cncf.io/resourceName key which makes recent value replaces the old one.

For example:

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ovs-conf
  annotations:
    k8s.v1.cni.cncf.io/resourceName: ovs-cni.network.kubevirt.io/br1
    k8s.v1.cni.cncf.io/resourceName: intel.com/mellanox_snic0
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br1",
      "vlan": 100
    }'
EOF

Can we have different annotation key for defining ovs bridge name like below ?

  annotations:
    k8s.v1.cni.cncf.io/bridgeName: ovs-cni.network.kubevirt.io/br1
    k8s.v1.cni.cncf.io/resourceName: intel.com/mellanox_snic0

Then we can enhance network resources injector to support both resources and pod spec can be injected with appropriate resource requests/limits.

OVS-CNI doesn't free IPs after container deleted

I have defined devnet with range 10.71.11.1-10.71.11.253 in my test environment.
I've created and deleted different depoloyments with pods that had IP assigned. I one moment I've got an error while creating pod:

Warning FailedCreatePodSandBox 52s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "86c8de27be9761bb120710364470dcf3f13db7de6b604f3a7e42000c004a43a4" network for pod "delme-slb-7fd58cb956-b7trg": networkPlugin cni failed to set up pod "delme-slb-7fd58cb956-b7trg_default" network: Multus: [default/delme-slb-7fd58cb956-b7trg]: error adding container to network "devnet": delegateAdd: error invoking DelegateAdd - "ovs": error in getting result from AddNetwork: failed to set up IPAM plugin type "host-local": failed to allocate for range 0: no IP addresses available in range set: 10.71.11.1-10.71.11.253, failed to clean up sandbox container "86c8de27be9761bb120710364470dcf3f13db7de6b604f3a7e42000c004a43a4" network for pod "delme-slb-7fd58cb956-b7trg": networkPlugin cni failed to teardown pod "delme-slb-7fd58cb956-b7trg_default" network: delegateDel: error invoking DelegateDel - "ovs": error in getting result from DelNetwork: Failed to obtain OVS port for given connection: failed to find object from table Port / delegateDel: error invoking ConflistDel - "cbr0": conflistDel: error converting the raw bytes into a conflist: error parsing configuration list: no 'plugins' key]

In /var/lib/cni/networks/devnet i've found:

root@k8s-1:/var/lib/cni/networks/devnet# ls -alF
итого 420
drwxr-xr-x 2 root root 4096 дек 11 10:14 ./
drwxr-xr-x 4 root root   32 окт 23 10:10 ../
-rw-r--r-- 1 root root   70 дек  4 10:50 10.71.11.1
-rw-r--r-- 1 root root   70 дек  4 10:55 10.71.11.10
-rw-r--r-- 1 root root   70 ноя 17 09:43 10.71.11.109
-rw-r--r-- 1 root root   70 дек  4 10:55 10.71.11.11
-rw-r--r-- 1 root root   70 ноя 17 09:43 10.71.11.113
-rw-r--r-- 1 root root   70 дек  4 10:55 10.71.11.12
-rw-r--r-- 1 root root   70 дек  4 10:55 10.71.11.13
-rw-r--r-- 1 root root   70 дек  4 10:56 10.71.11.14
-rw-r--r-- 1 root root   70 дек  4 10:56 10.71.11.15
-rw-r--r-- 1 root root   70 дек  4 10:56 10.71.11.16
-rw-r--r-- 1 root root   70 дек  1 10:18 10.71.11.164
-rw-r--r-- 1 root root   70 дек  1 10:18 10.71.11.165
-rw-r--r-- 1 root root   70 дек  1 10:19 10.71.11.166
-rw-r--r-- 1 root root   70 дек  1 10:21 10.71.11.168
-rw-r--r-- 1 root root   70 дек  1 11:06 10.71.11.169
-rw-r--r-- 1 root root   70 дек  4 10:57 10.71.11.17
-rw-r--r-- 1 root root   70 дек  1 11:33 10.71.11.170
-rw-r--r-- 1 root root   70 дек  1 11:33 10.71.11.171
-rw-r--r-- 1 root root   70 дек  1 11:33 10.71.11.172
-rw-r--r-- 1 root root   70 дек  1 11:33 10.71.11.173
-rw-r--r-- 1 root root   70 дек  1 11:34 10.71.11.174
-rw-r--r-- 1 root root   70 дек  1 11:34 10.71.11.175
-rw-r--r-- 1 root root   70 дек  1 11:34 10.71.11.176
-rw-r--r-- 1 root root   70 дек  1 11:35 10.71.11.177
-rw-r--r-- 1 root root   70 дек  1 11:37 10.71.11.178
-rw-r--r-- 1 root root   70 дек  1 11:37 10.71.11.179
-rw-r--r-- 1 root root   70 дек  4 10:57 10.71.11.18
-rw-r--r-- 1 root root   70 дек  1 11:37 10.71.11.180
-rw-r--r-- 1 root root   70 дек  1 11:37 10.71.11.181
-rw-r--r-- 1 root root   70 дек  1 11:37 10.71.11.182
-rw-r--r-- 1 root root   70 дек  1 11:37 10.71.11.183
-rw-r--r-- 1 root root   70 дек  1 11:37 10.71.11.184
-rw-r--r-- 1 root root   70 дек  1 11:37 10.71.11.185
-rw-r--r-- 1 root root   70 дек  1 11:39 10.71.11.186
-rw-r--r-- 1 root root   70 дек  1 11:39 10.71.11.187
-rw-r--r-- 1 root root   70 дек  1 11:39 10.71.11.188
-rw-r--r-- 1 root root   70 дек  1 11:40 10.71.11.189
-rw-r--r-- 1 root root   70 дек  4 10:57 10.71.11.19
-rw-r--r-- 1 root root   70 дек  1 11:40 10.71.11.190
-rw-r--r-- 1 root root   70 дек  1 11:40 10.71.11.191
-rw-r--r-- 1 root root   70 дек  1 11:40 10.71.11.192
-rw-r--r-- 1 root root   70 дек  1 11:40 10.71.11.193
-rw-r--r-- 1 root root   70 дек  4 10:57 10.71.11.20
-rw-r--r-- 1 root root   70 дек  4 10:57 10.71.11.21
-rw-r--r-- 1 root root   70 дек  4 12:30 10.71.11.22
-rw-r--r-- 1 root root   70 дек  1 12:29 10.71.11.220
-rw-r--r-- 1 root root   70 дек  1 12:29 10.71.11.221
-rw-r--r-- 1 root root   70 дек  1 12:44 10.71.11.222
-rw-r--r-- 1 root root   70 дек  1 12:44 10.71.11.223
-rw-r--r-- 1 root root   70 дек  1 12:44 10.71.11.224
-rw-r--r-- 1 root root   70 дек  1 12:44 10.71.11.225
-rw-r--r-- 1 root root   70 дек  1 12:44 10.71.11.226
-rw-r--r-- 1 root root   70 дек  1 12:44 10.71.11.227
-rw-r--r-- 1 root root   70 дек  4 12:30 10.71.11.23
-rw-r--r-- 1 root root   70 дек  1 12:47 10.71.11.230
-rw-r--r-- 1 root root   70 дек  1 12:47 10.71.11.231
-rw-r--r-- 1 root root   70 дек  1 12:47 10.71.11.232
-rw-r--r-- 1 root root   70 дек  1 12:47 10.71.11.233
-rw-r--r-- 1 root root   70 дек  1 12:48 10.71.11.234
-rw-r--r-- 1 root root   70 дек  1 12:48 10.71.11.235
-rw-r--r-- 1 root root   70 дек  1 12:48 10.71.11.236
-rw-r--r-- 1 root root   70 дек  1 12:48 10.71.11.237
-rw-r--r-- 1 root root   70 дек  1 12:51 10.71.11.238
-rw-r--r-- 1 root root   70 дек  4 12:30 10.71.11.24
-rw-r--r-- 1 root root   70 дек  2 17:41 10.71.11.243
-rw-r--r-- 1 root root   70 дек  2 17:41 10.71.11.244
-rw-r--r-- 1 root root   70 дек  2 17:41 10.71.11.245
-rw-r--r-- 1 root root   70 дек  2 17:41 10.71.11.246
-rw-r--r-- 1 root root   70 дек  2 17:44 10.71.11.247
-rw-r--r-- 1 root root   70 дек  2 17:44 10.71.11.248
-rw-r--r-- 1 root root   70 дек  4 12:30 10.71.11.25
-rw-r--r-- 1 root root   70 дек  4 10:50 10.71.11.253
-rw-r--r-- 1 root root   70 дек  4 12:30 10.71.11.26
-rw-r--r-- 1 root root   70 дек  4 12:30 10.71.11.27
-rw-r--r-- 1 root root   70 дек  7 12:20 10.71.11.28
-rw-r--r-- 1 root root   70 дек  7 12:20 10.71.11.29
-rw-r--r-- 1 root root   70 дек  7 12:20 10.71.11.30
-rw-r--r-- 1 root root   70 дек  7 12:20 10.71.11.31
-rw-r--r-- 1 root root   70 дек  7 12:30 10.71.11.32
-rw-r--r-- 1 root root   70 дек  7 12:30 10.71.11.33
-rw-r--r-- 1 root root   70 дек  7 12:30 10.71.11.34
-rw-r--r-- 1 root root   70 дек  7 12:30 10.71.11.35
-rw-r--r-- 1 root root   70 дек  7 12:31 10.71.11.36
-rw-r--r-- 1 root root   70 дек  7 12:31 10.71.11.37
-rw-r--r-- 1 root root   70 дек  7 12:31 10.71.11.38
-rw-r--r-- 1 root root   70 дек  7 12:31 10.71.11.39
-rw-r--r-- 1 root root   70 дек  7 12:32 10.71.11.40
-rw-r--r-- 1 root root   70 дек  7 12:32 10.71.11.41
-rw-r--r-- 1 root root   70 дек  7 13:15 10.71.11.42
-rw-r--r-- 1 root root   70 дек  7 13:15 10.71.11.43
-rw-r--r-- 1 root root   70 дек  7 13:15 10.71.11.44
-rw-r--r-- 1 root root   70 дек  7 13:15 10.71.11.45
-rw-r--r-- 1 root root   70 дек 11 10:14 10.71.11.46
-rw-r--r-- 1 root root   70 дек 11 10:14 10.71.11.47
-rw-r--r-- 1 root root   70 дек 11 10:14 10.71.11.48
-rw-r--r-- 1 root root   70 окт 28 11:12 10.71.11.49
-rw-r--r-- 1 root root   70 дек 11 10:14 10.71.11.50
-rw-r--r-- 1 root root   70 дек 11 10:14 10.71.11.51
-rw-r--r-- 1 root root   70 дек 11 10:14 10.71.11.52
-rw-r--r-- 1 root root   70 дек  4 10:53 10.71.11.6
-rw-r--r-- 1 root root   70 дек  4 10:53 10.71.11.7
-rw-r--r-- 1 root root   70 дек  4 10:53 10.71.11.8
-rw-r--r-- 1 root root   70 дек  4 10:53 10.71.11.9
-rw-r--r-- 1 root root   11 дек 11 10:14 last_reserved_ip.0
-rwxr-x--- 1 root root    0 окт 23 10:10 lock*

So I think IPAM doesn't deletes corresponding file in /var/lib/cni/networks/<network_name> when pod is deleted, so adress pool overflows.

pod creation fails with OF port not up error

While creating 100s of pod which uses secondary network using ovs-cni, some pods fails with following error The OF port xxxxx state is not up,

subsequently CmdDel is invoked to delete the pod, but that fails with Error deallocating IP: Did not find reserved IP for container and this error causes deadloop for sandbox deletion, CmdDel is invoked multiple times and causes kubelet log filled up with these error. There should be a fix to be done to avoid this deadloop.

Events:
  Type     Reason                  Age                 From               Message
  ----     ------                  ----                ----               -------
  Normal   Scheduled               16m                 default-scheduler  Successfully assigned bat-t1/cnf-complex-t1-1-net-947d44f4b-92d54 to pool08-n108-wk08-n055
  Normal   AddedInterface          16m                 multus             Add eth0 [192.168.146.114/32]
  Warning  FailedCreatePodSandBox  15m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "9e268d66cf446f1d8cc8ade7d6beb5f196488509b773de20e2d2c3b436bd6d83" network for pod "cnf-complex-t1-1-net-947d44f4b-92d54": networkPlugin cni failed to set up pod "cnf-complex-t1-1-net-947d44f4b-92d54_bat-t1" network: [bat-t1/cnf-complex-t1-1-net-947d44f4b-92d54:net-traf-1-1]: error adding container to network "net-traf-1-1": The OF port veth8ba8c86e state is not up, failed to clean up sandbox container "9e268d66cf446f1d8cc8ade7d6beb5f196488509b773de20e2d2c3b436bd6d83" network for pod "cnf-complex-t1-1-net-947d44f4b-92d54": networkPlugin cni failed to teardown pod "cnf-complex-t1-1-net-947d44f4b-92d54_bat-t1" network: delegateDel: error invoking DelegateDel - "ovs": error in getting result from DelNetwork: Error deallocating IP: Did not find reserved IP for container 9e268d66cf446f1d8cc8ade7d6beb5f196488509b773de20e2d2c3b436bd6d83 / delegateDel: error invoking DelegateDel - "ovs": error in getting result from DelNetwork: Error deallocating IP: Did not find reserved IP for container 9e268d66cf446f1d8cc8ade7d6beb5f196488509b773de20e2d2c3b436bd6d83 / delegateDel: error invoking DelegateDel - "ovs": error in getting result from DelNetwork: Error deallocating IP: Did not find reserved IP for container 9e268d66cf446f1d8cc8ade7d6beb5f196488509b773de20e2d2c3b436bd6d83]
  Normal   SandboxChanged          48s (x69 over 15m)  kubelet            Pod sandbox changed, it will be killed and re-created.

The configurable retry fix is here #158 and would be raising another PR to address the deadloop issue.

ovs-cni-marker privileges

Hi,
To limit ovs-cni pod securityContext. We removed privileges=true for ovs-cni-marker. We are tried with capabilities : CAP_IPC_LOCK/CAP_NET_BIND/CAP_NET_RAW which i think are required to connect to OVS DB. Could you please suggest what the minimum capability required for ovs-cni-marker container to work?
Error Seen:
F1206 07:05:08.670471 1 main.go:84] Failed to create a new marker object: Error creating the ovsdb connection: failed to connect to ovsdb error: failed to connect to ovsdb error: failed to connect to endpoints ["unix:/host/var/run/openvswitch/db.sock"]: dial unix /host/var/run/openvswitch/db.sock: connect: permission denied

Basic OVS CNI plugin implementation

With this initial implementation, ovs-cni would just connect container to an OVS bridge available on host and optionally assign connection port to a VLAN. Support for IPAM and other advanced features will be added later and described in separate issues. Pure L2 plugins sounds limiting, but it should be enough for KubeVirt VMs use-case.

This basic support should reflect bridge plugin from containernetworking/plugins. Code must have unit tests coverage, reasonable logging and standalone usage examples and usage example for local cluster.

Open vSwitch CNI Plugin

Overview

With ovs plugin, containers (on the same host) are plugged into an Open vSwitch bridge (virtual switch) that resides in the host network namespace. It's host adminitrator's responsibility to create such bridge and optinally connect it to broader network, be it using L2 directly, NAT or an overlay. The containers receive one end of the veth pair with the other end connected to the bridge.

Example configuration

{
    "name": "mynet",
    "type": "ovs",
    "bridge": "mynet0",
    "vlan": 100
}

Network configuration reference

• name (string, required): the name of the network.
• type (string, required): "ovs".
• bridge (string, required): name of the bridge to use.
• vlan (integer, optional): VLAN ID of attached port. Opened for all if not specified.

hard-code one supported CNI version

We should report only single supported CNI version.

Can I use k8s service with ovs-cni?

Hi guys,

k8s service is based on the kube-proxy and iptables . I don't know whether service is available while ovs-cni is enable in K8s ?

apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: myapp
ports:
- protocol: TCP
port: 80
targetPort: 80

Clean ports after host reboot

Hi, as described in this issue, I'm facing a problem on an OVS bridge after a restart of the node that hosts the pod (connected to the ovs bridge). A new port is added to the bridge (new interface of the container) but the older still exists and is in error state.

By calling the cleanPorts function in the cmdAdd function (pkg/plugin/plugin.go) may fix the problem.

Thanks for your help.

is its possible to ask for specific ofport id?

like with the following:
ovs-vsctl add-port br0 dummy -- set Interface dummy ofport_request=100
Thanks,

[ovs-cni] Apply ovs-cni.yaml fail

Hi,
We would like to set up ovs-cni.yaml on K8S. Below is our command:
kubectl apply -f ovs-cni.yaml

but found that the ovs-cni pod crash
kubectl get pod --all-namespaces
kube-system ovs-cni-amd64-sg5c2 0/1 CrashLoopBackOff 1 7s

and the container log is below
kubectl logs -f ovs-cni-amd64-sg5c2 -n kube-system --all-containers
I1012 03:18:41.081822 1 main.go:44] Found the OVS socket
F1012 03:18:41.082910 1 main.go:56] Failed to create a new marker object: Error creating the ovsdb connection: failed to connect to ovsdb error: dial unix /host/var/run/openvswitch/db.sock: connect: connection refused

Could you help to check this problem?

Please refer the attachment for our cni-ovs.yaml.
cni-ovs.txt

Thanks,
Willie

Use Network Addons Operator for deployment on arbitrary cluster

It will make deployment of Multus and OVS way easier.

Arm Support

Hello, is OVS-CNI usable for arm nodes? I noticed that it was crosscompiled to work for the arm environment, however, I'm getting an issue where the OVS CNI pods are crashing randomly only for my arm nodes.

The only error that's visible on the pod is as follows:

ovs-cni-marker qemu: uncaught target signal 11 (Segmentation fault) - core dumped

Is there a way to get more verbose logging options to determine where it had a segfault?

Thanks so much!

execute: `kubectl apply -f https://raw.githubusercontent.com/kubevirt/ovs-cni/master/examples/kubernetes-ovs-cni.yml`

I use kubeadm init initialize the cluster, and install cni according to https://github.com/kubevirt/ovs-cni/blob/master/docs/deployment-on-arbitrary-cluster.md file, the result is:

execute：kubectl describe po ovs-cni-amd64-8v9gm --namespace=kube-system

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  33m (x3 over 33m)    default-scheduler  0/2 nodes are available: 1 node(s) didn't match node selector, 1 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  9m22s (x5 over 14m)  default-scheduler  0/1 nodes are available: 1 node(s) didn't match node selector.

Please do not care about the first warning, I have deleted another node. I would like to ask why the mismatch is missing something？

OVS-CNI does not supprt tcp package between vms in different nodes

Hi

https://github.com/kubevirt/ovs-cni/blob/master/docs/demo.md

like this demo, i build two vms using ovs-cni network, this two virtual machines can ping each other and send UDP packets，but can not establish TCP connection。

Do you have this problem？

Relevant situations are shown in the figure below

1、node‘s ovs port info

2、vm‘s ping info

3、vm‘s udp info

4、vm‘s tcp info

Thanks everyone~

Remove built binaries from repo

I accidentally pushed plugin binary to this repository, we need to drop it.

Plugin not getting installed on new nodes

When adding new nodes to an existing cluster, I'm getting failed to find plugin "ovs" in path [/opt/cni/bin /opt/cni/bin] when trying to use the plugin for a pod running on the new node.

When looking into the indicated folder, I see that the old nodes do have the ovs file on the folder, but the new nodes do not have it.

I'm not sure if this is responsibility of the plugin, but letting you know anyway.
If I find a workaround I let you know.

Thanks.

Publish images on Quay.io

In order to make this plugin easily usable outside of kubevirtci local cluster, we should provide a public image of installer on Quay.io.

That should be done automatically, for master and all tags.

Drop OpenShift 3.11 from CI in favor of OKD4

We should focus on OKD4 and newer Kubernetes versions with both deployment and CI.

Is there a way to make ovs-cni as the master network with different vlans

It's easy to make ovs-cni as the master network with or without one vlan tag. It's also possible for me to make two ovs vlans with NetworkAttachmentDefinition , and each pod has one of the vlans as the second network with multus plugin.

However, there a case is that two pods use ovs-cni plugin as master network with different ovs vlan tags . Dose the ovs-cni support this scenario. If so, is there a guide availeble for me?

Thanks!