akihirosuda / cni-isolation

CNI Bridge Isolation Plugin (Merged into the firewall plugin v1.1.0)

🎉 Merged into the CNI firewall plugin (v1.1.0)

The isolation plugin was merged into the ingressPolicy parameter the CNI firewall plugin v1.1.0: https://github.com/containernetworking/plugins/commit/22dd6c553dbb9fd3e34502c0de16dd17c7461a83

{
  "type": "firewall",
  "backend": "iptables",
  "ingressPolicy": "same-bridge"
}

The standalone isolation plugin is now deprecated.

---

CNI Bridge Isolation Plugin (/opt/cni/bin/isolation)

The isolation plugin isolates CNI bridge networks as in Docker bridge networks.

This plugin is mostly expected to be used by the following projects:

• containerd (nerdctl)
• BuildKit
• Podman

This plugin is being proposed to be merged into the CNI upstream firewall plugin: containernetworking/plugins#584

Install

make && sudo make install

The binary is installed as /opt/cni/bin/isolation .

Configuration

nerdctl

nerdctl (>= 0.5.0) uses the isolation plugin by default.

Run nerdctl network inspect --mode-native <NETWORK> to confirm the current configuration.

$ nerdctl network inspect --mode=native bridge
[
    {
        "CNI": {
            "cniVersion": "0.4.0",
            "name": "bridge",
            "nerdctlID": 0,
            "nerdctlLabels": {},
            "plugins": [
                {
                    "type": "bridge",
                    "bridge": "nerdctl0",
                    "isGateway": true,
                    "ipMasq": true,
                    "hairpinMode": true,
                    "ipam": {
                        "ranges": [
                            [
                                {
                                    "gateway": "10.4.0.1",
                                    "subnet": "10.4.0.0/24"
                                }
                            ]
                        ],
                        "routes": [
                            {
                                "dst": "0.0.0.0/0"
                            }
                        ],
                        "type": "host-local"
                    }
                },
                {
                    "type": "portmap",
                    "capabilities": {
                        "portMappings": true
                    }
                },
                {
                    "type": "firewall"
                },
                {
                    "type": "tuning"
                },
                {
                    "type": "isolation"
                }
            ]
        },
        "NerdctlID": 0,
        "NerdctlLabels": {}
    }
]

Others

{"type":"isolation"} needs to be inserted AFTER {"type":"bridge"}.

When {"type":"firewall"} is also present, {"type":"isolation"} needs to be inserted AFTER {"type":"firewall"}.

Example:

{
   "cniVersion": "0.4.0",
   "name": "foo1",
   "plugins": [
      {
         "type": "bridge",
         "bridge": "cni1",
         "isGateway": true,
         "ipMasq": true,
         "hairpinMode": true,
         "ipam": {
            "type": "host-local",
            "routes": [
               {
                  "dst": "0.0.0.0/0"
               }
            ],
            "ranges": [
               [
                  {
                     "subnet": "10.88.3.0/24",
                     "gateway": "10.88.3.1"
                  }
               ]
            ]
         }
      },
      {
         "type": "firewall"
      },
      {
         "type": "isolation"
      }
   ]
}

How it works

The isolation plugin follows the behavior of Docker libnetwork (DOCKER-ISOLATION-STAGE-1 and DOCKER-ISOLATION-STAGE-2).

To isolate CNI bridges (cni1, cni2, ...), the isolation plugin executes the following iptables commands.

iptables -N CNI-ISOLATION-STAGE-1
iptables -N CNI-ISOLATION-STAGE-2
# NOTE: "-j CNI-ISOLATION-STAGE-1" needs to be before "CNI-FORWARD" created by CNI firewall plugin. So we use -I here.
iptables -I FORWARD -j CNI-ISOLATION-STAGE-1
iptables -A CNI-ISOLATION-STAGE-1 -i cni1 ! -o cni1 -j CNI-ISOLATION-STAGE-2
iptables -A CNI-ISOLATION-STAGE-1 -i cni2 ! -o cni2 -j CNI-ISOLATION-STAGE-2
iptables -A CNI-ISOLATION-STAGE-1 -j RETURN
iptables -A CNI-ISOLATION-STAGE-2 -o cni1 -j DROP
iptables -A CNI-ISOLATION-STAGE-2 -o cni2 -j DROP
iptables -A CNI-ISOLATION-STAGE-2 -j RETURN

The number of commands is O(N) where N is the number of the bridges (not the number of the containers).

Run sudo iptables-save -t filter to confirm the added rules.

FAQs

Is this plugin made for Kubernetes?

No, this plugin is mostly made for single-node CNI projects, including, but not limited to, containerd, BuildKit, and Podman.

Why isolation needs to be specified after firewall ?

When isolation is specified before firewall, the isolation rules are discarded, because firewall plugin inserts a FORWARD rule that jumps to CNI-FORWARD chain before our CNI-ISOLATION-STAGE-1 chain.

We will be able to relax this order constraint after adding isolation plugin to the CNI upstream.

iptables rules are not removed after deletion of all containers

The iptables rules are created per bridge, not per container. So the iptables rules remain even after deletion of all containers.

To remove iptables rules created by isolation plugin, use hack/show-commands-to-delete-isolation-rules.sh.

cni-isolation's Issues

Blocks access to published host ports

Thanks for building this plugin, it's great for isolating containers from each other for security. I noticed one issue when using it: It doesn't only block access to the internal networks of other containers, it also blocks access from containers to public host ports (tried to access via the public host ip) if those are published from a different container. I would like my containers to be isolated from each other's non-published ports, but I want them to be able to talk to the ports that are open and published to the internet anyways. I'm not sure if this is due to the initial SYN packet being blocked by iptables or whether the ACK gets blocked.

Example:

Host: server.example.com

Container 1: publishes port 80 to host port 80. Connected to other containers via internal networks.

Container 2: Connected to other containers via internal networks. Cannot access server.example.com:80 even though that's public and other hosts can access it through the public ip.