rootless-containers / bypass4netns Goto Github PK

View Code? Open in Web Editor NEW

111.0 111.0 6.0 409 KB

[Experimental] Accelerates slirp4netns using SECCOMP_IOCTL_NOTIF_ADDFD. As fast as `--net=host`.

Home Page: https://medium.com/nttlabs/accelerating-rootless-container-network-29d0e908dda4

License: Apache License 2.0

Shell 43.24% Makefile 0.43% Go 47.22% Dockerfile 0.45% Python 8.66%

bypass4netns's People

Contributors

Stargazers

Watchers

Forkers

denji beemesh akihirosuda naoki9911 redlinejoes amirulandalib

bypass4netns's Issues

Add a link to the presentation on July 2022

Naoki Matsumoto, Akihiro Suda, Accelerating TCP/IP Communications in Rootless Containers by Socket Switching. 研究報告システムソフトウェアとオペレーティング・システム（OS）, 2022-OS-156 pp. 1-7, SWoPP2022(海峡メッセ下関), 2022年7月

Paper (English)

IPSJ-OS22156009.pdf

Copyright notice

Redistributed in compliance of 情報処理学会著作権規程 (2017年6月2日改訂) 第5条5項.

（著作者の権利）
第5条
...
5．著作者は、投稿した論文等について本学会の出版物発行前後にかかわらず、いつでも著作者個人のWebサイト（著作者所属組織のサイトを含む。以下同じ。）において自ら創作した著作物を掲載することができる。ただし、掲載に際して「情報処理学会倫理綱領」に則ること、ならびに本学会の出版物にかかる出典（当該出版物が発行された場合）及び利用上の注意事項＊＊を明記しなければならない。

Notice for the use of this material

The copyright of this material is retained by the Information Processing Society of Japan (IPSJ).
This material is published on this web site with the agreement of the author (s) and the IPSJ.
Please be complied with Copyright Law of Japan and the Code of Ethics of the IPSJ if any users wish to reproduce, make derivative work, distribute or make available to the public any part or whole thereof.
All Rights Reserved, Copyright (C) Information Processing Society of Japan.
Comments are welcome. Mail to address editj＠ipsj.or.jp, please.

Slides (Japanese)

https://speakerdeck.com/mt2naoki/ip-communications-in-rootless-containers-by-socket-switching

Migrate Vagrant tests to GHA

bypass4netns/Vagrantfile

Line 5 in df17c2e

config.vm.box = "ubuntu/jammy64"

GitHub Actions now provides ubuntu-22.04 nodes, so we no longer need to use Vagrant

multinode: etcd should be protected with mTLS

`bind(2)`-related syscalls do not work well with nested containers

In the following example, nerdctl exec dind docker exec nginx wget -O- http://127.0.0.1 is working as expected, but nerdctl exec dind docker exec nginx wget -O- http://172.17.0.2 is failing with Host is unreachable

$ nerdctl run -d --name dind --annotation nerdctl/bypass4netns=1 --privileged docker:dind

$ nerdctl exec dind docker run -d --name nginx nginx:alpine

$ nerdctl exec dind docker exec nginx wget -O- http://127.0.0.1
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
[...]
</html>
Connecting to 127.0.0.1 (127.0.0.1:80)
writing to stdout
-                    100% |********************************|   615  0:00:00 ETA
written to stdout

$ nerdctl exec dind docker exec nginx ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

$ nerdctl exec dind docker exec nginx wget -O- http://172.17.0.2
Connecting to 172.17.0.2 (172.17.0.2:80)
wget: can't connect to remote host (172.17.0.2): Host is unreachable
FATA[0003] exec failed with exit code 1

nerdctl: v2.0.0-beta.3
bypass4netns: the current master 2794f7e

Probably the cause is same as:

cc @naoki9911

Use `SECCOMP_ADDFD_FLAG_SEND`

To inject it at socket(2) time safely, though, we need to use SECCOMP_ADDFD_FLAG_SEND in the addfd call. I added that flag to the kernel due to a race condition you can easily hit otherwise: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.17-rc2&id=0ae71c7720e3ae3aabd2e8a072d27f7bd173d25c.

Originally posted by @rata in #1 (comment)

Ignore

`bind(2)`: remove hard-coded 8080:5201

bypass4netns/pkg/bypass4netns/bypass4netns.go

Line 406 in ae6712e

if port != 5201 {

bypass4netns/pkg/bypass4netns/bypass4netns.go

Line 419 in ae6712e

Port: int(8080),

bypass4netns/Vagrantfile

Line 72 in ae6712e

iperf3 -c "$(cat /tmp/host_ip)" -p 8080

[Usernetes] kube-apiserver fails to connect to etcd: `dial tcp 127.0.0.1:2379: connect: connection refused`

I'm trying to run Usernetes (single-node w/o VXLAN, as a baby step) with bypass4netnsd, but kubeadm fails:

$ export CONTAINER_ENGINE=nerdctl
$ make up
$ make kubeadm-init
[...]
kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Looks like kube-apiserver is failing to connect to the local etcd due to dial tcp 127.0.0.1:2379: connect: connection refused,
although the etcd process is running with --listen-client-urls=https://127.0.0.1:2379,https://10.100.201.100:2379.

Version

nerdctl: v2.0.0-beta.3
bypass4netns: the current master 2794f7e
Usernetes: gen2-v20240404.1 + the following annotations

diff --git a/docker-compose.yaml b/docker-compose.yaml
index 2ae7291..a036d80 100644
--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -39,6 +39,11 @@ services:
       # In addition, `net.ipv4.conf.default.rp_filter`
       # has to be set to 0 (disabled) or 2 (loose)
       # in the daemon's network namespace.
+    annotations:
+      # bypass4netns annotations are recognized since nerdctl v2.0
+      # TODO: enable bypass4netns only when bypass4netnsd is running.
+      "nerdctl/bypass4netns": "true"
+      "nerdctl/bypass4netns-ignore-subnets": "[\"10.244.0.0/16\", \"${U7S_NODE_SUBNET}\"]"
 networks:
   default:
     ipam:

Logs

$ nerdctl exec usernetes-node-1 sh -euxc 'tail /var/log/containers/kube-apiserver*'
+ tail /var/log/containers/kube-apiserver-u7s-suda-ws01_kube-system_kube-apiserver-e3bb8e1d239fbd21df5a10150d7cf97cddf2ac60f255e7c95e0444372270a590.log
2024-04-04T10:33:23.014467147Z stderr F W0404 10:33:23.014061       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-04-04T10:33:23.525905133Z stderr F W0404 10:33:23.525525       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-04-04T10:33:23.650741563Z stderr F W0404 10:33:23.650270       1 logging.go:59] [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-04-04T10:33:26.746781506Z stderr F W0404 10:33:26.746363       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-04-04T10:33:27.177688658Z stderr F W0404 10:33:27.177096       1 logging.go:59] [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-04-04T10:33:27.255248965Z stderr F W0404 10:33:27.254843       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-04-04T10:33:32.660451075Z stderr F W0404 10:33:32.660277       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-04-04T10:33:32.928362209Z stderr F W0404 10:33:32.927956       1 logging.go:59] [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-04-04T10:33:34.194906969Z stderr F W0404 10:33:34.194549       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-04-04T10:33:38.191445386Z stderr F F0404 10:33:38.190882       1 instance.go:290] Error creating leases: error creating storage factory: context deadline exceeded

$ nerdctl exec usernetes-node-1 sh -euxc 'tail /var/log/containers/etcd*'
+ tail /var/log/containers/etcd-u7s-suda-ws01_kube-system_etcd-5e02fa990f84a688cc176a9c61737122ca6bcaa5545e3629ad824977f582365f.log
2024-04-04T10:32:16.394968121Z stderr F {"level":"info","ts":"2024-04-04T10:32:16.394467Z","caller":"embed/serve.go:103","msg":"ready to serve client requests"}
2024-04-04T10:32:16.395555112Z stderr F {"level":"info","ts":"2024-04-04T10:32:16.395204Z","caller":"embed/serve.go:103","msg":"ready to serve client requests"}
2024-04-04T10:32:16.395949052Z stderr F {"level":"info","ts":"2024-04-04T10:32:16.395687Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
2024-04-04T10:32:16.395974454Z stderr F {"level":"info","ts":"2024-04-04T10:32:16.39574Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
2024-04-04T10:32:16.397006609Z stderr F {"level":"info","ts":"2024-04-04T10:32:16.394859Z","caller":"etcdserver/server.go:2571","msg":"setting up initial cluster version using v2 API","cluster-version":"3.5"}
2024-04-04T10:32:16.398486722Z stderr F {"level":"info","ts":"2024-04-04T10:32:16.398168Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"3f59b4f74cf82b90","local-member-id":"f3b2aef06a662d72","cluster-version":"3.5"}
2024-04-04T10:32:16.399161396Z stderr F {"level":"info","ts":"2024-04-04T10:32:16.398924Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
2024-04-04T10:32:16.399912362Z stderr F {"level":"info","ts":"2024-04-04T10:32:16.399745Z","caller":"etcdserver/server.go:2595","msg":"cluster version is updated","cluster-version":"3.5"}
2024-04-04T10:32:16.403525952Z stderr F {"level":"info","ts":"2024-04-04T10:32:16.403166Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"127.0.0.1:2379"}
2024-04-04T10:32:16.405592265Z stderr F {"level":"info","ts":"2024-04-04T10:32:16.405178Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"10.100.201.100:2379"}

$ nerdctl exec usernetes-node-1 ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 10:30 ?        00:00:01 /sbin/init
root         108       1  0 10:30 ?        00:00:00 /lib/systemd/systemd-journald
root         121       1  0 10:30 ?        00:00:00 /lib/systemd/systemd-udevd
root         129       1 10 10:30 ?        00:00:26 /usr/local/bin/containerd
root         149       0  1 10:30 pts/1    00:00:02 kubeadm init --config /tmp/kubeadm-config.yaml --skip-token-print
root         388       1  3 10:32 ?        00:00:04 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9 --runtime-cgroups=/system.slice/containerd.service --cloud-provider=external --node-labels=usernetes/host-ip=192.168.60.11
root         443       1  0 10:32 ?        00:00:00 /usr/local/bin/containerd-shim-runc-v2 -namespace k8s.io -id e0baeca2362cf60e66347e1bca8e1663b0bc33f173f049cc4c288cd31f0a021f -address /run/containerd/containerd.sock
root         453       1  0 10:32 ?        00:00:00 /usr/local/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4862c3b92124466a817f12e7ce4882e3602f01e73ff70c68157d758d26777122 -address /run/containerd/containerd.sock
root         487       1  0 10:32 ?        00:00:00 /usr/local/bin/containerd-shim-runc-v2 -namespace k8s.io -id aaa4c48903c861beda8db0d2acc6ccddf7006032512d948df01a9d9009b97657 -address /run/containerd/containerd.sock
root         508       1  0 10:32 ?        00:00:00 /usr/local/bin/containerd-shim-runc-v2 -namespace k8s.io -id e8179f9642fbd7b705a4c221d2efd167cebeabc47863b90f2966a16bee0f3ca0 -address /run/containerd/containerd.sock
65535        536     443  0 10:32 ?        00:00:00 /pause
65535        547     453  0 10:32 ?        00:00:00 /pause
65535        560     487  0 10:32 ?        00:00:00 /pause
65535        576     508  0 10:32 ?        00:00:00 /pause
root         680     443  0 10:32 ?        00:00:01 kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cloud-provider=external --cluster-cidr=10.244.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/16 --use-service-account-credentials=true
root         691     487  1 10:32 ?        00:00:02 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/scheduler.conf --authorization-kubeconfig=/etc/kubernetes/scheduler.conf --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
root         815     508  1 10:32 ?        00:00:01 etcd --advertise-client-urls=https://10.100.201.100:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --experimental-initial-corrupt-check=true --experimental-watch-progress-notify-interval=5s --initial-advertise-peer-urls=https://10.100.201.100:2380 --initial-cluster=u7s-suda-ws01=https://10.100.201.100:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://10.100.201.100:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://10.100.201.100:2380 --name=u7s-suda-ws01 --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
root        1111       0  0 10:34 ?        00:00:00 ps -ef

Running with bypass4netns on kubernetes docker in docker

I've been having a read of https://pibvt.net/IPSJ-OS22156009.pdf and trying to understand how we can implement bypass4netns into our existing Kubernetes based docker in docker implementation.

I'm not entirely sure where this would need to run. We currently launch rootless docker in docker host using: https://github.com/harrison-ai/cobalt-docker-rootless-nvidia-dind/blob/main/entrypoint.sh.

Would we run bypass4net inside that container or on the underlying host itself, presenting the socket all the way through?

Ideally we could run it in the container and pass through a seccomp profile and keep it all. but I fear that bypass4net needs to actually listen on the host itself?

create `bypass4netnsd.lock` to prohibit running multiple `bypass4netnsd` simultaneously

Port forwarding does not work for `httpd` while it works for `nginx`

$ nerdctl run -d --name nginx --label nerdctl/bypass4netns=1 -p 0.0.0.0:8080:80 nginx:alpine
00918a185bc32375e7e313a895d739a5f35472f6aa52066df65f3d6081412ea2

$ nerdctl run -d --name httpd --label nerdctl/bypass4netns=1 -p 0.0.0.0:8081:80 httpd:alpine
1192539b3e3feb5db8777b387c407d90a2e6b0a9f1ee91e3845d23646b6590e5

$ curl localhost:8080                                                                                                                                                                                    [34/34]
<!DOCTYPE html>                                                                                                                     
<html>                                                                                                                                                                                                                                                                   
<head>                                                                                                                                                                                                                                                                   
<title>Welcome to nginx!</title>                                                                                                                                                                                                                                         
<style>                                                                                                                                                                                                                                                                  
html { color-scheme: light dark; }                                                                                                                                                                                                                                       
body { width: 35em; margin: 0 auto;                                                                                                                                                                                                                                      
font-family: Tahoma, Verdana, Arial, sans-serif; }                                                                                  
</style>                                                                                                                            
</head>                                                                                                                             
<body>                                                                                                                              
<h1>Welcome to nginx!</h1>                                                                                                          
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>                                                                                     
                                                                                                                                    
<p>For online documentation and support please refer to                                                                             
<a href="http://nginx.org/">nginx.org</a>.<br/>                                                                                     
Commercial support is available at                                                                                                  
<a href="http://nginx.com/">nginx.com</a>.</p>                                                                                      
                                                                                                                                    
<p><em>Thank you for using nginx.</em></p>                                                                                          
</body>                                                                                                                             
</html>

$ curl localhost:8081                                                                                                                                                                                           
curl: (7) Failed to connect to localhost port 8081 after 0 ms: Connection refused

Log:

$ cat ~/.local/share/nerdctl/1935db59/containers/default/1192539b3e3feb5db8777b387c407d90a2e6b0a9f1ee91e3845d23646b6590e5/bypass4netns.log 
time="2022-08-19T16:58:07+09:00" level=info msg="LogFilePath: /home/suda/.local/share/nerdctl/1935db59/containers/default/1192539b3e3feb5db8777b387c407d90a2e6b0a9f1ee91e3845d23646b6590e5/bypass4netns.log"
time="2022-08-19T16:58:07+09:00" level=info msg="SocketPath: /run/user/1001/bypass4netns/1192539b3e3feb5.sock"
time="2022-08-19T16:58:07+09:00" level=info msg="127.0.0.0/8 is added to ignore"
time="2022-08-19T16:58:07+09:00" level=info msg="10.0.0.0/8 is added to ignore"
time="2022-08-19T16:58:07+09:00" level=info msg="fowarding port 8081:80 (host=8081 container=80) is added"
time="2022-08-19T16:58:07+09:00" level=info msg="Waiting for seccomp file descriptors"
time="2022-08-19T16:58:07+09:00" level=info msg="notify ready fd=3"
time="2022-08-19T16:58:07+09:00" level=info msg="accept connection"
time="2022-08-19T16:58:07+09:00" level=info msg="Received new seccomp fd: 3"
time="2022-08-19T16:58:07+09:00" level=info msg="handle port=65535, ip=[127 0 0 1]" pid=27124 sockfd=3 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 10" pid=27124 sockfd=3 syscall=connect
time="2022-08-19T16:58:07+09:00" level=info msg="handle port=65535, ip=[0 0 0 0]" pid=27124 sockfd=3 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 10" pid=27124 sockfd=3 syscall=connect
time="2022-08-19T16:58:07+09:00" level=info msg="handle port=65535, ip=[127 0 0 1]" pid=27124 sockfd=3 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 10" pid=27124 sockfd=3 syscall=connect
time="2022-08-19T16:58:07+09:00" level=info msg="handle port=65535, ip=[0 0 0 0]" pid=27124 sockfd=3 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 10" pid=27124 sockfd=3 syscall=connect
time="2022-08-19T16:58:07+09:00" level=info msg="handle port=65535, ip=[127 0 0 1]" pid=27124 sockfd=5 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 10" pid=27124 sockfd=5 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 10" pid=27124 sockfd=4 syscall=bind
time="2022-08-19T16:58:07+09:00" level=info msg="handle port=65535, ip=[127 0 0 1]" pid=27124 sockfd=5 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 10" pid=27124 sockfd=5 syscall=connect
time="2022-08-19T16:58:07+09:00" level=info msg="handle port=65535, ip=[0 0 0 0]" pid=27124 sockfd=5 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 10" pid=27124 sockfd=5 syscall=connect
time="2022-08-19T16:58:07+09:00" level=info msg="handle port=65535, ip=[127 0 0 1]" pid=27124 sockfd=5 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 10" pid=27124 sockfd=5 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 1" pid=27227 sockfd=8 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 1" pid=27226 sockfd=8 syscall=connect
time="2022-08-19T16:58:07+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 1" pid=27228 sockfd=8 syscall=connect

Host:

bypass4netns 752f90f
containerd/nerdctl@31455b1
Ubuntu 22.04, kernel 5.15

multinode: watch Kubernetes service resources instead of depending on etcd

Deploying etcd is hard, so maybe we should just watch Kubernetes services instead.

cc @naoki9911

`/run/user/1000/bypass4netns/{.pid, .sock}` should be garbage collected

/run/user/1000/bypass4netns/{*.pid, *.sock} are created on nerdctl run and remain unremoved

CI: stuck at benchmark (iperf3/iperf3)

Stuck at nerdctl exec iperf3-server hostname -i

It must succeed and SERVER_IP returned.

Update test scripts and CI after nerdctl v2.0 released

containerd/nerdctl#2906

In nerdctl v2, labels not passed through the OCI runtime since nerdctl v2.0, with an exception for nerdctl/bypass4netns.
We should migrate to --annotation (--annotation nerdctl/bypass4netns).

`apt-get` hangs with bypass4netns

$ nerdctl run -it --rm --label nerdctl/bypass4netns=1 debian:11
root@1fac92b42639:/# apt-get update
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
0% [1 InRelease 64.9 kB/116 kB 56%]

dnf also seems very slow

bypass4netns v0.3.0, nerdctl v0.23.0 (Lima v0.12.0)

Certain containers do not work

The simplest container I could find that wouldn't work was this(webinterface cannot be accessed):

nerdctl run -it --rm -p 8080:8080 --label nerdctl/bypass4netns=true lscr.io/linuxserver/qbittorrent

it works fine without bypass4netns. I tired to debug this further and came up empty, but the behavior does seem reproducible across systems.

Getting the logs by stracing gave me https://pastebin.com/5GemdrEA (too long to drop it here directly)

`connect(2)`: remove hard-coded 10.0.0.0/8 and 172.0.0.0/8

bypass4netns/pkg/bypass4netns/bypass4netns.go

Lines 339 to 344 in ae6712e

    
           case 10: 
        
           	logrus.Infof("skipping (possibly) (`podmain|nerdctl) network create` network ip=%v", addrInet.Addr) 
        
           	return 
        
           case 172: 
        
           	logrus.Infof("skipping (possibly) (`docker network create` network ip=%v", addrInet.Addr) 
        
           	return

rewrite in Go or in Rust

Eliminate TOCTOU races for connect(2)

The TOCTOU races mentioned in the README can be eliminated if, instead of allowing the connect(2) syscall to continue, we run the syscall on behalf of the container and then just forward the return code & errno.

I mean, we can copy the syscall params and make sure they are out of localhost and such, then we run the syscall.Connect ourselves, and then answer the seccomp notification with the return value & errno we got when running it. In this way, we can chose to run it only when it is safe to do so.

What do you think?

Avoid assuming `seccompFd` idx to be 0

bypass4netns/main.c

Line 250 in d1ad9bd

if ((notify_fd = recvfd(accept_fd)) < 0) {

We should parse The Container Process State JSON rather than assuming the index of seccompFd to be always 0.

fds (array, OPTIONAL) is a string array containing the names of the file descriptors passed.
The index of the name in this array corresponds to index of the file descriptors in the SCM_RIGHTS array.

Add back UDP support

UDP support was temporarily removed in v0.4:

Port forwarding does not work for `iperf3 -s` while it works for `iperf3 -s -4`

$ nerdctl run -it --rm --label nerdctl/bypass4netns=1 -p 0.0.0.0:5201:5201 alpine
/ # apk add iperf3
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
(1/1) Installing iperf3 (3.11-r0)
Executing busybox-1.35.0-r17.trigger
OK: 6 MiB in 15 packages
/ # iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------

$ iperf3 -c 192.168.60.11
iperf3: error - unable to send control message: Bad file descriptor

$ cat ~/.local/share/nerdctl/1935db59/containers/default/42abba709dbacda0d46796458c49ed46bc1b3826fbfc675bc8278acb20198cd8/bypass4netns.log 
time="2022-08-19T17:25:57+09:00" level=info msg="LogFilePath: /home/suda/.local/share/nerdctl/1935db59/containers/default/42abba709dbacda0d46796458c49ed46bc1b3826fbfc675bc8278acb20198cd8/bypass4netns.log"
time="2022-08-19T17:25:57+09:00" level=info msg="SocketPath: /run/user/1001/bypass4netns/42abba709dbacda.sock"
time="2022-08-19T17:25:57+09:00" level=info msg="127.0.0.0/8 is added to ignore"
time="2022-08-19T17:25:57+09:00" level=info msg="10.0.0.0/8 is added to ignore"
time="2022-08-19T17:25:57+09:00" level=info msg="fowarding port 5201:5201 (host=5201 container=5201) is added"
time="2022-08-19T17:25:57+09:00" level=info msg="Waiting for seccomp file descriptors"
time="2022-08-19T17:25:57+09:00" level=info msg="notify ready fd=3"
time="2022-08-19T17:25:57+09:00" level=info msg="accept connection"
time="2022-08-19T17:25:57+09:00" level=info msg="Received new seccomp fd: 3"
time="2022-08-19T17:25:59+09:00" level=info msg="handle port=0, ip=[0 0 0 0]" pid=2074 sockfd=7 syscall=bind
time="2022-08-19T17:25:59+09:00" level=info msg="port=0 is not target of port forwarding." pid=2074 sockfd=7 syscall=bind
time="2022-08-19T17:25:59+09:00" level=info msg="handle port=53, ip=[10 0 2 3]" pid=2074 sockfd=7 syscall=sendto
time="2022-08-19T17:25:59+09:00" level=info msg="handle port=53, ip=[10 0 2 3]" pid=2074 sockfd=7 syscall=sendto
time="2022-08-19T17:25:59+09:00" level=info msg="handle port=53, ip=[10 0 2 3]" pid=2074 sockfd=7 syscall=sendto
time="2022-08-19T17:25:59+09:00" level=info msg="handle port=53, ip=[10 0 2 3]" pid=2074 sockfd=7 syscall=sendto
time="2022-08-19T17:25:59+09:00" level=info msg="handle port=65535, ip=[146 75 114 133]" pid=2074 sockfd=7 syscall=connect
time="2022-08-19T17:25:59+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 10" pid=2074 sockfd=7 syscall=connect
time="2022-08-19T17:25:59+09:00" level=info msg="handle port=443, ip=[146 75 114 133]" pid=2074 sockfd=7 syscall=connect
time="2022-08-19T17:26:02+09:00" level=error msg="failed to read addrInet4 from process: not AF_INET addr: 10" pid=2077 sockfd=3 syscall=bind

Possibly relevant:

Looks like we need to implement AF_INET6 support

	case 10:
	logrus.Infof("skipping (possibly) (`podmain\|nerdctl) network create` network ip=%v", addrInet.Addr)
	return
	case 172:
	logrus.Infof("skipping (possibly) (`docker network create` network ip=%v", addrInet.Addr)
	return

rootless-containers / bypass4netns Goto Github PK

bypass4netns's People

Contributors

Stargazers

Watchers

Forkers

bypass4netns's Issues

Paper (English)

Copyright notice

Slides (Japanese)

Version

Logs

Recommend Projects

Recommend Topics

Recommend Org