networkop / meshnet-cni Goto Github PK
View Code? Open in Web Editor NEWa (K8s) CNI plugin to create arbitrary virtual network topologies
License: BSD 3-Clause "New" or "Revised" License
meshnet-cni's People
Stargazers
Watchers
Forkers
wasabi222 th3architect gregory-development lachlan748 vparames86 tahir24434 qlyoung naguilar2 mhines01 hellt tiger7456 c479096292 azurecloudmonk packerliu silvengs futurewei-cloud mojiiba cerebus alexmasi h-fam bpandipp ashutshkumr y-young open-traffic-generator suniave sar772004 whuhxb maxgoffartmeshnet-cni's Issues
Can meshnet-cni support the SDN?
Hello, meshenet-cni can support the network created by k8s-topo well. Can it support the SDN (e.g., created by Mininet)? Is there an example? Thank you!
Bug: Meshnet `cmdDel` is not idempotent, leading to state changes when CNI ADD and CNI DEL commands overlap
I need help debugging this, b/c I really don't know what's going on.
When I create pods connected with meshnet, sometimes meshnet doesn't add the declared interfaces. This can happen at deployment or redeployment of a static pod. I set up my static pods with init containers that wait on the expected interfaces to be added, so it's easy to see when this happens--the Pod gets stuck in Init:0.
What I need to be able to do is reliably delete and restore static Pods from a topology. I do this by deleting the Pod, and /most/ of the time this works flawlessly--but sometimes it doesn't. It's difficult to reproduce this reliably. When it happens, I can sometimes recover by deleting the entire topology (all pods and topos) and re-deploying it. Sometimes the condition persists and I have to rollout restart meshnet. And sometimes I have to reboot the node or cluster to recover.
Looking at the meshnet log, when this happens meshnet thinks the other pod isn't up. As an example, at startup lf1 and lf5 link fine:
time="2023-03-22T15:27:06Z" level=info msg="Creating Veth struct with NetNS:/var/run/netns/cni-83b86931-410c-4f80-9846-bf9539cee642 and intfName: eth4, IP:192.168.0.6/31"
time="2023-03-22T15:27:06Z" level=info msg="Pod lf1 is retrieving peer pod lf5 information from meshnet daemon"
time="2023-03-22T15:27:06Z" level=info msg="Is peer pod lf5 alive?: true"
time="2023-03-22T15:27:06Z" level=info msg="Peer pod lf5 is alive"
time="2023-03-22T15:27:06Z" level=info msg="lf1 and lf5 are on the same host"
Then I delete pod lf1, wait a bit, then re-deploy lf1 from the same resource declaration. Then I get:
time="2023-03-22T15:39:06Z" level=info msg="Creating Veth struct with NetNS:/var/run/netns/cni-fff6bab6-5bb0-e511-f423-282bde768df2 and intfName: eth4, IP:192.168.0.6/31"
time="2023-03-22T15:39:06Z" level=info msg="Pod lf1 is retrieving peer pod lf5 information from meshnet daemon"
time="2023-03-22T15:39:06Z" level=info msg="Is peer pod lf5 alive?: false"
time="2023-03-22T15:39:06Z" level=info msg="Peer pod lf5 isn't alive yet, continuing"
meshnetd's log looks normal:
2023-03-22T10:39:06-05:00 time="2023-03-22T15:39:06Z" level=info msg="Retrieving lf5's metadata from K8s..." daemon=meshnetd
2023-03-22T10:39:06-05:00 time="2023-03-22T15:39:06Z" level=info msg="Reading pod lf5 from K8s" daemon=meshnetd
2023-03-22T10:39:06-05:00 time="2023-03-22T15:39:06Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=meshnet.v1beta1.Local grpc.start_time="2023-03-22T15:39:06Z" grpc.time_ms=8.889 peer.address="[::1]:46588" span.kind=server system=grpc
2023-03-22T10:39:06-05:00 time="2023-03-22T15:39:06Z" level=info msg="Skipping of pod lf5 by pod lf1" daemon=meshnetd
2023-03-22T10:39:06-05:00 time="2023-03-22T15:39:06Z" level=info msg="Reading pod lf1 from K8s" daemon=meshnetd
2023-03-22T10:39:06-05:00 time="2023-03-22T15:39:06Z" level=info msg="Update pod status lf1 from K8s" daemon=meshnetd
2023-03-22T10:39:06-05:00 time="2023-03-22T15:39:06Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Skip grpc.service=meshnet.v1beta1.Local grpc.start_time="2023-03-22T15:39:06Z" grpc.time_ms=27.629 peer.address="[::1]:46588" span.kind=server system=grpc
I've tried /also/ deleting the lf1 and/or lf5 topo, but that doesn't seem to have an effect.
If I recover and redeploy and eventually recreate this condition, the affected pods or interfaces will change even with the same topology and pod declarations.
Am I bumping up against some resource limit somewhere?
v0.4.0 causes core dump in kind cluster pods
#30 resolves the deploy issue, but this started happening on kind:
~/projects/mimesis/scratch/topo controller ^
mimesis > kubectl get pods
NAME READY STATUS RESTARTS AGE
test 1/1 Running 0 4s
test2 1/1 Running 0 4s
~/projects/mimesis/scratch/topo controller ^
mimesis > kubectl exec test -- ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 62:45:9e:05:0c:a9 brd ff:ff:ff:ff:ff:ff
29: eth1@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether c6:a8:2d:49:2b:ec brd ff:ff:ff:ff:ff:ff
~/projects/mimesis/scratch/topo controller ^
mimesis > kubectl exec test2 -- ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 2e:7f:61:75:cb:64 brd ff:ff:ff:ff:ff:ff
30: eth1@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 82:6b:fc:f4:42:e3 brd ff:ff:ff:ff:ff:ff
~/projects/mimesis/scratch/topo controller ^
mimesis > kubectl exec test2 -- ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
command terminated with exit code 139
ifconfig executes properly and this doesn't seem to affect functionality (though I haven't tested w/ my tc sidecar yet).
only one side veth can come up
hi,
Thanks for creating this, i'm new to this and trying to follow your post to lab ceos.
the issue i'm facing now is only one side of a veth can come up ( i have a master and two worker node, all are vm, cluster is using flannel for existing cni)
jack@ubuntu:~/meshnet-cni/tests$ k version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:03:42Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
jack@ubuntu:~/meshnet-cni/tests$ k get nodes
NAME STATUS ROLES AGE VERSION
kmaster.example.com Ready master 62m v1.17.4
kworker1.example.com Ready <none> 56m v1.17.4
kworker2.example.com Ready <none> 50m v1.17.4
jack@ubuntu:~$ k get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6955765f44-jcmzk 1/1 Running 0 66m
coredns-6955765f44-sgdcj 1/1 Running 0 66m
etcd-kmaster.example.com 1/1 Running 0 66m
kube-apiserver-kmaster.example.com 1/1 Running 0 66m
kube-controller-manager-kmaster.example.com 1/1 Running 0 66m
kube-flannel-ds-amd64-4nxbj 1/1 Running 0 66m
kube-flannel-ds-amd64-h4688 1/1 Running 1 61m
kube-flannel-ds-amd64-z4jbg 1/1 Running 1 55m
kube-proxy-gh6kv 1/1 Running 0 55m
kube-proxy-pxctp 1/1 Running 0 66m
kube-proxy-xf2pz 1/1 Running 0 61m
kube-scheduler-kmaster.example.com 1/1 Running 0 66m
jack@ubuntu:~$ k get pod -n meshnet
NAME READY STATUS RESTARTS AGE
meshnet-9zhpz 1/1 Running 0 44m
meshnet-b7l9r 1/1 Running 0 44m
meshnet-xlng2 1/1 Running 0 44m
jack@ubuntu:~$ k get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
r1 1/1 Running 0 14m 10.244.1.128 kworker1.example.com <none> <none>
r2 0/1 ContainerCreating 0 14m <none> kworker2.example.com <none> <none>
[root@kworker1 ~]# cat /etc/cni/net.d/*
{
"cniVersion": "0.2.0",
"name": "meshnet_network",
"type": "meshnet",
"delegate": {
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
}
}
{
"cniVersion": "0.2.0",
"name": "cbr0",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
{
"cniVersion": "0.2.0",
"name": "meshnet_network",
"type": "meshnet",
"delegate": {
"name": "dind0",
"bridge": "dind0",
"type": "bridge",
"isDefaultGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.244.1.0/24",
"gateway": "10.244.1.1"
}
}
}
describe non-working pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m56s default-scheduler Successfully assigned default/r2 to kworker2.example.com
Warning FailedCreatePodSandBox 5m54s kubelet, kworker2.example.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a4bf9de6d9998a7e03e0bab09fc3c31f2e822a76cc836e5ec23356d2a72c2812" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Warning FailedCreatePodSandBox 5m50s kubelet, kworker2.example.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "896b47ff23220843a2add57c5b67cf8a76b2f63a403ef38b8b8329a4b0ba445d" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Warning FailedCreatePodSandBox 5m47s kubelet, kworker2.example.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f836de51e920629873ceaf8b07cfcffc6dd96872020ab166233d33f86a510d68" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Warning FailedCreatePodSandBox 5m44s kubelet, kworker2.example.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c7f84caaa299043f9b5c8f776614f0e1b8d027772b34ab79130bf759310e447d" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Warning FailedCreatePodSandBox 5m40s kubelet, kworker2.example.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "973a1923112987391137d79583400963cf7159b32e0bb52a9e005ff0dfada274" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Warning FailedCreatePodSandBox 5m37s kubelet, kworker2.example.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9ff656f600bd75ad0b53e168c84bd3166beff9063201ff0c0f4cc6f08f363afe" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Warning FailedCreatePodSandBox 5m33s kubelet, kworker2.example.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "0642829533fbd8811538ba051ad5d59db7ad8d9481ec6d8c9322b0a09da4a18c" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Warning FailedCreatePodSandBox 5m30s kubelet, kworker2.example.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "36c95135ebd358611088027ce21237bdd3519efbbd233873adba21084b8d8a7a" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Warning FailedCreatePodSandBox 5m27s kubelet, kworker2.example.com Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "91bb09251676b381e2814c21fb6eb64c88316be948dd4cab7cbbb879b366543c" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Normal SandboxChanged 5m16s (x12 over 5m53s) kubelet, kworker2.example.com Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 51s (x81 over 5m23s) kubelet, kworker2.example.com (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b663f582c25b2b7fd30809243599d286f0a1ef6e14ac332cae7ec780e9b4d4bb" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
jack@ubuntu:~/meshnet-cni/tests$ k get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6955765f44-jcmzk 1/1 Running 0 53m
coredns-6955765f44-sgdcj 1/1 Running 0 53m
etcd-kmaster.example.com 1/1 Running 0 53m
kube-apiserver-kmaster.example.com 1/1 Running 0 53m
kube-controller-manager-kmaster.example.com 1/1 Running 0 53m
kube-flannel-ds-amd64-4nxbj 1/1 Running 0 53m
kube-flannel-ds-amd64-h4688 1/1 Running 1 48m
kube-flannel-ds-amd64-z4jbg 1/1 Running 1 41m
kube-proxy-gh6kv 1/1 Running 0 41m
kube-proxy-pxctp 1/1 Running 0 53m
kube-proxy-xf2pz 1/1 Running 0 48m
kube-scheduler-kmaster.example.com 1/1 Running 0 53m
jack@ubuntu:~/meshnet-cni/tests$ k get pod -n meshnet
NAME READY STATUS RESTARTS AGE
meshnet-9zhpz 1/1 Running 0 31m
meshnet-b7l9r 1/1 Running 0 31m
meshnet-xlng2 1/1 Running 0 31m
logs from kworker1
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: 2020/03/20 04:42:50 Parsing cni .conf file
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: 2020/03/20 04:42:50 Parsing CNI_ARGS environment variable
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Processing ADD POD in namespace default
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Calling delegateAdd for flannel
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 About to delegate Add to flannel
Mar 20 04:42:50 kworker1.example.com kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Mar 20 04:42:50 kworker1.example.com kernel: cni0: port 1(veth5a9d0475) entered blocking state
Mar 20 04:42:50 kworker1.example.com kernel: cni0: port 1(veth5a9d0475) entered disabled state
Mar 20 04:42:50 kworker1.example.com kernel: device veth5a9d0475 entered promiscuous mode
Mar 20 04:42:50 kworker1.example.com kernel: cni0: port 1(veth5a9d0475) entered blocking state
Mar 20 04:42:50 kworker1.example.com kernel: cni0: port 1(veth5a9d0475) entered forwarding state
Mar 20 04:42:50 kworker1.example.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Mar 20 04:42:50 kworker1.example.com NetworkManager[4533]: <info> [1584679370.9024] device (veth5a9d0475): carrier: link connected
Mar 20 04:42:50 kworker1.example.com NetworkManager[4533]: <info> [1584679370.9026] manager: (veth5a9d0475): new Veth device (/org/freedesktop/NetworkManager/Devices/257)
Mar 20 04:42:50 kworker1.example.com NetworkManager[4533]: <info> [1584679370.9032] device (cni0): carrier: link connected
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Master plugin has finished
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Master plugin result is IP4:{IP:{IP:10.244.1.128 Mask:ffffff00} Gateway:10.244.1.1 Routes:[{Dst:{IP:10.244.0.0 Mask:ffff0000} GW:10.244.1.1} {Dst:{IP:0.0.0.0 Mask:00000000} GW:10.244.1.1}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Looking up a default route to get the intf and IP for vxlan
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Default route is via 10.0.2.15@eth0
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Attempting to connect to local meshnet daemon
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Retrieving local pod information from meshnet daemon
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Setting pod alive status on meshnet daemon
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Starting to traverse all links
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Creating Veth struct with NetNS:/proc/6702/ns/net and intfName: eth1, IP:12.12.12.1/24
Mar 20 04:42:50 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:50 Retrieving peer pod r2 information from meshnet daemon
Mar 20 04:42:51 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:51 Is peer pod r2 alive?: false
Mar 20 04:42:51 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:51 Peer pod r2 isn't alive yet, continuing
Mar 20 04:42:51 kworker1.example.com kubelet[6089]: ADD| r1 |==> 2020/03/20 04:42:51 Connected all links, exiting with result IP4:{IP:{IP:10.244.1.128 Mask:ffffff00} Gateway:10.244.1.1 Routes:[{Dst:{IP:10.244.0.0 Mask:ffff0000} GW:10.244.1.1} {Dst:{IP:0.0.0.0 Mask:00000000} GW:10.244.1.1}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}
Mar 20 04:42:55 kworker1.example.com containerd[5573]: time="2020-03-20T04:42:55.382733489Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/56686fcac4ee4f9428e19071fd7ebc90fadcf57f5573d31f7d0e7c69c8fa02c4/shim.sock" debug=false pid=6868
logs from kworker2 (non-working side):
Mar 20 04:43:23 kworker2.example.com kubelet[6084]: 2020/03/20 04:43:23 Parsing cni .conf file
Mar 20 04:43:23 kworker2.example.com kubelet[6084]: 2020/03/20 04:43:23 Parsing CNI_ARGS environment variable
Mar 20 04:43:23 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:23 Processing ADD POD in namespace default
Mar 20 04:43:23 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:23 Calling delegateAdd for flannel
Mar 20 04:43:23 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:23 About to delegate Add to flannel
Mar 20 04:43:23 kworker2.example.com kubelet[6084]: W0320 04:43:23.994553 6084 pod_container_deletor.go:75] Container "6df5a24cda367c04f46af4e9f2b07b7f3759b7aa44e35bc74ad6b71903626dd1" not found in pod's containers
Mar 20 04:43:24 kworker2.example.com kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Mar 20 04:43:24 kworker2.example.com kernel: cni0: port 1(vethefc8bc7a) entered blocking state
Mar 20 04:43:24 kworker2.example.com kernel: cni0: port 1(vethefc8bc7a) entered disabled state
Mar 20 04:43:24 kworker2.example.com kernel: device vethefc8bc7a entered promiscuous mode
Mar 20 04:43:24 kworker2.example.com kernel: cni0: port 1(vethefc8bc7a) entered blocking state
Mar 20 04:43:24 kworker2.example.com kernel: cni0: port 1(vethefc8bc7a) entered forwarding state
Mar 20 04:43:24 kworker2.example.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Mar 20 04:43:24 kworker2.example.com NetworkManager[4518]: <info> [1584679404.2519] device (vethefc8bc7a): carrier: link connected
Mar 20 04:43:24 kworker2.example.com NetworkManager[4518]: <info> [1584679404.2521] manager: (vethefc8bc7a): new Veth device (/org/freedesktop/NetworkManager/Devices/145)
Mar 20 04:43:24 kworker2.example.com NetworkManager[4518]: <info> [1584679404.2528] device (cni0): carrier: link connected
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Master plugin has finished
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Master plugin result is IP4:{IP:{IP:10.244.2.73 Mask:ffffff00} Gateway:10.244.2.1 Routes:[{Dst:{IP:10.244.0.0 Mask:ffff0000} GW:10.244.2.1} {Dst:{IP:0.0.0.0 Mask:00000000} GW:10.244.2.1}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Looking up a default route to get the intf and IP for vxlan
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Default route is via 10.0.2.15@eth0
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Attempting to connect to local meshnet daemon
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Retrieving local pod information from meshnet daemon
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Setting pod alive status on meshnet daemon
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Starting to traverse all links
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Creating Veth struct with NetNS:/proc/24134/ns/net and intfName: eth1, IP:12.12.12.2/24
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Retrieving peer pod r1 information from meshnet daemon
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Is peer pod r1 alive?: true
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Peer pod r1 is alive
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 r2 and r1 are on the same host
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Creating Veth struct with NetNS:/proc/6702/ns/net and intfName: eth1, IP:12.12.12.1/24
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Does the link already exist? Local:false, Peer:false
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Neither link exists. Checking if we've been skipped
Mar 20 04:43:24 kworker2.example.com kernel: IPv6: ADDRCONF(NETDEV_UP): koko4004429188: link is not ready
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Have we been skipped by our peer r1? &{true {} [] %!t(int32=0)}
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 DO we have a higher priority? true
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Peer POD has skipped us or we have a higher priority
Mar 20 04:43:24 kworker2.example.com kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
Mar 20 04:43:24 kworker2.example.com NetworkManager[4518]: <info> [1584679404.4756] manager: (koko2028365937): new Veth device (/org/freedesktop/NetworkManager/Devices/146)
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 Error when creating a new VEth pair with koko: failed to Statfs "/proc/6702/ns/net": no such file or directory
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 MY VETH STRUCT: (*api.VEth)(0xc00033c4e0)({
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: NsName: (string) (len=18) "/proc/24134/ns/net",
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: LinkName: (string) (len=4) "eth1",
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: IPAddr: ([]net.IPNet) (len=1 cap=1) {
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: (net.IPNet) 12.12.12.2/24
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: },
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: MirrorEgress: (string) "",
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: MirrorIngress: (string) ""
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: })
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: ADD| r2 |==> 2020/03/20 04:43:24 PEER STRUCT: (*api.VEth)(0xc00033c600)({
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: NsName: (string) (len=17) "/proc/6702/ns/net",
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: LinkName: (string) (len=4) "eth1",
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: IPAddr: ([]net.IPNet) (len=1 cap=1) {
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: (net.IPNet) 12.12.12.1/24
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: },
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: MirrorEgress: (string) "",
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: MirrorIngress: (string) ""
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: })
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: E0320 04:43:24.511239 6084 cni.go:364] Error adding default_r2/6df5a24cda367c04f46af4e9f2b07b7f3759b7aa44e35bc74ad6b71903626dd1 to network meshnet/meshnet_network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: DEL | r2 |==> 2020/03/20 04:43:24 Processing DEL request
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: DEL | r2 |==> 2020/03/20 04:43:24 Retrieving pod's metadata from meshnet daemon
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: DEL | r2 |==> 2020/03/20 04:43:24 Topology data still exists in CRs, cleaning up it's status
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: DEL | r2 |==> 2020/03/20 04:43:24 Iterating over each link for clean-up
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: DEL | r2 |==> 2020/03/20 04:43:24 Creating Veth struct with NetNS:/proc/24134/ns/net and intfName: eth1, IP:12.12.12.2/24
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: DEL | r2 |==> 2020/03/20 04:43:24 Removing link eth1
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: time="2020-03-20T04:43:24Z" level=info msg="koko: remove veth link eth1"
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: DEL | r2 |==> 2020/03/20 04:43:24 Error removing Veth link: failed to lookup "eth1" in "/proc/24134/ns/net": Link not found
Mar 20 04:43:24 kworker2.example.com kubelet[6084]: DEL | r2 |==> 2020/03/20 04:43:24 Setting skip-reverse flag on peer r1
Mar 20 04:43:25 kworker2.example.com kubelet[6084]: DEL | r2 |==> 2020/03/20 04:43:25 Calling delegateDel for flannel
Mar 20 04:43:25 kworker2.example.com kernel: cni0: port 1(vethefc8bc7a) entered disabled state
Mar 20 04:43:25 kworker2.example.com kernel: device vethefc8bc7a left promiscuous mode
Mar 20 04:43:25 kworker2.example.com kernel: cni0: port 1(vethefc8bc7a) entered disabled state
Mar 20 04:43:25 kworker2.example.com NetworkManager[4518]: <info> [1584679405.6315] device (vethefc8bc7a): released from master device cni0
Mar 20 04:43:25 kworker2.example.com containerd[5547]: time="2020-03-20T04:43:25.779911710Z" level=info msg="shim reaped" id=6df5a24cda367c04f46af4e9f2b07b7f3759b7aa44e35bc74ad6b71903626dd1
Mar 20 04:43:25 kworker2.example.com dockerd[5548]: time="2020-03-20T04:43:25.790773644Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Mar 20 04:43:25 kworker2.example.com kubelet[6084]: E0320 04:43:25.878963 6084 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "6df5a24cda367c04f46af4e9f2b07b7f3759b7aa44e35bc74ad6b71903626dd1" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Mar 20 04:43:25 kworker2.example.com kubelet[6084]: E0320 04:43:25.879003 6084 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "r2_default(b422aa7a-3959-40f2-8a61-62c2108f1811)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "6df5a24cda367c04f46af4e9f2b07b7f3759b7aa44e35bc74ad6b71903626dd1" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Mar 20 04:43:25 kworker2.example.com kubelet[6084]: E0320 04:43:25.879015 6084 kuberuntime_manager.go:729] createPodSandbox for pod "r2_default(b422aa7a-3959-40f2-8a61-62c2108f1811)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "6df5a24cda367c04f46af4e9f2b07b7f3759b7aa44e35bc74ad6b71903626dd1" network for pod "r2": networkPlugin cni failed to set up pod "r2_default" network: failed to Statfs "/proc/6702/ns/net": no such file or directory
Mar 20 04:43:25 kworker2.example.com kubelet[6084]: E0320 04:43:25.879049 6084 pod_workers.go:191] Error syncing pod b422aa7a-3959-40f2-8a61-62c2108f1811 ("r2_default(b422aa7a-3959-40f2-8a61-62c2108f1811)"), skipping: failed to "CreatePodSandbox" for "r2_default(b422aa7a-3959-40f2-8a61-62c2108f1811)" with CreatePodSandboxError: "CreatePodSandbox for pod \"r2_default(b422aa7a-3959-40f2-8a61-62c2108f1811)\" failed: rpc error: code = Unknown desc = failed to set up sandbox container \"6df5a24cda367c04f46af4e9f2b07b7f3759b7aa44e35bc74ad6b71903626dd1\" network for pod \"r2\": networkPlugin cni failed to set up pod \"r2_default\" network: failed to Statfs \"/proc/6702/ns/net\": no such file or directory"
Mar 20 04:43:26 kworker2.example.com kubelet[6084]: W0320 04:43:26.031197 6084 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "r2_default": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "6df5a24cda367c04f46af4e9f2b07b7f3759b7aa44e35bc74ad6b71903626dd1"
Mar 20 04:43:26 kworker2.example.com kubelet[6084]: W0320 04:43:26.036593 6084 pod_container_deletor.go:75] Container "6df5a24cda367c04f46af4e9f2b07b7f3759b7aa44e35bc74ad6b71903626dd1" not found in pod's containers
Mar 20 04:43:26 kworker2.example.com kubelet[6084]: W0320 04:43:26.044131 6084 cni.go:331] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "6df5a24cda367c04f46af4e9f2b07b7f3759b7aa44e35bc74ad6b71903626dd1"
Besides, I noticed a few seems inconsistent in readme documentation, it's mentioned to use
"kubectl apply -f manifests/meshnet.yml"
it seems the path is incorrect, i loaded this one instead
"kubectl apply -f manifests/base/meshnet.yml"
2nd, the tests/2node.yml, this seems consists more than 2 nodes, i removed the extra r3, ( i tried other topo as well and getting the same error though)
jack@ubuntu:~/meshnet-cni/tests$ cat 2node.yml
---
apiVersion: v1
kind: List
items:
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: r1
spec:
links:
- uid: 1
peer_pod: r2
local_intf: eth1
local_ip: 12.12.12.1/24
peer_intf: eth1
peer_ip: 12.12.12.2/24
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: r2
spec:
links:
- uid: 1
peer_pod: r1
local_intf: eth1
local_ip: 12.12.12.2/24
peer_intf: eth1
peer_ip: 12.12.12.1/24
- apiVersion: v1
kind: Pod
metadata:
name: r1
spec:
containers:
- image: alpine
name: r1
command: ["/bin/sh", "-c", "sleep 2000000000000"]
- apiVersion: v1
kind: Pod
metadata:
name: r2
spec:
containers:
- image: alpine
name: r2
command: ["/bin/sh", "-c", "sleep 2000000000000"]
Overlay- manifests
Hello, if applying the manifests/overlay, will it create vxlan interface? The overaly/e2e is same as base. Would you please explain more details about how to create vxlan by meshnet-cni? Thank you!
Emulate netlink Link Attributes
Currently we can't set the link attributes like queuing which can be used to emulate the link parameters like speed. Enabling these features will aid in creating a more realistic network lab emulation.
Please mention that this only supports x86_64
I wanted to use this on my ARM cluster, spent some 3 hours working through various issues only to discover there's only hardcoded support for x86_64 here. I beg of you, please call this out in your docs or readme or something to save others the trouble.
Add new pod with meshnet
Hi,
Would there be a way with meshnet, to add a new pod with a link , to a node of a topology that was already created (The topology was previously created with KNE), without having to destroy or redeploy the topology?
For example, I am creating this yaml, which uses pod r1 of the topology created in the namespace 2ceos, pod r3 is created, but the links connecting the two pods are not created.
kubectl get pods -n 2ceos
kubectl exec -it r3-0 -n 2ceos -- /bin/sh
Docker image building failed
Thanks for the project. I'm hitting some error while building a new docker image on Ubuntu18.04.
Context: I have a scenario where I need to create a vxlan tunnel with a bare-metal machine. So, peer is not a pod but a machine.
For that I was changing the code and had to build a new image.
Error: While running 'make build', I hit the error
ubuntu@ip-172-31-30-58:~/meshnet-cni$ DOCKER_BUILDKIT=1 docker build -t meshnet -f docker/Dockerfile .
[+] Building 1.7s (21/30)
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 38B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.7s
=> [internal] load metadata for docker.io/library/golang:1.12.7 0.7s
=> [internal] load build context 0.0s
=> => transferring context: 3.25kB 0.0s
=> [stage-3 1/9] FROM docker.io/library/alpine:latest@sha256:c19173c5ada610a5989151111163d28a67368362762534d8a8121ce95cf2bd5a 0.0s
=> [proto_base 1/8] FROM docker.io/library/golang:1.12.7@sha256:f5486a917b57f8b14be4345604bc4654147416a327d6d63271a0c52c907001c4 0.0s
=> CACHED [stage-3 2/9] RUN apk add --no-cache jq 0.0s
=> CACHED [build_base 2/5] WORKDIR /go/src/github.com/tahir24434/meshnet-cni 0.0s
=> CACHED [build_base 3/5] COPY go.mod . 0.0s
=> CACHED [build_base 4/5] COPY go.sum . 0.0s
=> CANCELED [build_base 5/5] RUN go mod download 0.9s
=> CACHED [proto_base 2/8] RUN apt-get update && apt-get -y install curl unzip 0.0s
=> CACHED [proto_base 3/8] RUN curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v3.9.1/protoc-3.9.1-linux-x86_64.zip && unzip protoc-3.9.1-linux-x86_64.zip 0.0s
=> CACHED [proto_base 4/8] RUN go get -u github.com/golang/protobuf/protoc-gen-go 0.0s
=> CACHED [proto_base 5/8] COPY daemon/ daemon/ 0.0s
=> CACHED [proto_base 6/8] COPY Makefile . 0.0s
=> CACHED [proto_base 7/8] COPY .mk/ .mk/ 0.0s
=> CACHED [proto_base 8/8] RUN make proto 0.0s
=> CACHED [build 1/4] COPY . . 0.0s
=> ERROR [build 2/4] COPY --from=proto_base daemon/generated daemon/generated 0.0s
------
> [build 2/4] COPY --from=proto_base daemon/generated daemon/generated:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: failed to walk /var/lib/docker/tmp/buildkit-mount009672704/daemon: lstat /var/lib/docker/tmp/buildkit-mount009672704/daemon: no such file or directory
What are the steps to generate the new docker image holding modified-code/binary.
`make install` fails
Going through the README demo, I get to the point of running make install but it fails to execute
popos@pop-os:~/meshnet-cni$ make install /home/popos/go/bin/kind load docker-image --name "meshnet" networkop/meshnet:4bf3db7-dirty Image: "networkop/meshnet:4bf3db7-dirty" with ID "sha256:8eaa415b1910d98d19dad3a6396c85893e27e40f8842ed6a1c9a47e27962d293" not yet present on node "meshnet-control-plane", loading... Image: "networkop/meshnet:4bf3db7-dirty" with ID "sha256:8eaa415b1910d98d19dad3a6396c85893e27e40f8842ed6a1c9a47e27962d293" not yet present on node "meshnet-worker", loading... Image: "networkop/meshnet:4bf3db7-dirty" with ID "sha256:8eaa415b1910d98d19dad3a6396c85893e27e40f8842ed6a1c9a47e27962d293" not yet present on node "meshnet-worker3", loading... Image: "networkop/meshnet:4bf3db7-dirty" with ID "sha256:8eaa415b1910d98d19dad3a6396c85893e27e40f8842ed6a1c9a47e27962d293" not yet present on node "meshnet-worker2", loading... ERROR: command "docker save -o /tmp/images-tar964290207/images.tar networkop/meshnet:4bf3db7-dirty" failed with error: exit status 1 Command Output: failed to save image: invalid output path: directory "/tmp/images-tar964290207" does not exist make: *** [.mk/kind.mk:33: kind-load] Error 1
I have tried this on clean VMs of both Ubuntu 21.04 and 21.10.
Replace REST API with gRPC
right now meshnet-cni will talk to meshnetd over a simple REST API. this works fine and there are no issues with it. However, ideally, I'd like to replace REST with gRPC with protobuf encoding to make things more...fun?
VXLan existed issues
Hello, I just encountered the vxlan existed issue when creating topo through meshnet. Any suggestions? Thank you!
Warning FailedCreatePodSandBox 5s kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "15ec95fd9c2564055c051b9b1b0f6dade81bb856edecac2aabcb001ebb101ce0" network for pod "vswitch-12": networkPlugin cni failed to set up pod "vswitch-12_default" network: vxlan add failed: Failed to add vxlan koko4039455774: file exists
Q: virtual switching in meshnet
Has anyone ever tried putting a vSwtich in a Pod and connecting it up with meshnet? How did you do it?
If this is something meshnet can't do, why not?
(thinking about new features here)
Help - Dropped packets
I don't know if this is a networking issue in my emulation of a problem with meshnet or a problem with kind.
- Meshnet-CNI on 1-node kind cluster.
- Deployed 3x3 grid topology.
- All links are /31 issued from a /16 with no overlaps.
- All pods running quagga ospfd, eth0 passive, and the /16 set as area 0
When I ping sweep, some nodes are unreachable. I set rp_filter=0 on the pods, the docker kind container, and the host, and while that cleared up some issues it didn't solve all of them.
If I set up a ping to an unreachable host and start dumping, I can see the ping arrive at the destination. The echo reply gets dropped in transit. The node where it's dropped is where equal-weight paths diverge.
E.g., n0 pinging n5. Requests go n0-n1-n2-n5, replies go n5-n4-n1-n0. If I tcpdump on n1, I can see the reply arrive but it's silently dropped. iface drop stat is 0. log_martians is on but nothing in the logs.
- OSPF routes are present
ip routeaggrees w/ ospfd- Take the inbound iface down on n1, and pings arrive at n0
- Take the same iface back up, pings continue to arrive at n0 for a bit, then stop.
I think this is just me not gr0kking OSPF, but I'm out of ideas.
Cluster data plane fails after initial deploy
Conditions:
- New kind cluster with kindnet
- meshnet-cni @v0.3.0 installed
Intermittently, Pods deployed immediately after meshnet come up with the cluster network unavailable. E.g., kube-prometheus-stack initializes with a Job, but it fails to talk to the API server:
> kubectl -n mimesis-data logs mimesis-mon-mimesis-data-admission-create-cslj5
W0829 17:07:15.999396 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
{"err":"Get \"https://10.96.0.1:443/api/v1/namespaces/mimesis-data/secrets/mimesis-mon-mimesis-data-admission\": dial tcp 10.96.0.1:443: connect: no route to host","level":"fatal","msg":"error getting secret","source":"k8s/k8s.go:109","time":"2021-08-29T17:07:19Z"}
When this condition occurs, it happens with all Pods. I can exec into a Pod and try to ping cluster-cidr addresses and all return no route to host.
I can sometimes kick networking over by generating some external network traffic (e.g., apt-get update from the kindnet pod).
Enhancement Request: Support for Physical device peer
Request
Currently Meshnet assume that all devices in topology are virtual and running inside K8s pods.
In real world, you may have a topology where some of the devices are physical while some other running as virtual devices in K8s.
Summary
With physical device support, user will be able to define a device in topology file for which a pod will not be created. Nothing else will be changed, all other objects like CustomeResource will be created as before.
The configuration of this device will be delegated to the operator. Meshnet will not be responsible for its configuration as there are many possible combination of vendor/series and way of configurations.
Motivation
- Devices running outside of the pods can become part of the topology.
- Any device for which virtual image is not readily available can be tested.
- Before bringing up a site, one can bring up a virtual site and test it with existing physical site.
Implementation Details
Changes are needed in both k8s-topo and meshnet.
K8s-topo changes
Links between virtual devices will be bring up as before. Changes will be needed only if one of the endpoint in link is physical.
- Introduce 'phy' keyword to represent a physical device.
endpoints: ["vmx-1:eth2:10.40.5.2/24", "phy-1:eth1:172.31.26.56/24"]
phy-1 will be a physical device. vmx-1 is virtual device. - No pod will be created for the 'phy' peer.
- vxlan link will be created on the virtual device with 'phy' ip as remote address.
Meshnet changes
- Just like in macvlan case, see whether peer device has 'phy' prefix? If it does, then just create a vxlan link on virtual device with 'phy' ip as remote and continue.
Debug commands
hi, I have created the connections between 2 PODS within same hosts and across hosts as well and ping is working as expected. However would like see the VXLAN tunnel interfaces and VNI's used to communicate across hosts for my understanding. Tried "ip a" and brctl commands in the host but nothing showing anything. Could you please share the steps to see the veth and vxlan interfaces created by the meshnet CNI?
Specifying MAC, MTU and other link attrs like IFLA_AF_SPEC
Can you please help me with an example to specify the MAC, MTU and other link Attributes of the veth link? I don;t see it currently supported, but just checking if there is some way to do this.
Based on some code browsing , This is what Koko is using internally, and may need to be extended
https://github.com/vishvananda/netlink/blob/main/link.go#L24
Add reconciliation for grpc-wires
When a grpc-wire enabled meshnet pod in a node restarts (due to OOM / Error, etc.) the grpc-wire info (wire/handler maps) is not persisted or reconciled on restart.
meshnet-cni/daemon/grpcwire/grpcwire.go
Line 143 in d3ae648
This leads to errors like the following:
SendToOnce (wire id - 77): Could not find local handle. err:interface 77 is not active
stemming from:
meshnet-cni/daemon/grpcwire/grpcwire.go
Line 254 in d3ae648
To make grpc-wire add on more resilient, reconciliation should be added (likely using the topology CRD)
Meshnet + init wait container getting stuck after pod deletion + recreation
We have a KNE (https://github.com/openconfig/kne) user seeing errors with init-wait container hanging waiting for interfaces after pod deletion + recreation using the command kubectl get pod -n <namespace> <container> -o yaml | kubectl replace --force -f -.
@robshakir to add more details on specific log messages
IP confilcts with k8s coredns pod
Problem:
I'm using the meshnet-cni with flannel as base network. After installing flannel and meshnet-cni, I found the IP allocated to new pods would confilct with the IP of k8s system pods coredns. In each k8s cluster, there are two coredns pod, and their IP are allocated by flannel, usually with a suffix of 2 or 3. When starting to deploy new pods, the pod IP also are allocated from 2, which cause IP conficts and then the coredns pod would fail for liveness probe and restarted. At the same time, the user created pod also does not work properly. I'm not sure if this the problem due to CNI chainning. Please let me know if you know how to solve it. Thanks!
Reproduce the bug:
- Create a k8s cluster with kubeadm, pod cidr is 10.244.0.0/16
- Deploy the flannel cni
- Deploy the meshnet-cni
- Deploy new pods and check the IP, it would conflict with IP of coredns (not every time, but the IP is not excluded to be used).
Question - Whether meshnet-cni recognizes pods created as a deployment/stateful set?
I have been using the examples from k8s-topo to deploy a topology via meshnet-cni, but the example yamls use the Kind: Pod for deploying the pods. Since the pods are not deployed as a Deployment/Stateful set, they disappear randomly breaking the topology. Will meshnet create the p2p links If I deploy the pods as Deployment/Stateful Set?
Error while adding MacVlan/VxLAN Interface
I keep getting an error while creating a POD and using the meshnet cni.
Warning FailedCreatePodSandBox 21m kubelet, k8s-agentpool1-40367033-vmss000001 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2d6c4d562a8f7438450ebb390d56bf0617e18371485f54c99c1a02422a62be13" network for pod "testdevice02": NetworkPlugin cni failed to set up pod "testdevice02_default" network: Cannot add IPaddr/netns failed: failed to rename link koko2610346067 -> eth0: file exists
This error seems to be coming from the koko library in function "SetVethLink". I see this function getting invoked at three places in the library in "MakeVxLan", "MakeVLan" & "MakeMacVlan".
In one of the functions they call.
if err = veth1.SetVethLink(link); err != nil {
netlink.LinkDel(link)
return fmt.Errorf("Cannot add IPaddr/netns failed: %v", err)
}
but at other places I don't see the highlighted code.
I notice this error while I delete a topology and try to re-deploy the same again. I guess something is getting messed up during deleting a link and it ends up remaining stale.
On kubernetes 1.24.0-rc0+k3s1, CNI v1.0.0 complains about meshnet version.
Haven't dug in, but:
- kubelet complains thusly:
13m Warning FailedCreatePodSandBox pod/mimesis-grafana-565fc7d86b-x4lgc (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9268d95fc10da9222135c3d889ed602708463680572730e05415213e48e706c3": plugin type="meshnet" name="meshnet" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]
- meshnet plugin complains thusly:
time="2022-05-12T16:45:02Z" level=error msg="failed to run meshnet cni: <nil>"
- Applicable code appears to be here:
Lines 432 to 436 in de89b2e
Workaround by editing /etc/cni/00-meshnet.conf and setting cniVersion to 0.4.0, but that's fragile on a reinstall.
core DNS goes to Error state after master node restart
I am facing an issue while using meshnet cni. I am running my cluster in a single linux VM. After restarting the node the core DNS goes to error state. I have to remove the config under /etc/cni/net.d/00-meshnet and delete the meshnet cni yml to make it come to running state again. I don't see any specific errors in pod describe or logs. I guess the core DNS is not able to find a CNI and is stuck. We have the delegate statement in the conf to use flannel but for some reason core DNS is ignoring that.
Attached the logs.
vparames86@UbuntuBionic:~$ journalctl -l -u kubelet -n 100
-- Logs begin at Sat 2019-07-06 04:59:33 UTC, end at Sat 2019-07-06 21:05:49 UTC. --
Jul 06 21:05:33 UbuntuBionic kubelet[1269]: E0706 21:05:33.542137 1269 kuberuntime_manager.go:841] Failed to stop sandbox {"docker" "0445295849a36ccdf41f2b0fa507858c9cf1898e7fbd29159e39c4bc2ba89bd8"}
Jul 06 21:05:33 UbuntuBionic kubelet[1269]: W0706 21:05:33.544430 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "2850d872054
Jul 06 21:05:33 UbuntuBionic kubelet[1269]: xxx| etcd1 |===> 2019/07/06 21:05:33 Processing Del for POD etcd1
Jul 06 21:05:34 UbuntuBionic kubelet[1269]: W0706 21:05:34.533182 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "40cac378fff
Jul 06 21:05:34 UbuntuBionic kubelet[1269]: W0706 21:05:34.533182 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "0b3601b16bb
Jul 06 21:05:34 UbuntuBionic kubelet[1269]: xxx| coredns-5c98db65d4-5mtqr |===> 2019/07/06 21:05:34 Processing Del for POD coredns-5c98db65d4-5mtqr
Jul 06 21:05:34 UbuntuBionic kubelet[1269]: xxx| qrtr-3 |===> 2019/07/06 21:05:34 Processing Del for POD qrtr-3
Jul 06 21:05:34 UbuntuBionic kubelet[1269]: E0706 21:05:34.549119 1269 cni.go:352] Error deleting default_qrtr-4/1837e6fadeca3cdbcb8fcd4f47e91d3caa084a87b0dcb80fb5f8d0d6462daf56 from network meshnet/me
Jul 06 21:05:34 UbuntuBionic kubelet[1269]: E0706 21:05:34.549787 1269 remote_runtime.go:128] StopPodSandbox "1837e6fadeca3cdbcb8fcd4f47e91d3caa084a87b0dcb80fb5f8d0d6462daf56" from runtime service fail
Jul 06 21:05:34 UbuntuBionic kubelet[1269]: E0706 21:05:34.549832 1269 kuberuntime_manager.go:841] Failed to stop sandbox {"docker" "1837e6fadeca3cdbcb8fcd4f47e91d3caa084a87b0dcb80fb5f8d0d6462daf56"}
Jul 06 21:05:34 UbuntuBionic kubelet[1269]: E0706 21:05:34.549886 1269 kuberuntime_manager.go:636] killPodWithSyncResult failed: failed to "KillPodSandbox" for "efc9e914-e6de-40b9-bbdb-1568dae34283" wi
Jul 06 21:05:34 UbuntuBionic kubelet[1269]: E0706 21:05:34.549911 1269 pod_workers.go:190] Error syncing pod efc9e914-e6de-40b9-bbdb-1568dae34283 ("qrtr-4_default(efc9e914-e6de-40b9-bbdb-1568dae34283)"
Jul 06 21:05:36 UbuntuBionic kubelet[1269]: W0706 21:05:36.533649 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "0b6350f0c3a
Jul 06 21:05:36 UbuntuBionic kubelet[1269]: W0706 21:05:36.533647 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "b6b515b648d
Jul 06 21:05:36 UbuntuBionic kubelet[1269]: xxx| etcd2 |===> 2019/07/06 21:05:36 Processing Del for POD etcd2
Jul 06 21:05:36 UbuntuBionic kubelet[1269]: xxx| internal-docker-registry-8df88bdd9-rwsxn |===> 2019/07/06 21:05:36 Processing Del for POD internal-docker-registry-8df88bdd9-rwsxn
Jul 06 21:05:36 UbuntuBionic kubelet[1269]: E0706 21:05:36.549682 1269 cni.go:352] Error deleting default_qrtr-1/b206128cf2355e3833b80141fdcf1d87aaccf3b574786f1883fe7f5dedc8364d from network meshnet/me
Jul 06 21:05:36 UbuntuBionic kubelet[1269]: E0706 21:05:36.550566 1269 remote_runtime.go:128] StopPodSandbox "b206128cf2355e3833b80141fdcf1d87aaccf3b574786f1883fe7f5dedc8364d" from runtime service fail
Jul 06 21:05:36 UbuntuBionic kubelet[1269]: E0706 21:05:36.550623 1269 kuberuntime_manager.go:841] Failed to stop sandbox {"docker" "b206128cf2355e3833b80141fdcf1d87aaccf3b574786f1883fe7f5dedc8364d"}
Jul 06 21:05:36 UbuntuBionic kubelet[1269]: E0706 21:05:36.550692 1269 kuberuntime_manager.go:636] killPodWithSyncResult failed: failed to "KillPodSandbox" for "6817cdac-dfb1-43af-bb61-4b793e2ea0e1" wi
Jul 06 21:05:36 UbuntuBionic kubelet[1269]: E0706 21:05:36.550714 1269 pod_workers.go:190] Error syncing pod 6817cdac-dfb1-43af-bb61-4b793e2ea0e1 ("qrtr-1_default(6817cdac-dfb1-43af-bb61-4b793e2ea0e1)"
Jul 06 21:05:37 UbuntuBionic kubelet[1269]: W0706 21:05:37.533058 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "f5469e027c3
Jul 06 21:05:37 UbuntuBionic kubelet[1269]: xxx| qrtr-5 |===> 2019/07/06 21:05:37 Processing Del for POD qrtr-5
Jul 06 21:05:37 UbuntuBionic kubelet[1269]: E0706 21:05:37.552226 1269 cni.go:352] Error deleting default_etcd1/2850d872054bc1c1734c3b3c44cef0ec23b43013bd98d0451e96829e442c7521 from network meshnet/mes
Jul 06 21:05:37 UbuntuBionic kubelet[1269]: E0706 21:05:37.552902 1269 remote_runtime.go:128] StopPodSandbox "2850d872054bc1c1734c3b3c44cef0ec23b43013bd98d0451e96829e442c7521" from runtime service fail
Jul 06 21:05:37 UbuntuBionic kubelet[1269]: E0706 21:05:37.552954 1269 kuberuntime_manager.go:841] Failed to stop sandbox {"docker" "2850d872054bc1c1734c3b3c44cef0ec23b43013bd98d0451e96829e442c7521"}
Jul 06 21:05:37 UbuntuBionic kubelet[1269]: E0706 21:05:37.553028 1269 kuberuntime_manager.go:636] killPodWithSyncResult failed: failed to "KillPodSandbox" for "159436c1-25af-4bef-9509-70e138ebe018" wi
Jul 06 21:05:37 UbuntuBionic kubelet[1269]: E0706 21:05:37.553060 1269 pod_workers.go:190] Error syncing pod 159436c1-25af-4bef-9509-70e138ebe018 ("etcd1_default(159436c1-25af-4bef-9509-70e138ebe018)")
Jul 06 21:05:38 UbuntuBionic kubelet[1269]: E0706 21:05:38.543511 1269 cni.go:352] Error deleting kube-system_coredns-5c98db65d4-5mtqr/40cac378fff9b9b2baa93ebf8c036a1d0955932df180588ba57601739d0fd4c9 f
Jul 06 21:05:38 UbuntuBionic kubelet[1269]: E0706 21:05:38.545913 1269 remote_runtime.go:128] StopPodSandbox "40cac378fff9b9b2baa93ebf8c036a1d0955932df180588ba57601739d0fd4c9" from runtime service fail
Jul 06 21:05:38 UbuntuBionic kubelet[1269]: E0706 21:05:38.545963 1269 kuberuntime_manager.go:841] Failed to stop sandbox {"docker" "40cac378fff9b9b2baa93ebf8c036a1d0955932df180588ba57601739d0fd4c9"}
Jul 06 21:05:38 UbuntuBionic kubelet[1269]: W0706 21:05:38.548362 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "82b65dc0b65
Jul 06 21:05:38 UbuntuBionic kubelet[1269]: E0706 21:05:38.549936 1269 cni.go:352] Error deleting default_qrtr-3/0b3601b16bb1da68ee7dc8b3f98c9ea26b7b264b2903321fd5ead6a17bea3b96 from network meshnet/me
Jul 06 21:05:38 UbuntuBionic kubelet[1269]: E0706 21:05:38.550530 1269 remote_runtime.go:128] StopPodSandbox "0b3601b16bb1da68ee7dc8b3f98c9ea26b7b264b2903321fd5ead6a17bea3b96" from runtime service fail
Jul 06 21:05:38 UbuntuBionic kubelet[1269]: E0706 21:05:38.550572 1269 kuberuntime_manager.go:841] Failed to stop sandbox {"docker" "0b3601b16bb1da68ee7dc8b3f98c9ea26b7b264b2903321fd5ead6a17bea3b96"}
Jul 06 21:05:38 UbuntuBionic kubelet[1269]: E0706 21:05:38.550701 1269 kuberuntime_manager.go:636] killPodWithSyncResult failed: failed to "KillPodSandbox" for "e3262060-b8dc-496c-8435-8da15b677684" wi
Jul 06 21:05:38 UbuntuBionic kubelet[1269]: E0706 21:05:38.550733 1269 pod_workers.go:190] Error syncing pod e3262060-b8dc-496c-8435-8da15b677684 ("qrtr-3_default(e3262060-b8dc-496c-8435-8da15b677684)"
Jul 06 21:05:38 UbuntuBionic kubelet[1269]: xxx| coredns-5c98db65d4-5mtqr |===> 2019/07/06 21:05:38 Processing Del for POD coredns-5c98db65d4-5mtqr
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: W0706 21:05:40.533114 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "84a1a67acdb
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: xxx| etcd0 |===> 2019/07/06 21:05:40 Processing Del for POD etcd0
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: E0706 21:05:40.543101 1269 cni.go:352] Error deleting default_etcd2/0b6350f0c3acd929ea1b6a95eddfb24358b25d33a200db1b742e8a8d82be81a1 from network meshnet/mes
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: E0706 21:05:40.543700 1269 remote_runtime.go:128] StopPodSandbox "0b6350f0c3acd929ea1b6a95eddfb24358b25d33a200db1b742e8a8d82be81a1" from runtime service fail
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: E0706 21:05:40.543741 1269 kuberuntime_manager.go:841] Failed to stop sandbox {"docker" "0b6350f0c3acd929ea1b6a95eddfb24358b25d33a200db1b742e8a8d82be81a1"}
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: W0706 21:05:40.545533 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "6a19063eeec
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: E0706 21:05:40.547020 1269 cni.go:352] Error deleting default_internal-docker-registry-8df88bdd9-rwsxn/b6b515b648db344e493db673da53da7619b99b42b55d84662b505c
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: E0706 21:05:40.547768 1269 remote_runtime.go:128] StopPodSandbox "b6b515b648db344e493db673da53da7619b99b42b55d84662b505c728ff42ff2" from runtime service fail
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: E0706 21:05:40.547803 1269 kuberuntime_manager.go:841] Failed to stop sandbox {"docker" "b6b515b648db344e493db673da53da7619b99b42b55d84662b505c728ff42ff2"}
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: W0706 21:05:40.550593 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "9cdf59d9555
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: xxx| etcd2 |===> 2019/07/06 21:05:40 Processing Del for POD etcd2
Jul 06 21:05:40 UbuntuBionic kubelet[1269]: xxx| internal-docker-registry-8df88bdd9-rwsxn |===> 2019/07/06 21:05:40 Processing Del for POD internal-docker-registry-8df88bdd9-rwsxn
Jul 06 21:05:41 UbuntuBionic kubelet[1269]: E0706 21:05:41.541071 1269 cni.go:352] Error deleting default_qrtr-5/f5469e027c389390a1aee55ef90a11be27ef12d14fbcbed9a30c2d80f78b2515 from network meshnet/me
Jul 06 21:05:41 UbuntuBionic kubelet[1269]: E0706 21:05:41.542058 1269 remote_runtime.go:128] StopPodSandbox "f5469e027c389390a1aee55ef90a11be27ef12d14fbcbed9a30c2d80f78b2515" from runtime service fail
Jul 06 21:05:41 UbuntuBionic kubelet[1269]: E0706 21:05:41.542113 1269 kuberuntime_manager.go:841] Failed to stop sandbox {"docker" "f5469e027c389390a1aee55ef90a11be27ef12d14fbcbed9a30c2d80f78b2515"}
Jul 06 21:05:41 UbuntuBionic kubelet[1269]: E0706 21:05:41.542197 1269 kuberuntime_manager.go:636] killPodWithSyncResult failed: failed to "KillPodSandbox" for "14296cb2-be2e-40f9-a170-4ec2afbf02fb" wi
Jul 06 21:05:41 UbuntuBionic kubelet[1269]: E0706 21:05:41.542229 1269 pod_workers.go:190] Error syncing pod 14296cb2-be2e-40f9-a170-4ec2afbf02fb ("qrtr-5_default(14296cb2-be2e-40f9-a170-4ec2afbf02fb)"
Jul 06 21:05:42 UbuntuBionic kubelet[1269]: E0706 21:05:42.556580 1269 cni.go:352] Error deleting kube-system_coredns-5c98db65d4-5mtqr/82b65dc0b65ca62eccb96061eb6f6016f0a6d5dcaa251f80864b018b4c50da86 f
Jul 06 21:05:42 UbuntuBionic kubelet[1269]: E0706 21:05:42.557242 1269 remote_runtime.go:128] StopPodSandbox "82b65dc0b65ca62eccb96061eb6f6016f0a6d5dcaa251f80864b018b4c50da86" from runtime service fail
Jul 06 21:05:42 UbuntuBionic kubelet[1269]: E0706 21:05:42.557360 1269 kuberuntime_manager.go:841] Failed to stop sandbox {"docker" "82b65dc0b65ca62eccb96061eb6f6016f0a6d5dcaa251f80864b018b4c50da86"}
Jul 06 21:05:42 UbuntuBionic kubelet[1269]: E0706 21:05:42.557438 1269 kuberuntime_manager.go:636] killPodWithSyncResult failed: failed to "KillPodSandbox" for "a1af1182-4082-43ac-8c50-47e4008b80c2" wi
Jul 06 21:05:42 UbuntuBionic kubelet[1269]: E0706 21:05:42.557478 1269 pod_workers.go:190] Error syncing pod a1af1182-4082-43ac-8c50-47e4008b80c2 ("coredns-5c98db65d4-5mtqr_kube-system(a1af1182-4082-43
Jul 06 21:05:44 UbuntuBionic kubelet[1269]: W0706 21:05:44.533229 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "1f2a871eb93
Jul 06 21:05:44 UbuntuBionic kubelet[1269]: E0706 21:05:44.541883 1269 cni.go:352] Error deleting default_etcd0/84a1a67acdb87980d16b05a9daa4c22fbf8dafea5b4db85a57487ae224843824 from network meshnet/mes
Jul 06 21:05:44 UbuntuBionic kubelet[1269]: xxx| qrtr-2 |===> 2019/07/06 21:05:44 Processing Del for POD qrtr-2
Jul 06 21:05:44 UbuntuBionic kubelet[1269]: E0706 21:05:44.543065 1269 remote_runtime.go:128] StopPodSandbox "84a1a67acdb87980d16b05a9daa4c22fbf8dafea5b4db85a57487ae224843824" from runtime service fail
Jul 06 21:05:44 UbuntuBionic kubelet[1269]: E0706 21:05:44.543108 1269 kuberuntime_manager.go:841] Failed to stop sandbox {"docker" "84a1a67acdb87980d16b05a9daa4c22fbf8dafea5b4db85a57487ae224843824"}
Jul 06 21:05:44 UbuntuBionic kubelet[1269]: W0706 21:05:44.545217 1269 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "8e5885ec7cf
Jul 06 21:05:44 UbuntuBionic kubelet[1269]: xxx| etcd0 |===> 2019/07/06 21:05:44 Processing Del for POD etcd0
vparames86@UbuntuBionic:~$ kubectl logs coredns-5c98db65d4-79b6h --namespace kube-system
.:53
2019-07-05T23:15:33.957Z [INFO] CoreDNS-1.3.1
2019-07-05T23:15:33.957Z [INFO] linux/amd64, go1.11.4, 6b56a9c
CoreDNS-1.3.1
linux/amd64, go1.11.4, 6b56a9c
2019-07-05T23:15:33.957Z [INFO] plugin/reload: Running configuration MD5 = 5d5369fbc12f985709b924e721217843
[INFO] SIGTERM: Shutting down servers then terminating
vparames86@UbuntuBionic:~$ kubectl describe pod coredns-5c98db65d4-79b6h --namespace kube-system
Name: coredns-5c98db65d4-79b6h
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: ubuntubionic/10.0.0.4
Start Time: Wed, 03 Jul 2019 23:19:08 +0000
Labels: k8s-app=kube-dns
pod-template-hash=5c98db65d4
Annotations:
Status: Running
IP:
Controlled By: ReplicaSet/coredns-5c98db65d4
Containers:
coredns:
Container ID: docker://b9847e634ecb4bfb353d89ee40ed8572b702acd94f2110b658e4b2ccfe2f8d25
Image: k8s.gcr.io/coredns:1.3.1
Image ID: docker-pullable://k8s.gcr.io/coredns@sha256:02382353821b12c21b062c59184e227e001079bb13ebd01f9d3270ba0fcbf1e4
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Terminated
Reason: Error
Exit Code: 255
Started: Fri, 05 Jul 2019 23:15:33 +0000
Finished: Sat, 06 Jul 2019 03:08:21 +0000
Ready: False
Restart Count: 3
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
Mounts:
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-8d5cc (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
coredns-token-8d5cc:
Type: Secret (a volume populated by a Secret)
SecretName: coredns-token-8d5cc
Optional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
Normal SandboxChanged 98m (x3369 over 17h) kubelet, ubuntubionic Pod sandbox changed, it will be killed and re-created.
Normal SandboxChanged 18s (x20 over 6m56s) kubelet, ubuntubionic Pod sandbox changed, it will be killed and re-created.
vparames86@UbuntuBionic:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default etcd0 0/1 Error 3 2d22h
default etcd1 0/1 Error 3 2d22h
default etcd2 0/1 Error 3 2d22h
default internal-docker-registry-8df88bdd9-rwsxn 0/1 Error 0 22h
default k8s-topo 1/1 Running 2 21h
default qrtr-1 0/1 Init:Error 0 20h
default qrtr-2 0/1 Init:Error 0 20h
default qrtr-3 0/1 Init:Error 0 20h
default qrtr-4 0/1 Init:Error 0 20h
default qrtr-5 0/1 Init:Error 0 20h
kube-system coredns-5c98db65d4-5mtqr 0/1 Error 3 2d21h
kube-system coredns-5c98db65d4-79b6h 0/1 Error 3 2d21h
kube-system etcd-ubuntubionic 1/1 Running 7 3d2h
kube-system kube-apiserver-ubuntubionic 1/1 Running 7 3d2h
kube-system kube-controller-manager-ubuntubionic 1/1 Running 10 3d2h
kube-system kube-flannel-ds-amd64-z2vvm 1/1 Running 5 2d21h
kube-system kube-meshnet-wrz8n 1/1 Running 3 25h
kube-system kube-proxy-vw8v9 1/1 Running 7 3d2h
kube-system kube-scheduler-ubuntubionic 1/1 Running 10 3d2h
Release workflow failed after merging changes for gRPC link
Connect pods with IPv4/IPv6 address
Hello, I want to set both ipv4 and ipv6 addresses for the pod, I try the following configuration :
# router-2
- uid: 2237123
peer_pod: router-3
local_intf: eth6
peer_intf: eth6
local_ip: 172.30.6.1/24
peer_ip: 172.30.6.2/24
- uid: 2237123
peer_pod: router-3
local_intf: eth6
peer_intf: eth6
local_ip: 2001:DB8::1/64
peer_ip: 2001:DB8::2/64Then I found that only the ipv4 address was successfully configured :
# router-2
eth6 Link encap:Ethernet HWaddr 1A:28:65:AF:B3:C2
inet addr:172.30.6.1 Bcast:172.30.6.255 Mask:255.255.255.0
inet6 addr: fe80::1828:65ff:feaf:b3c2/64 Scope:LinkI wonder how to configure ipv4 and ipv6 addresses at the same time, maybe like this :
- uid: 2237123
peer_pod: router-3
local_intf: eth6
peer_intf: eth6
local_ip: 172.30.6.1/24,2001:DB8::1/64
peer_ip: 172.30.6.2/24,2001:DB8::2/64Ablity to choose a specific NIC for VXLAN bind
Problem:
Way to define specific port on the compute to carry the VXLAN traffic.
Issue:
Currently, I am trying to achieve the MTU of vxlan bind atleast 9232.
By the way it seems the MTU currently is being picked from the underlying interface and its being tied to the primary address as defined here.
https://github.com/networkop/meshnet-cni/blob/master/manifests/base/daemonset.yaml#L41
We want to use a specific NIC for carrying the vxlan binds. and i m not finding a way to define the endpoint IP to be picked up from a specific interface on the computes.
Any idea how i can point to a specific underlay interface for the VXLAN binds to use ?
Once i have the chosen interface(which supports larger MTU) i could have a large MTU on that and probably proceed.
Also this allows us to control entire meshnet traffic on a specific interface rather than on the k8s control/mgmt port.
Problem while creating network interfaces
Hi, I'm currently using your plugin into my project but I'm having issues with interfaces creation.
In particular, sometimes there are situations where a particular eth can have more peers than just one. For example, look at this simple network:
And take r1 for example. It has 2 peers on eth0 with dns1 and user, and 2 peers on eth1 with r2 and r3. This is my json file for this device (with eth numbers increased by one ofc):
[
{
"local_intf": "eth1",
"local_ip": "10.0.0.1/24",
"peer_intf": "eth1",
"peer_ip": "10.0.0.2/24",
"peer_pod": "user",
"uuid": 916
},
{
"local_intf": "eth1",
"local_ip": "10.0.0.1/24",
"peer_intf": "eth1",
"peer_ip": "10.0.0.3/24",
"peer_pod": "dns1",
"uuid": 246
},
{
"local_intf": "eth2",
"local_ip": "1.0.0.1/16",
"peer_intf": "eth2",
"peer_ip": "1.0.0.2/16",
"peer_pod": "r2",
"uuid": 182
},
{
"local_intf": "eth2",
"local_ip": "1.0.0.1/16",
"peer_intf": "eth2",
"peer_ip": "1.0.0.3/16",
"peer_pod": "r3",
"uuid": 318
}
]Sometimes i get this error during pod creation:
Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "367e71d2ab6d4c9df4fd053477506a5da3a0a2afc1a4fa240ace3a425da1c4e5" network for pod "r1": NetworkPlugin cni failed to set up pod "r1_default" network: failed to rename link kokoa34a2299 -> eth1: file exists
Or sometimes the pod runs but with non-deterministic interfaces created:
For example, eth1 is missing in the image.
Is it possibile to fix this issue? Also, can it be an "uuid" problem? What does it mean?
Low performance using meshnet-cni
Hi,
I am currently utilizing Meshnet CNI for network topology creation. However, I've encountered low performance among the pods within the scenario using Meshnet. When measuring performance over the Flannel network connecting these same pods, the performance is significantly better. I'm reaching out to understand if there might be specific reasons behind this. Would you know why this might be happening or any ideas on how I could fix it?.
-
Test with meshnet-cni:
-
Test with network flannel:
Flannel as default CNI: 00-meshnet conf file has empty delegate
Plugin does not create the right meshnet.conf file on worker and master nodes on ec2 based cluster with Flannel CNI (deployed using Kops).
It generates below file
{
"cniVersion": "0.2.0",
"name": "meshnet_network",
"type": "meshnet",
"delegate": {}
}
Expected one is something like below
{
"cniVersion": "0.2.0",
"name": "meshnet_network",
"type": "meshnet",
"delegate": {
"name": "cbr0",
"type": "flannel",
"forceAddress": true,
"isDefaultGateway": true,
"hairpinMode": true
}
}
Meshnet fails to set up when the node-internal network is not on the default route
Conditions: A k8s cluster where nodes are on a private network routed over a different interface than the default route.
Expectation: getVxLanSource() should discover the interface and IP that's guaranteed to be the node-internal network.
Actual: The IP for the interface carrying the default route is used instead.
Culprit code is here:
Line 81 in 4bf3db7
Reproduction: I found this trying to spin up meshnet on a multi-node minikube cluster. The VBox driver on macOS creates a host-only network for the cluster nodes and attaches it to eth1, using eth0 for NAT networking (which also gets the default route). Further, all the nodes in the cluster get the same IP on eth0. As a result, each pod gets the same srcIP, fails the peer pod IsAlive check, and does a skip. Lather, rinse, repeat.
Even if the nodes had distinct addresses, nodes are not reachable over the NAT network, so the vxlan link would fail to carry traffic (though setup might complete).
Suggested fix: IMHO, the best answer would be to get the node's srcIP off the Node resource status.addresses[] array InternalIP entry(s). At least one InternalIP address must be present, so when multiple are present just take the first one. The convention is that all addresses of InternalIP type are reachable by all nodes in the cluster, but this isn't guaranteed. However, where it isn't the case (e.g., a large multi-tenant cluster with separate worker pools) I would also expect to be using taints and affinities, so all the nodes with meshnet Pods scheduled will all be in the same pool and thus can talk to each other.
References: https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/node-v1/#NodeStatus
Failure to AddWireRemote in large topology in multi-node cluster using grpc-wire
I am having some issues when using meshnet with grpc-wire.
I have a >300 node topology (running in KNE) spread across a 5 node k8 cluster (4-workers).
Many of the router pods get stuck in Init:0/1 state and some of the meshnet pods get stuck in a crash loop:
$ kubectl get pods -n meshnet -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
meshnet-6gzft 1/1 Running 0 4d9h 10.240.0.11 controller <none> <none>
meshnet-6vqtf 1/1 Running 0 4d9h 10.240.0.21 worker-1 <none> <none>
meshnet-849jl 1/1 Running 279 (3d2h ago) 4d9h 10.240.0.23 worker-3 <none> <none>
meshnet-g7swq 0/1 CrashLoopBackOff 1141 (108s ago) 4d9h 10.240.0.24 worker-4 <none> <none>
meshnet-pwr6k 0/1 CrashLoopBackOff 1176 (4m56s ago) 4d9h 10.240.0.22 worker-2 <none> <none>
$ kubectl logs meshnet-g7swq -n meshnet
Distributing files
Starting meshnetd daemon
time="2022-12-13T02:24:29Z" level=info msg="Trying in-cluster configuration"
time="2022-12-13T02:24:29Z" level=info msg="[core] [Server #1] Server created" system=system
time="2022-12-13T02:24:29Z" level=info msg="Starting meshnet daemon...with grpc support"
time="2022-12-13T02:24:29Z" level=info msg="GRPC server has started on port: 51111"
time="2022-12-13T02:24:29Z" level=info msg="[core] [Server #1 ListenSocket #2] ListenSocket created" system=system
time="2022-12-13T02:25:03Z" level=info msg="Error creating vEth pair (in:eth36 <--> out:bx04reth36-0001). Error-> failed to rename link koko3901300170 -> eth36: file exists"
time="2022-12-13T02:25:03Z" level=error msg="AddWireRemote err : failed to rename link koko3901300170 -> eth36: file exists"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x140d56a]
goroutine 98 [running]:
github.com/networkop/meshnet-cni/daemon/meshnet.(*Meshnet).AddGRPCWireRemote(0x0?, {0x0?, 0x0?}, 0x0?)
/go/src/github.com/networkop/meshnet-cni/daemon/meshnet/handler.go:348 +0xaa
...
I see that the failed pod:eth (bx04r:eth36) is reverse-skipped by the peer pod on that link (bx03s:eth1): Reverse-skipping of pod bx04rno10 by pod bx03sfo03
excerpt from KNE topology:
links: {
a_node: "bx03sfo03"
a_int: "eth1"
z_node: "bx04rno10"
z_int: "eth36"
}
However I do not see an entry in the ip link table for bx03s* even though bx03 is stuck in Init state on a worker with a healthy meshnet pod.
On another note, even the running pods report strange errors (which seem like simple no-op bugs upon inspection):
$ kubectl logs meshnet-6vqtf -n meshnet | grep bx07iad60 | head -100
time="2022-12-08T16:46:47Z" level=info msg="Retrieving bx07iad60's metadata from K8s..."
time="2022-12-08T16:46:47Z" level=info msg="Reading pod bx07iad60 from K8s"
time="2022-12-08T16:46:47Z" level=info msg="Skipping of pod bx07iad60 by pod bx09iad60"
time="2022-12-08T17:04:59Z" level=info msg="Retrieving bx07iad60's metadata from K8s..."
time="2022-12-08T17:04:59Z" level=info msg="Reading pod bx07iad60 from K8s"
time="2022-12-08T17:04:59Z" level=info msg="Setting bx07iad60's SrcIp=10.240.0.21 and NetNs=/proc/11867/ns/net"
time="2022-12-08T17:04:59Z" level=info msg="Reading pod bx07iad60 from K8s"
time="2022-12-08T17:05:00Z" level=info msg="Update pod status bx07iad60 from K8s"
time="2022-12-08T17:05:01Z" level=info msg="Checking if bx06iad23 is skipped by bx07iad60"
time="2022-12-08T17:05:03Z" level=info msg="Checking if bx09iad60 is skipped by bx07iad60"
time="2022-12-08T17:05:04Z" level=info msg="Checking if bx06iad30 is skipped by bx07iad60"
time="2022-12-08T17:05:05Z" level=info msg="Removing grpc-wire for pod arista-kne-topo:bx07iad60"
time="2022-12-08T17:05:06Z" level=info msg="Removed all grpc-wire for pod: eth6@bx07iad60"
time="2022-12-08T17:05:06Z" level=error msg="finished unary call with code Unknown" error="failed to remove gRPC wire for pod bx07iad60: %!w(<nil>)" grpc.code=Unknown grpc.method=RemGRPCWire grpc.service=meshnet.v1beta1.Local grpc.start_time="2022-12-08T17:05:05Z" grpc.time_ms=94.334 peer.address="127.0.0.1:40130" span.kind=server system=grpc
In the worker nodes with the failed meshnet pods I see many koko links waiting to be renamed:
worker-2:~$ ip -d link show | grep koko | head
1024: koko2916450998@if1025: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
3073: koko2154029822@if3074: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
2817: koko3259706345@if2818: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
2561: koko832348741@if2562: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
2305: koko2799390267@if2306: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
2049: koko1542542355@if2050: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
1793: koko4212678092@if1794: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
1281: koko1342903858@if1282: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
3075: koko497316275@if3076: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
2819: koko1392270664@if2820: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
Any help in debugging this further would be appreciated
PODs connected by veth in the same node
I am deploying a topology with meshnet, where the containers are VM based containers, both are on the same node so a pair of veth interfaces is created to establish a link between the two pods, however when testing connectivity I have no connectivity between the pods, what could it be due to?
PodR1:
60: eth2@if61: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 12:96:dc:06:20:6b brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet 192.168.0.1/24 scope global eth2
valid_lft forever preferred_lft forever
inet6 fe80::1096:dcff:fe06:206b/64 scope link
PodR2
61: eth2@if60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether c6:d9:e4:c7:f5:fb brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet 192.168.0.2/24 scope global eth2
valid_lft forever preferred_lft forever
inet6 fe80::c4d9:e4ff:fec7:f5fb/64 scope link
valid_lft forever preferred_lft forever
root@r1:/# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
92 bytes from kne (192.168.0.1): Destination Host Unreachable
92 bytes from kne (192.168.0.1): Destination Host Unreachable
92 bytes from kne (192.168.0.1): Destination Host Unreachable
92 bytes from kne (192.168.0.1): Destination Host Unreachable
The POD connected by vxLAN cannot be pinged through
I have an OSPF topology with 10 nodes and they all run the same frrouting image
I want to test this topology with meshnet-cni
My k8s cluster has a total of 4 nodes connected through calico BGP mode
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready control-plane 7h49m v1.24.0 192.168.22.1 <none> Ubuntu 20.04.4 LTS 5.4.0-113-generic containerd://1.6.4
node-d Ready <none> 7h47m v1.24.0 192.168.22.3 <none> Ubuntu 20.04.4 LTS 5.4.0-113-generic containerd://1.6.4
node-i Ready <none> 7h45m v1.24.0 192.168.22.6 <none> Ubuntu 20.04.4 LTS 5.4.0-113-generic containerd://1.6.4
node-k Ready <none> 7h46m v1.24.0 192.168.22.7 <none> Ubuntu 20.04.4 LTS 5.4.0-110-generic containerd://1.6.4
I create my topology and distribute it on two nodes
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
router-1 1/1 Running 0 11m 10.224.71.138 node-d <none> <none>
router-10 1/1 Running 0 11m 10.224.100.91 node-k <none> <none>
router-2 1/1 Running 0 11m 10.224.100.86 node-k <none> <none>
router-3 1/1 Running 0 11m 10.224.100.87 node-k <none> <none>
router-4 1/1 Running 0 11m 10.224.71.137 node-d <none> <none>
router-5 1/1 Running 0 11m 10.224.100.88 node-k <none> <none>
router-6 1/1 Running 0 11m 10.224.100.89 node-k <none> <none>
router-7 1/1 Running 0 11m 10.224.100.90 node-k <none> <none>
router-8 1/1 Running 0 11m 10.224.71.140 node-d <none> <none>
router-9 1/1 Running 0 11m 10.224.71.139 node-d <none> <none>
For example, Router4 should establish a neighbor relationship with R3, R5, and R7, but they are not established and cannot ping through each other
router-4# show ip ospf neighbor
Neighbor ID Pri State Up Time Dead Time Address Interface RXmtL RqstL DBsmL
10.224.100.87 1 Init/DROther 22m21s 38.205s 10.0.4.1 eth1:10.0.4.2 0 0 0
10.224.100.88 1 Init/DROther 22m21s 38.093s 10.0.7.2 eth2:10.0.7.1 0 0 0
router-4# ping 10.0.4.1
PING 10.0.4.1 (10.0.4.1): 56 data bytes
^C
--- 10.0.4.1 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
However, if all the pods are running on the same node and connected via veth, this is successful
Neighbor ID Pri State Up Time Dead Time Address Interface RXmtL RqstL DBsmL
10.224.15.148 1 Full/DR 3h40m52s 38.032s 10.0.4.1 eth1:10.0.4.2 0 0 0
10.224.15.149 1 Full/DR 3h40m46s 33.848s 10.0.8.2 eth3:10.0.8.1 0 0 0
10.224.219.86 1 Full/Backup 3h34m08s 31.814s 10.0.7.2 eth2:10.0.7.1 0 0 0
Can someone help me?
issue with kind0.7.0, Error invoking Delegate Add missing network name
test env: kind0.7.0 with default kindnet
meshnet-cni manifest applied: meshnet-cni/manifests/base/meshnet.yml
pod stuck at ContainerCreating
Warning FailedCreatePodSandBox 23s (x17 over 3m48s) kubelet, kindnet-worker2 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "645a8daac55cc54853a1be8ff1e4db675505c3e55585f55f1660d65d106e2869": Error invoking Delegate Add missing network name
journalctl from worker2 node
Mar 21 02:19:09 kindnet-worker2 kubelet[539]: I0321 02:19:09.457259 539 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-7nvc2" (UniqueName: "kubernetes.io/secret/a8c8f3cd-4f
3e-476a-8c0d-7f94f246a1ac-default-token-7nvc2") pod "r1" (UID: "a8c8f3cd-4f3e-476a-8c0d-7f94f246a1ac")
Mar 21 02:19:09 kindnet-worker2 kubelet[539]: E0321 02:19:09.677373 539 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to setup network for sandbox "4ba52ab25c5a31e7177
248e5122b4c8cdeaae14e060d1a67da3de28f534498bb": Error invoking Delegate Add missing network name
Mar 21 02:19:09 kindnet-worker2 kubelet[539]: E0321 02:19:09.677472 539 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "r1_default(a8c8f3cd-4f3e-476a-8c0d-7f94f246a1ac)" failed: rpc error: code = Unknown desc = failed to s
etup network for sandbox "4ba52ab25c5a31e7177248e5122b4c8cdeaae14e060d1a67da3de28f534498bb": Error invoking Delegate Add missing network name
Mar 21 02:19:09 kindnet-worker2 kubelet[539]: E0321 02:19:09.677495 539 kuberuntime_manager.go:729] createPodSandbox for pod "r1_default(a8c8f3cd-4f3e-476a-8c0d-7f94f246a1ac)" failed: rpc error: code = Unknown desc = failed to
setup network for sandbox "4ba52ab25c5a31e7177248e5122b4c8cdeaae14e060d1a67da3de28f534498bb": Error invoking Delegate Add missing network name
Mar 21 02:19:09 kindnet-worker2 kubelet[539]: E0321 02:19:09.677614 539 pod_workers.go:191] Error syncing pod a8c8f3cd-4f3e-476a-8c0d-7f94f246a1ac ("r1_default(a8c8f3cd-4f3e-476a-8c0d-7f94f246a1ac)"), skipping: failed to "Creat
ePodSandbox" for "r1_default(a8c8f3cd-4f3e-476a-8c0d-7f94f246a1ac)" with CreatePodSandboxError: "CreatePodSandbox for pod \"r1_default(a8c8f3cd-4f3e-476a-8c0d-7f94f246a1ac)\" failed: rpc error: code = Unknown desc = failed to setup
network for sandbox \"4ba52ab25c5a31e7177248e5122b4c8cdeaae14e060d1a67da3de28f534498bb\": Error invoking Delegate Add missing network name"
/etc/cni/net.d/
-rw-r--r-- 1 root root 411 Mar 21 02:17 00-meshnet.conf
-rw-r--r-- 1 root root 391 Mar 21 02:17 10-kindnet.conflist
-rw-r--r-- 1 root root 326 Mar 21 02:17 meshnet.conf
{
"cniVersion": "0.2.0",
"name": "meshnet_network",
"type": "meshnet",
"delegate": {
"type": "ptp",
"ipMasq": false,
"ipam": {
"type": "host-local",
"dataDir": "/run/cni-ipam-state",
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"ranges": [
[
{
"subnet": "10.244.2.0/24"
}
]
]
}
}
}
{
"cniVersion": "0.3.1",
"name": "kindnet",
"plugins": [
{
"type": "ptp",
"ipMasq": false,
"ipam": {
"type": "host-local",
"dataDir": "/run/cni-ipam-state",
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"ranges": [
[
{
"subnet": "10.244.2.0/24"
}
]
]
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
{
"cniVersion": "0.2.0",
"name": "meshnet_network",
"type": "meshnet",
"delegate": {
"name": "dind0",
"bridge": "dind0",
"type": "bridge",
"isDefaultGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.244.1.0/24",
"gateway": "10.244.1.1"
}
}
}
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.