DNS Problem about multus-cni

[sara4dev]

I am also having the same issue. Its not about DNS. Its the routing to the service ip cidr (10.96.0.0/12) is missing in the container.

[rkamudhan]

@DigiDaz I edited your issue with Mardown for understanding. Please have a look how to include the "resolvConf": "/etc/resolv.conf" in the https://github.com/containernetworking/plugins/tree/master/plugins/ipam/host-local. What is your /run/flannel/subnet.env ? Kubernetes will get only Flannel IP, which is the master plugin in this case.

[PBXForums]

@rkamudhan Thank you, my /run/flannel/subnet.env is:

FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.1.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

I will look at the resolv.conf stuff but do I not need to access the dns at 10.96.0.10?

[PBXForums]

Setting the resolvConf option does not work for me, I thought my syntax may be wrong but I can echo it from the command line perfectly fine:

 echo '{ "name": "default", "ipam": {"type": "host-local","subnet": "137.74.152.128/26","rangeStart": "137.74.152.129","rangeEnd": "137.74.152.189","resolvConf": "/etc/resolv.conf","routes": [{ "dst": "0.0.0.0/0","gw":"137.74.152.190" }],"gateway": "137.74.152.190"} }'  | CNI_COMMAND=ADD CNI_CONTAINERID=example CNI_NETNS=/dev/null CNI_IFNAME=dummy0 CNI_PATH=. ./host-local
{
    "cniVersion": "0.2.0",
    "ip4": {
        "ip": "137.74.152.130/26",
        "gateway": "137.74.152.190",
        "routes": [
            {
                "dst": "0.0.0.0/0",
                "gw": "137.74.152.190"
            }
        ]
    },
    "dns": {
        "nameservers": [
            "127.0.0.1",
            "213.186.33.99"
        ],
        "search": [
            "ovh.net"
        ]
    }
}

In a new container:

cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local ovh.net
options ndots:5

more /etc/cni/net.d/10-multus.conf
{
  "name": "multus-demo",
  "type": "multus",
  "delegates": [
    {
      "type": "macvlan",
      "master": "eth1",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "137.74.152.128/26",
        "rangeStart": "137.74.152.129",
        "rangeEnd": "137.74.152.189",
        "resolvConf": "/etc/resolv.conf",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "137.74.152.190"
     }
    },
    {
      "type": "flannel",
      "masterplugin": true,
      "delegate": {
        "isDefaultGateway": false
      }
    }
  ]
}

[rkamudhan]

@DigiDaz

From Host-local documentation:

• host-local IPAM allocates IPv4 and IPv6 addresses out of a specified address range. Optionally, it can include a DNS configuration from a resolv.conf file on the host.
• resolvConf (string, optional): Path to a resolv.conf on the host to parse and return as the DNS configuration

Container must have the following data

 "dns": {
        "nameservers": [
            "127.0.0.1",
            "213.186.33.99"
        ],
        "search": [
            "ovh.net"
        ]
    }

Are you sure the application is not rewriting any thing in the container /etc/resolv.conf ?

[PBXForums]

This is the container resolv.conf

cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local ovh.net
options ndots:5

This is host resolv.conf:

more /etc/resolv.conf
nameserver 127.0.0.1
nameserver 213.186.33.99
search ovh.net

[rkamudhan]

@DigiDaz Is that all nodes in your cluster have same "resolvConf": "/etc/resolv.conf" . Can you check the /etc/resolv.conf in the node, where the container is running. Can you display both your nodes and running containers /etc/resolv.conf

[PBXForums]

I only have two:

The master:

more /etc/resolv.conf
nameserver 213.186.33.99
search local

and the one the containers are being created on:

nameserver 127.0.0.1
nameserver 213.186.33.99
search ovh.net

The containers all have:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local ovh.net
options ndots:5

It baffles me because because I can do the echo '{ "name": "default", "ipam": {"type": "host-local","subnet": "137.74.152.128/26, etc, etc from the cli on the hosts and see what looks like the correct info.

Is there anyway this could be getting set as expected and then overridden at a later stage of the container creation process?

[PBXForums]

kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
If you don't see a command prompt, try pressing enter.
/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local ovh.net
options ndots:5

/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 66:6B:72:07:09:AA
          inet addr:10.244.1.4  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::646b:72ff:fe07:9aa/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:5 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:438 (438.0 B)  TX bytes:690 (690.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

net0      Link encap:Ethernet  HWaddr EA:1F:65:03:49:07
          inet addr:137.74.152.136  Bcast:0.0.0.0  Mask:255.255.255.192
          inet6 addr: fe80::e81f:65ff:fe03:4907/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1142 errors:0 dropped:0 overruns:0 frame:0
          TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:68520 (66.9 KiB)  TX bytes:2304 (2.2 KiB)

/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         137.74.152.190  0.0.0.0         UG    0      0        0 net0
10.244.0.0      10.244.1.1      255.255.0.0     UG    0      0        0 eth0
10.244.1.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
137.74.152.128  0.0.0.0         255.255.255.192 U     0      0        0 net0

[rkamudhan]

Can you display kubectl get svc --all-namespaces and find your service with name kube-dns. I think you running kude-dns with cluster-IP 10.96.0.10

[PBXForums]

Yes, that seems the case, is that not correct?

NAMESPACE     NAME              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
default       kubernetes        ClusterIP   10.96.0.1        <none>        443/TCP         1d
default       wordpress-mysql   ClusterIP   None             <none>        3306/TCP        23h
kube-system   kube-dns          ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP   1d
kube-system   tiller-deploy     ClusterIP   10.107.196.214   <none>        44134/TCP       1d

[rkamudhan]

@DigiDaz Please set the dnspolicy in the pod refer this webpage for more details https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/ to your deployment mode and requirement. But it is good to raise this issue in CNI community and clarifies the issue.

[rkamudhan]

Hi @saravanakumar-periyasamy, Can you display the route in your container with Multus and without Multus along with your CNI conf and Network object you used in pod spec. I want to make sure Multus is not giving any preference in the routing.

[dougbtv]

@DigiDaz -- could you please close this issue? If you still have concerns, feel free to re-start the conversation and we'll address them, thank you!