apiserver was unable to write a fallback JSON response: http: Handler timeout about clusternet

[dixudx]

clusternet-hub print error msg , but It seems that application distribution is normal.

No need to get into a panic on these error messages. The application distribution is running as expected.

[silenceper]

ok

[dixudx]

@silenceper Cannot reproduce this.

Please make sure your clusternet-hub deployed successfully.

📌 📌
Since clusternet-hub is running as an AA, please make sure that parent apiserver could visit the clusternet-hub service.

From the error message,

E0909 07:06:10.472963 1 wrap.go:54] timeout or abort while handling: GET "/apis/shadow/v1alpha1"
I0909 07:06:10.473122 1 httplog.go:89] "HTTP" verb="GET" URI="/apis/shadow/v1alpha1" latency="740.108µs" userAgent="Go-http-client/2.0" srcIP="172.17.0.1:21105" resp=200

I guess that your kube-apiserver cannot visit clusternet-hub service.

[TanyaSingh-espina]

@dixudx can you please help in resolving the above issue. I am also facing similar issue while trying to create the Subscriptions on child cluster using kubectl clusternet (replication scheduling).

[dixudx]

@TanyaSingh-espina Would you please share some related logs of clusternet-hub. And also please do run kubectl clusternet apply -f a-demo-file.yaml -v 10 with verbose. Let's see what happened.

[TanyaSingh-espina]

@dixudx
clusternet-hub logs :

E0418 03:35:32.504329       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/shadow/v1alpha1" audit-ID="2e8b0cad-674e-444b-aecf-423ac1322c42"
E0418 03:35:32.504756       1 writers.go:122] apiserver was unable to write a JSON response: http2: stream closed
E0418 03:35:32.504796       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/shadow/v1alpha1" audit-ID="fc8fe5e2-8ed0-4e39-9a14-613a0b8c8ed3"
E0418 03:35:32.504894       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/shadow/v1alpha1" audit-ID="ef936b5d-694a-4194-98ad-d6655c5a8da1"
E0418 03:35:32.505997       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0418 03:35:32.507115       1 timeout.go:142] post-timeout activity - time-elapsed: 2.206435ms, GET "/apis/shadow/v1alpha1" result: <nil>
E0418 03:35:32.508182       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0418 03:35:32.509253       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0418 03:35:32.512479       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0418 03:35:32.516795       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0418 03:35:32.517855       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0418 03:35:32.518925       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0418 03:35:32.520011       1 timeout.go:142] post-timeout activity - time-elapsed: 15.690053ms, GET "/apis/shadow/v1alpha1" result: <nil>
E0418 03:35:32.521076       1 timeout.go:142] post-timeout activity - time-elapsed: 16.737069ms, GET "/apis/shadow/v1alpha1" result: <nil>
E0418 03:35:32.522140       1 timeout.go:142] post-timeout activity - time-elapsed: 17.335479ms, GET "/apis/shadow/v1alpha1" result: <nil>

kube-apiserver logs from parent cluster :

W0418 02:33:21.416453       1 handler_proxy.go:93] no RequestInfo found in the context
E0418 02:33:21.416499       1 controller.go:102] loading OpenAPI spec for "v1alpha1.proxies.clusternet.io" failed with: failed to download v1alpha1.proxies.clusternet.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0418 02:33:21.418516       1 available_controller.go:460] v1alpha1.proxies.clusternet.io failed with: failing or missing response from https://10.96.24.34:443/apis/proxies.clusternet.io/v1alpha1: Get "https://10.96.24.34:443/apis/proxies.clusternet.io/v1alpha1": dial tcp 10.96.24.34:443: connect: connection refused
E0418 02:33:21.418616       1 available_controller.go:460] v1alpha1.shadow failed with: failing or missing response from https://10.96.24.34:443/apis/shadow/v1alpha1: Get "https://10.96.24.34:443/apis/shadow/v1alpha1": dial tcp 10.96.24.34:443: connect: connection refused
I0418 02:33:21.418636       1 controller.go:109] OpenAPI AggregationController: action for item v1alpha1.proxies.clusternet.io: Rate Limited Requeue.
E0418 02:33:22.513163       1 available_controller.go:460] v1alpha1.proxies.clusternet.io failed with: failing or missing response from https://10.96.24.34:443/apis/proxies.clusternet.io/v1alpha1: Get "https://10.96.24.34:443/apis/proxies.clusternet.io/v1alpha1": dial tcp 10.96.24.34:443: connect: connection refused
E0418 02:33:22.641113       1 available_controller.go:460] v1alpha1.shadow failed with: failing or missing response from https://10.96.24.34:443/apis/shadow/v1alpha1: Get "https://10.96.24.34:443/apis/shadow/v1alpha1": dial tcp 10.96.24.34:443: connect: connection refused

clusternet-hub service :

NAME             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
clusternet-hub   ClusterIP   10.96.24.34   <none>        443/TCP   43h

I created the subscriptions using exapmle : [(https://github.com/clusternet/clusternet/tree/main/examples/replication-scheduling)]

clickedge-master1:~$ kubectl clusternet get mcls -A
NAMESPACE          NAME                       CLUSTER ID                             SYNC MODE   KUBERNETES   STATUS   AGE
clusternet-l6f8l   clusternet-cluster-wdkvk   741d35bb-e1dd-4161-84f2-4e00ea98565a   Dual        v1.28.8      True     42h
clickedge-master1:~$ kubectl clusternet get subs -A
NAMESPACE   NAME       AGE
default     app-demo   17h
clickedge-master1:~$ kubectl clusternet get chart
NAME    CHART   VERSION   REPO                                 STATUS   AGE
mysql   mysql   9.2.0     https://charts.bitnami.com/bitnami   Found    17h
clickedge-master1:~$  kubectl clusternet get ns
NAME   STATUS   AGE
abc             42h
foo             17h
clickedge-master1:~$ kubectl clusternet get svc -n foo
NAME           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
my-nginx-svc   ClusterIP   <none>       <none>        80/TCP    17h
clickedge-master1:~$ kubectl clusternet get deploy -n foo
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
my-nginx   0/3     0            0           17h

Clusternet-controller logs :

I0418 02:33:46.596231       1 helmchart.go:254] successfully synced HelmChart "default/mysql"
E0418 02:33:51.585483       1 base.go:280] error syncing 'clusternet-l6f8l/default-app-demo': helmchart.apps.clusternet.io "mysql-operator" not found, requeuing
E0418 02:34:01.826475       1 base.go:280] error syncing 'clusternet-l6f8l/default-app-demo': helmchart.apps.clusternet.io "mysql-operator" not found, requeuing
E0418 02:34:22.306948       1 base.go:280] error syncing 'clusternet-l6f8l/default-app-demo': helmchart.apps.clusternet.io "mysql-operator" not found, requeuing
E0418 02:35:03.268713       1 base.go:280] error syncing 'clusternet-l6f8l/default-app-demo': helmchart.apps.clusternet.io "mysql-operator" not found, requeuing
E0418 02:36:25.188712       1 base.go:280] error syncing 'clusternet-l6f8l/default-app-demo': helmchart.apps.clusternet.io "mysql-operator" not found, requeuing
E0418 02:39:09.029847       1 base.go:280] error syncing 'clusternet-l6f8l/default-app-demo': helmchart.apps.clusternet.io "mysql-operator" not found, requeuing
E0418 02:44:36.710123       1 base.go:280] error syncing 'clusternet-l6f8l/default-app-demo': helmchart.apps.clusternet.io "mysql-operator" not found, requeuing
E0418 02:55:32.071274       1 base.go:280] error syncing 'clusternet-l6f8l/default-app-demo': helmchart.apps.clusternet.io "mysql-operator" not found, requeuing
E0418 03:12:12.072160       1 base.go:280] error syncing 'clusternet-l6f8l/default-app-demo': helmchart.apps.clusternet.io "mysql-operator" not found, requeuing
E0418 03:28:52.073259       1 base.go:280] error syncing 'clusternet-l6f8l/default-app-demo': helmchart.apps.clusternet.io "mysql-operator" not found, requeuing

I have created the parent and child clusternet components using helm chart method. I am able to access the child cluster from parent cluster using kubeconfig. But the resources are not being created from parent to child.

clickedge-master1:~$ kubectl clusternet get desc -A
No resources found

[dixudx]

v1alpha1.shadow failed with: failing or missing response from https://10.96.24.34:443/apis/shadow/v1alpha1: Get "https://10.96.24.34:443/apis/shadow/v1alpha1": dial tcp 10.96.24.34:443: connect: connection refused

NAME             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
clusternet-hub   ClusterIP   10.96.24.34   <none>        443/TCP   43h

@TanyaSingh-espina From above logs, the API service clusternet-hub cannot be accessed. Please check your Kubernetes cluster settings. You can follow this tutorial on debugging services.

[TanyaSingh-espina]

Hi @dixudx thanks for the help. Although I am not able to resolve this issue yet. Clusternet-hub service is working fine as checked with the link you have given above.

Also I want to know what does it mean by the following statement :

I have created the cluster with default settings provided by clusternet, what else I have to do to make sure that clusternet-hub service is reachable by api-server.

[TanyaSingh-espina]

Can some please help me out with above issue. Also if there is any documentation for the above issue please do share.

[dixudx]

@TanyaSingh-espina clusternet-hub is running as a service inside the Kubernetes cluster. In order to make sure kube-apiserver can access service clusternet-hub, please do follow this tutorial on debugging services. You can share some logs/outputs when following the tutorial.

[TanyaSingh-espina]

Hi @dixudx I tried out the working of services from above example which you shared. According to my understanding the service is working fine. Below I have attached the logs and output for the same. Still dont understand why service is not working for clusternet-hub-svc.

clickedge-master1:~$ kubectl get pods -l app=hostnames
NAME                         READY   STATUS    RESTARTS   AGE
hostnames-54b85cf467-96gw8   1/1     Running   0          10s
hostnames-54b85cf467-wx4cx   1/1     Running   0          10s
hostnames-54b85cf467-xxcrs   1/1     Running   0          18s

@clickedge-master1:~$ kubectl get pods -l app=hostnames \
>     -o go-template='{{range .items}}{{.status.podIP}}{{"\n"}}{{end}}'
192.168.227.85
192.168.227.84
192.168.227.83

clickedge-master1:~$ for ep in 192.168.227.85:9376 192.168.227.84:9376 192.168.227.83:9376; do     wget -qO
- $ep; done
hostnames-54b85cf467-96gw8
hostnames-54b85cf467-wx4cx
hostnames-54b85cf467-xxcrs

clickedge-master1:~$ kubectl get svc hostnames
NAME        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
hostnames   ClusterIP   10.98.183.102   <none>        80/TCP    8s

clickedge-master1:~$ kubectl run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox sh
If you don't see a command prompt, try pressing enter.
/ #
/ #
/ #
/ # nslookup hostnames
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      hostnames
Address 1: 10.98.183.102 hostnames.default.svc.cluster.local

/ # wget -qO- 10.98.183.102:80
hostnames-54b85cf467-96gw8
/ # wget -qO- 10.98.183.102:80
hostnames-54b85cf467-wx4cx
/ # wget -qO- 10.98.183.102:80
hostnames-54b85cf467-96gw8
/ # wget -qO- 10.98.183.102:80
hostnames-54b85cf467-wx4cx
/ # wget -qO- 10.98.183.102:80
hostnames-54b85cf467-xxcrs

clickedge-master1:~$ kubectl get service hostnames -o json
{
    "apiVersion": "v1",
    "kind": "Service",
    "metadata": {
        "creationTimestamp": "2024-04-30T07:24:05Z",
        "labels": {
            "app": "hostnames"
        },
        "name": "hostnames",
        "namespace": "default",
        "resourceVersion": "27937",
        "uid": "dae287be-f6f2-4101-a609-9b67ad29a63f"
    },
    "spec": {
        "clusterIP": "10.98.183.102",
        "clusterIPs": [
            "10.98.183.102"
        ],
        "internalTrafficPolicy": "Cluster",
        "ipFamilies": [
            "IPv4"
        ],
        "ipFamilyPolicy": "SingleStack",
        "ports": [
            {
                "port": 80,
                "protocol": "TCP",
                "targetPort": 9376
            }
        ],
        "selector": {
            "app": "hostnames"
        },
        "sessionAffinity": "None",
        "type": "ClusterIP"
    },
    "status": {
        "loadBalancer": {}
    }
}

clickedge-master1:~$ kubectl get pods -l app=hostnames
NAME                         READY   STATUS    RESTARTS   AGE
hostnames-54b85cf467-96gw8   1/1     Running   0          16m
hostnames-54b85cf467-wx4cx   1/1     Running   0          16m
hostnames-54b85cf467-xxcrs   1/1     Running   0          16m

clickedge-master1:~$ kubectl get endpoints hostnames
NAME        ENDPOINTS                                                     AGE
hostnames   192.168.227.83:9376,192.168.227.84:9376,192.168.227.85:9376   12m

clickedge-master1:~$ ps auxw | grep kube-proxy
root        9161  0.0  0.6 1281800 52516 ?       Ssl  05:01   0:01 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=clickedge-master1
tanyasi+  105561  0.0  0.0   8168  2472 pts/1    S+   07:37   0:00 grep --color=auto kube-proxy

clickedge-master1:~$ sudo iptables-save | grep hostnames
-A KUBE-SEP-OB7NLICFMWEZEL65 -s 192.168.227.83/32 -m comment --comment "default/hostnames" -j KUBE-MARK-MASQ
-A KUBE-SEP-OB7NLICFMWEZEL65 -p tcp -m comment --comment "default/hostnames" -m tcp -j DNAT --to-destination 192.168.227.83:9376
-A KUBE-SEP-PYRIOMGF6XAO6LWT -s 192.168.227.84/32 -m comment --comment "default/hostnames" -j KUBE-MARK-MASQ
-A KUBE-SEP-PYRIOMGF6XAO6LWT -p tcp -m comment --comment "default/hostnames" -m tcp -j DNAT --to-destination 192.168.227.84:9376
-A KUBE-SEP-RPUX3Z7NWZ5XLRR7 -s 192.168.227.85/32 -m comment --comment "default/hostnames" -j KUBE-MARK-MASQ
-A KUBE-SEP-RPUX3Z7NWZ5XLRR7 -p tcp -m comment --comment "default/hostnames" -m tcp -j DNAT --to-destination 192.168.227.85:9376
-A KUBE-SERVICES -d 10.98.183.102/32 -p tcp -m comment --comment "default/hostnames cluster IP" -m tcp --dport 80 -j KUBE-SVC-YN5D6RYVEVZOH44Q
-A KUBE-SVC-YN5D6RYVEVZOH44Q ! -s 192.168.0.0/16 -d 10.98.183.102/32 -p tcp -m comment --comment "default/hostnames cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SVC-YN5D6RYVEVZOH44Q -m comment --comment "default/hostnames -> 192.168.227.83:9376" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-OB7NLICFMWEZEL65
-A KUBE-SVC-YN5D6RYVEVZOH44Q -m comment --comment "default/hostnames -> 192.168.227.84:9376" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-PYRIOMGF6XAO6LWT
-A KUBE-SVC-YN5D6RYVEVZOH44Q -m comment --comment "default/hostnames -> 192.168.227.85:9376" -j KUBE-SEP-RPUX3Z7NWZ5XLRR7
[email protected]@clickedge-master1:~$ kubectl get po -o wide | grep host
hostnames-54b85cf467-96gw8      1/1     Running   0          19m   192.168.227.85   clickedge-master1   <none>           <none>
hostnames-54b85cf467-wx4cx      1/1     Running   0          19m   192.168.227.84   clickedge-master1   <none>           <none>
hostnames-54b85cf467-xxcrs      1/1     Running   0          19m   192.168.227.83   clickedge-master1   <none>           <none>

clickedge-master1:~$ curl 10.98.183.102:80
hostnames-54b85cf467-xxcrs

[dixudx]

@TanyaSingh-espina Please debug service clusternet-hub running in namespace clusternet-system.

[TanyaSingh-espina]

@dixudx
The clusternet-hub service is not running as expected. I tried out similar tests for clusternet-hub, but the clusternet-hub service is not able to resolve its endpoints.

clickedge-master1:~$ kubectl run -it --rm --restart=Never busybox  -n clusternet-system --image=gcr.io/goog
le-containers/busybox sh
If you don't see a command prompt, try pressing enter.
/ # wget -qO - 192.168.227.78:443
wget: server returned error: HTTP/1.0 400 Bad Request
/ #

/ # nslookup clusternet-hub
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      clusternet-hub
Address 1: 10.96.124.7 clusternet-hub.clusternet-system.svc.cluster.local

Also there is no error logs in kube-proxy pod :

clickedge-master1:~$ kubectl logs  kube-proxy-rwx66 -n kube-system
I0430 05:01:07.072455       1 server_others.go:69] "Using iptables proxy"
I0430 05:01:07.080261       1 node.go:141] Successfully retrieved node IP: 10.122.131.6
I0430 05:01:07.081329       1 conntrack.go:52] "Setting nf_conntrack_max" nfConntrackMax=131072
I0430 05:01:07.081444       1 conntrack.go:100] "Set sysctl" entry="net/netfilter/nf_conntrack_tcp_timeout_close_wait" value=3600
I0430 05:01:07.134513       1 server.go:632] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
I0430 05:01:07.136806       1 server_others.go:152] "Using iptables Proxier"
I0430 05:01:07.136952       1 server_others.go:421] "Detect-local-mode set to ClusterCIDR, but no cluster CIDR for family" ipFamily="IPv6"
I0430 05:01:07.137028       1 server_others.go:438] "Defaulting to no-op detect-local"
I0430 05:01:07.137107       1 proxier.go:250] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses"
I0430 05:01:07.137394       1 server.go:846] "Version info" version="v1.28.9"
I0430 05:01:07.137640       1 server.go:848] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0430 05:01:07.139270       1 config.go:188] "Starting service config controller"
I0430 05:01:07.139437       1 shared_informer.go:311] Waiting for caches to sync for service config
I0430 05:01:07.139706       1 config.go:97] "Starting endpoint slice config controller"
I0430 05:01:07.143886       1 shared_informer.go:311] Waiting for caches to sync for endpoint slice config
I0430 05:01:07.141833       1 config.go:315] "Starting node config controller"
I0430 05:01:07.144175       1 shared_informer.go:311] Waiting for caches to sync for node config
I0430 05:01:07.244642       1 shared_informer.go:318] Caches are synced for node config
I0430 05:01:07.244688       1 shared_informer.go:318] Caches are synced for service config
I0430 05:01:07.244871       1 shared_informer.go:318] Caches are synced for endpoint slice config

clickedge-master1:~$ kubectl get svc clusternet-hub -n clusternet-system -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: clusternet-hub
    meta.helm.sh/release-namespace: clusternet-system
  creationTimestamp: "2024-04-30T05:39:10Z"
  labels:
    app: clusternet-hub
    app.kubernetes.io/instance: clusternet-hub
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: clusternet-hub
    helm.sh/chart: clusternet-hub-0.13.0
  name: clusternet-hub
  namespace: clusternet-system
  resourceVersion: "3614"
  uid: ef09b038-49df-4880-b251-f393ca2df186
spec:
  clusterIP: 10.96.124.7
  clusterIPs:
  - 10.96.124.7
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: clusternet-hub
    app.kubernetes.io/instance: clusternet-hub
    app.kubernetes.io/name: clusternet-hub
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Please do let me know if need any specific logs.

[TanyaSingh-espina]

In addition I have already added the parameter --aggregator-reject-forwarding-redirect=false in api-server for clusternet-hub to act as AA.

----
    Host Port:     <none>
    Command:
      kube-apiserver
      --advertise-address=10.122.131.6
      --allow-privileged=true
      --authorization-mode=Node,RBAC
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --aggregator-reject-forwarding-redirect=false
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
      --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
      --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
      --etcd-servers=https://127.0.0.1:2379
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key

[dixudx]

but the clusternet-hub service is not able to resolve its endpoints.

Are there any endpoints for service clusternet-hub. Please also check the healthy status of clusternet-hub pods, including logs, etc.

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      clusternet-hub
Address 1: 10.96.124.7 clusternet-hub.clusternet-system.svc.cluster.local

Would you be able to telnet 10.96.124.7 on port 443 or ping the matching endpoints?

And what is the version of your Kubernetes cluster?

[dixudx]

In addition I have already added the parameter --aggregator-reject-forwarding-redirect=false in api-server for clusternet-hub to act as AA.

----
    Host Port:     <none>
    Command:
      kube-apiserver
      --advertise-address=10.122.131.6
      --allow-privileged=true
      --authorization-mode=Node,RBAC
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --aggregator-reject-forwarding-redirect=false
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
      --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
      --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
      --etcd-servers=https://127.0.0.1:2379
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key

Not seen this in the command arguments. Have you restarted kube-apiserver after modifying the configurations?

[TanyaSingh-espina]

but the clusternet-hub service is not able to resolve its endpoints.

Are there any endpoints for service clusternet-hub. Please also check the healthy status of clusternet-hub pods, including logs, etc.

Yes ednspoints are mapped in the clusternet-hub svc. Clusternet-hub pods are in running state, although the active clusternet-hub po is showing the error apiserver is not able to talk to clusternet-hub svc

nil>
E0430 09:20:28.719199       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0430 09:20:28.721345       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0430 09:20:28.723490       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0430 09:20:28.726690       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0430 09:20:28.728861       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0430 09:20:28.729934       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0430 09:20:28.731013       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0430 09:20:28.732082       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0430 09:20:28.733178       1 timeout.go:142] post-timeout activity - time-elapsed: 18.890959ms, GET "/apis/shadow/v1alpha1" result: <nil>

Would you be able to telnet 10.96.124.7 on port 443 or ping the matching endpoints?

telnet needs to done from inside the clusternet-hub pods ?

And what is the version of your Kubernetes cluster?

clickedge-master1:~$ kubectl version
Client Version: v1.28.9
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.9

Have you restarted kube-apiserver after modifying the configurations?

Yes. After changing the manifest file of kube-apiserver from location /etc/kuberentes/manifests/kube-apiserver.yaml , I restarted the kubelet service. And new api-server pod started.

[TanyaSingh-espina]

From kube-proxy logs :

I0430 09:10:55.520244     246 endpointslicecache.go:358] "Setting endpoints for service port name" portName="clusternet-system/clusternet-hub:https" endpoints=["192.168.227.77:443","192.168.227.78:443","192.168.227.79:443"]

There are some error logs also in the kube-proxy :

E0430 09:10:55.414663     246 server.go:787] "Healthz server failed" err="failed to start proxier healthz on 0.0.0.0:10256: listen tcp 0.0.0.0:10256: bind: address already in use"
E0430 09:10:55.415292     246 server.go:830] starting metrics server failed: listen tcp 127.0.0.1:10249: bind: address already in use

-----

E0430 09:11:00.415851     246 server.go:830] starting metrics server failed: listen tcp 127.0.0.1:10249: bind: address already in useE0430 09:11:00.415859     246 server.go:787] "Healthz server failed" err="failed to start proxier healthz on 0.0.0.0:10256: listen tcp 0.0.0.0:10256: bind: address already in use"                                                                                       
E0430 09:11:05.417073     246 server.go:830] starting metrics server failed: listen tcp 127.0.0.1:10249: bind: address already in useE0430 09:11:05.417080     246 server.go:787] "Healthz server failed" err="failed to start proxier healthz on 0.0.0.0:10256: listen tcp 0.0.0.0:10256: bind: address already in use"                                                                                      
 E0430 09:11:10.418204     246 server.go:787] "Healthz server failed" err="failed to start proxier healthz on 0.0.0.0:10256: listen tcp 0.0.0.0:10256: bind: address already in use"                                                                                      
 E0430 09:11:10.418480     246 server.go:830] starting metrics server failed: listen tcp 127.0.0.1:10249: bind: address already in use
E0430 09:11:15.419478     246 server.go:830] starting metrics server failed: listen tcp 127.0.0.1:10249: bind: address already in use
E0430 09:11:15.419491     246 server.go:787] "Healthz server failed" err="failed to start proxier healthz on 0.0.0.0:10256: listen tcp 0.0.0.0:10256: bind: address already in use"                 

[TanyaSingh-espina]

@dixudx
I did telnet from inside another pod to the clusternet-hub service.

clickedge-master1:~$ kubectl run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox
sh
If you don't see a command prompt, try pressing enter.
/ #
/ #
/ # telnet  10.96.124.7 443
Connection closed by foreign host
/ #

Below are the tcpdump captured for 443 port :

 sudo tcpdump -i any port 443 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
10:08:58.189706 IP 10.122.131.6.63650 > 192.168.227.78.443: Flags [P.], seq 955145564:955145606, ack 3942321037, win 986, options [nop,nop,TS val 3150705559 ecr 3837364658], length 42
10:08:58.189777 IP 10.122.131.6.63650 > 192.168.227.78.443: Flags [P.], seq 42:84, ack 1, win 986, options [nop,nop,TS val 3150705559 ecr 3837364658], length 42
10:08:58.189830 IP 192.168.227.78.443 > 10.122.131.6.63650: Flags [.], ack 84, win 9836, options [nop,nop,TS val 3837365553 ecr 3150705559], length 0
10:08:58.190510 IP 192.168.227.78.443 > 10.122.131.6.63650: Flags [P.], seq 1:155, ack 84, win 9836, options [nop,nop,TS val 3837365554 ecr 3150705559], length 154
10:08:58.190553 IP 10.122.131.6.63650 > 192.168.227.78.443: Flags [.], ack 155, win 986, options [nop,nop,TS val 3150705560 ecr 3837365554], length 0
10:08:58.190596 IP 192.168.227.78.443 > 10.122.131.6.63650: Flags [P.], seq 155:411, ack 84, win 9836, options [nop,nop,TS val 3837365554 ecr 3150705560], length 256
10:08:58.190616 IP 10.122.131.6.63650 > 192.168.227.78.443: Flags [.], ack 411, win 985, options [nop,nop,TS val 3150705560 ecr 3837365554], length 0
10:08:58.190964 IP 10.122.131.6.63650 > 192.168.227.78.443: Flags [P.], seq 84:125, ack 411, win 986, options [nop,nop,TS val 3150705560 ecr 3837365554], length 41
10:08:58.191579 IP 192.168.227.78.41420 > 10.96.0.1.443: Flags [P.], seq 3492077288:3492077323, ack 1998140385, win 6097, options [nop,nop,TS val 55164054 ecr 3691775137], length 35
10:08:58.191612 IP 10.96.0.1.443 > 192.168.227.78.41420: Flags [.], ack 35, win 501, options [nop,nop,TS val 3691776034 ecr 55164054], length 0
10:08:58.191615 IP 192.168.227.78.443 > 10.122.131.6.63650: Flags [P.], seq 411:477, ack 125, win 9836, options [nop,nop,TS val 3837365555 ecr 3150705560], length 66
10:08:58.191654 IP 192.168.227.78.443 > 10.122.131.6.63650: Flags [P.], seq 477:733, ack 125, win 9836, options [nop,nop,TS val 3837365555 ecr 3150705560], length 256
10:08:58.191795 IP 10.122.131.6.63650 > 192.168.227.78.443: Flags [.], ack 733, win 986, options [nop,nop,TS val 3150705561 ecr 3837365555], length 0
10:08:58.191868 IP 192.168.227.78.41420 > 10.96.0.1.443: Flags [P.], seq 35:79, ack 1, win 6097, options [nop,nop,TS val 55164054 ecr 3691776034], length 44
10:08:58.191887 IP 10.96.0.1.443 > 192.168.227.78.41420: Flags [.], ack 79, win 501, options [nop,nop,TS val 3691776034 ecr 55164054], length 0
10:08:58.191960 IP 192.168.227.78.41420 > 10.96.0.1.443: Flags [P.], seq 79:390, ack 1, win 6097, options [nop,nop,TS val 55164054 ecr 3691776034], length 311
10:08:58.191980 IP 10.96.0.1.443 > 192.168.227.78.41420: Flags [.], ack 390, win 501, options [nop,nop,TS val 3691776034 ecr 55164054], length 0
10:08:58.192202 IP 10.122.131.6.63650 > 192.168.227.78.443: Flags [P.], seq 125:166, ack 733, win 986, options [nop,nop,TS val 3150705561 ecr 3837365555], length 41
10:08:58.193461 IP 10.96.0.1.443 > 192.168.227.78.41420: Flags [P.], seq 1:94, ack 390, win 501, options [nop,nop,TS val 3691776036 ecr 55164054], length 93
10:08:58.193481 IP 192.168.227.78.41420 > 10.96.0.1.443: Flags [.], ack 94, win 6097, options [nop,nop,TS val 55164056 ecr 3691776036], length 0
10:08:58.193614 IP 10.96.0.1.443 > 192.168.227.78.41420: Flags [P.], seq 94:812, ack 390, win 501, options [nop,nop,TS val 3691776036 ecr 55164056], length 718
10:08:58.193626 IP 192.168.227.78.41420 > 10.96.0.1.443: Flags [.], ack 812, win 6094, options [nop,nop,TS val 55164056 ecr 3691776036], length 0
10:08:58.194881 IP 192.168.227.78.443 > 10.122.131.6.63650: Flags [P.], seq 733:853, ack 166, win 9836, options [nop,nop,TS val 3837365558 ecr 3150705561], length 120
10:08:58.195241 IP 192.168.227.78.41420 > 10.96.0.1.443: Flags [P.], seq 390:435, ack 812, win 6097, options [nop,nop,TS val 55164057 ecr 3691776036], length 45
10:08:58.195283 IP 192.168.227.78.41420 > 10.96.0.1.443: Flags [P.], seq 435:761, ack 812, win 6097, options [nop,nop,TS val 55164057 ecr 3691776036], length 326
10:08:58.195359 IP 10.96.0.1.443 > 192.168.227.78.41420: Flags [.], ack 761, win 501, options [nop,nop,TS val 3691776037 ecr 55164057], length 0
10:08:58.195383 IP 192.168.227.78.443 > 10.122.131.6.63650: Flags [P.], seq 853:7993, ack 166, win 9836, options [nop,nop,TS val 3837365558 ecr 3150705561], length 7140
10:08:58.195417 IP 192.168.227.78.443 > 10.122.131.6.63650: Flags [P.], seq 7993:13705, ack 166, win 9836, options [nop,nop,TS val 3837365558 ecr 3150705561], length 5712
10:08:58.195647 IP 10.122.131.6.63650 > 192.168.227.78.443: Flags [.], ack 13705, win 986, options [nop,nop,TS val 3150705565 ecr 3837365558], length 0
10:08:58.195666 IP 192.168.227.78.443 > 10.122.131.6.63650: Flags [P.], seq 13705:27281, ack 166, win 9836, options [nop,nop,TS val 3837365559 ecr 3150705565], length 13576
10:08:58.195712 IP 10.122.131.6.63650 > 192.168.227.78.443: Flags [.], ack 27281, win 986, options [nop,nop,TS val 3150705565 ecr 3837365559], length 0
10:08:58.195831 IP 10.122.131.6.63650 > 192.168.227.78.443: Flags [P.], seq 166:214, ack 27281, win 986, options [nop,nop,TS val 3150705565 ecr 3837365559], length 48
10:08:58.196475 IP 192.168.227.78.443 > 10.122.131.6.63650: Flags [P.], seq 27281:27312, ack 214, win 9836, options [nop,nop,TS val 3837365560 ecr 3150705565], length 31
10:08:58.196879 IP 10.96.0.1.443 > 192.168.227.78.41420: Flags [P.], seq 812:882, ack 761, win 501, options [nop,nop,TS val 3691776039 ecr 55164057], length 70
10:08:58.196966 IP 10.96.0.1.443 > 192.168.227.78.41420: Flags [P.], seq 882:1615, ack 761, win 501, options [nop,nop,TS val 3691776039 ecr 55164057], length 733
10:08:58.197392 IP 10.122.131.6.63650 > 192.168.227.78.443: Flags [P.], seq 214:417, ack 27312, win 986, options [nop,nop,TS val 3150705566 ecr 3837365560], length 203
10:08:58.198444 IP 192.168.227.78.41420 > 10.96.0.1.443: Flags [.], ack 1615, win 6097, options [nop,nop,TS val 55164061 ecr 3691776039], length 0
10:08:58.198569 IP 192.168.227.78.41420 > 10.96.0.1.443: Flags [P.], seq 761:796, ack 1615, win 6097, options [nop,nop,TS val 5516406

For refrence :

clickedge-master1:~$ kubectl get po -A -o wide | grep hub
clusternet-system   clusternet-hub-5d94d8b586-9rpsg                  1/1     Running   0              4h16m   192.168.227.77   clickedge-master1   <none>           <none>
clusternet-system   clusternet-hub-5d94d8b586-svdw8                  1/1     Running   0              4h16m   192.168.227.79   clickedge-master1   <none>           <none>
clusternet-system   clusternet-hub-5d94d8b586-vp7bx                  1/1     Running   0              4h16m   192.168.227.78   clickedge-master1   <none>           <none>
clickedge-master1:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
clusterip    ClusterIP   10.100.185.107   <none>        80/TCP         179m
hostnames    ClusterIP   10.98.183.102    <none>        80/TCP         168m
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        5h12m
nodeport     NodePort    10.98.37.96      <none>        80:32659/TCP   175m

[dixudx]

@TanyaSingh-espina Would you please share the logs from kube-apiserver and clusternet-hub? Also what's your Kubernetes cluster version? Are there any special settings when you setup the Kubernetes cluster?

[TanyaSingh-espina]

Kube-apiserver logs :

I0430 05:39:10.611123       1 handler.go:275] Adding GroupVersion clusters.clusternet.io v1beta1 to ResourceManager
W0430 05:39:11.341070       1 handler_proxy.go:93] no RequestInfo found in the context
E0430 05:39:11.341125       1 controller.go:113] loading OpenAPI spec for "v1alpha1.shadow" failed with: Error, could not get list of group versions for APIService
W0430 05:39:11.341070       1 handler_proxy.go:93] no RequestInfo found in the context
I0430 05:39:11.341301       1 controller.go:126] OpenAPI AggregationController: action for item v1alpha1.shadow: Rate Limited Requeue.
E0430 05:39:11.341362       1 controller.go:102] loading OpenAPI spec for "v1alpha1.shadow" failed with: failed to download v1alpha1.shadow: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0430 05:39:11.342516       1 controller.go:109] OpenAPI AggregationController: action for item v1alpha1.shadow: Rate Limited Requeue.
W0430 05:39:11.348997       1 handler_proxy.go:93] no RequestInfo found in the context
E0430 05:39:11.349088       1 controller.go:113] loading OpenAPI spec for "v1alpha1.proxies.clusternet.io" failed with: Error, could not get list of group versions for APIService
I0430 05:39:11.349102       1 controller.go:126] OpenAPI AggregationController: action for item v1alpha1.proxies.clusternet.io: Rate Limited Requeue.
W0430 05:39:11.348997       1 handler_proxy.go:93] no RequestInfo found in the context
E0430 05:39:11.349169       1 controller.go:102] loading OpenAPI spec for "v1alpha1.proxies.clusternet.io" failed with: failed to download v1alpha1.proxies.clusternet.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0430 05:39:11.350268       1 controller.go:109] OpenAPI AggregationController: action for item v1alpha1.proxies.clusternet.io: Rate Limited Requeue.

-----

E0430 05:39:12.861601       1 available_controller.go:460] v1alpha1.shadow failed with: failing or missing response from https://10.96.124.7:443/apis/shadow/v1alpha1: Get "https://10.96.124.7:443/apis/shadow/v1alpha1": dial tcp 10.96.124.7:443: connect: connection refused                                                                                                                               E0430 05:39:12.862339       1 available_controller.go:460] v1alpha1.proxies.clusternet.io failed with: failing or missing response from https://10.96.124.7:443/apis/proxies.clusternet.io/v1alpha1: Get "https://10.96.124.7:443/apis/proxies.clusternet.io/v1alpha1": dial tcp 10.96.124.7:443: connect: connection refused                                                                                  W0430 05:39:12.862440       1 handler_proxy.go:93] no RequestInfo found in the context
E0430 05:39:12.863674       1 available_controller.go:460] v1alpha1.shadow failed with: failing or missing response from https://10.96.124.7:443/apis/shadow/v1alpha1: Get "https://10.96.124.7:443/apis/shadow/v1alpha1": dial tcp 10.96.124.7:443: connect: connection refused                                                                                                                               



-----
al tcp 10.96.124.7:443: connect: connection refused
I0430 05:39:19.349760       1 handler.go:275] Adding GroupVersion proxies.clusternet.io v1alpha1 to ResourceManager                  I0430 05:39:19.350579       1 handler.go:275] Adding GroupVersion shadow v1alpha1 to ResourceManager
I0430 05:43:06.184342       1 controller.go:624] quota admission added evaluator for: managedclusters.clusters.clusternet.io
I0430 05:43:06.244778       1 controller.go:624] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0430 05:43:06.265903       1 controller.go:624] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
E0430 05:44:12.181352       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/proxies.clusternet.io/v1alpha1/sockets/8ed1a7cf-6abb-4c0e-93f7-efaf8c34d770" audit-ID="17cc0b42-148c-4431-8c37-0c5ad5bad5d3"
W0430 05:55:52.623639       1 handler_proxy.go:93] no RequestInfo found in the context
E0430 05:55:52.624708       1 controller.go:146] Error updating APIService "v1alpha1.proxies.clusternet.io" with err: failed to download v1alpha1.proxies.clusternet.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0430 05:55:52.625119       1 available_controller.go:460] v1alpha1.proxies.clusternet.io failed with: failing or missing response from https://10.96.124.7:443/apis/proxies.clusternet.io/v1alpha1: bad status from https://10.96.124.7:443/apis/proxies.clusternet.io/v1alpha1: 503

---------
E0430 10:03:35.112975       1 watch.go:287] unable to encode watch object *v1.WatchEvent: http2: stream closed (&streaming.encoderWithAllocator{writer:responsewriter.outerWithCloseNotifyAndFlush{UserProvidedDecorator:(*metrics.ResponseWriterDelegator)(0xc0050082d0), InnerCloseNotifierFlusher:struct { httpsnoop.Unwrapper; http.ResponseWriter; http.Flusher; http.CloseNotifier; http.Pusher }{Unwrapper:(*httpsnoop.rw)(0xc0029d80a0), ResponseWriter:(*httpsnoop.rw)(0xc0029d80a0), Flusher:(*httpsnoop.rw)(0xc0029d80a0), CloseNotifier:(*httpsnoop.rw)(0xc0029d80a0), Pusher:(*httpsnoop.rw)(0xc0029d80a0)}}, encoder:(*versioning.codec)(0xc00382e460), memAllocator:(*runtime.Allocator)(0xc009bc0078)})
E0430 10:32:43.718996       1 upgradeaware.go:439] Error proxying data from backend to client: write tcp 10.122.131.6:6443->10.122.131.6:36500: write: broken pipe
E0430 10:33:14.802764       1 upgradeaware.go:425] Error proxying data from client to backend: write tcp 10.122.131.6:38602->10.122.131.6:10250: write: broken pipe
E0430 10:44:44.479601       1 upgradeaware.go:439] Error proxying data from backend to client: write tcp 10.122.131.6:6443->10.122.131.6:44118: write: broken pipe
E0430 10:44:44.479781       1 upgradeaware.go:425] Error proxying data from client to backend: write tcp 10.122.131.6:47694->10.122.131.6:10250: write: broken pipe   

Clusternet-hub logs :

E0430 05:55:58.479687       1 writers.go:122] apiserver was unable to write a JSON response: http2: stream closed
E0430 05:55:58.479922       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0430 05:55:58.480109       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/shadow/v1alpha1" audit-ID="47fb736b-011d-4be4-8a7b-3e70297c2955"
E0430 05:55:58.481633       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0430 05:55:58.483918       1 timeout.go:142] post-timeout activity - time-elapsed: 3.711941ms, GET "/apis/shadow/v1alpha1" result: <nil>
E0430 05:56:28.387531       1 writers.go:122] apiserver was unable to write a JSON response: http2: stream closed
E0430 05:56:28.387637       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed

------
E0430 05:56:58.377191       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed                                                                                                 E0430 05:56:58.379501       1 timeout.go:142] post-timeout activity - time-elapsed: 30.247122ms, GET "/apis/shadow/v1alpha1" result: <nil>
E0430 05:56:58.380718       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout          E0430 05:56:58.381955       1 timeout.go:142] post-timeout activity - time-elapsed: 33.003551ms, GET "/apis/shadow/v1alpha1" result: <nil>
E0430 05:56:58.383191       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0430 05:56:58.392251       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0430 05:56:58.397702       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0430 05:56:58.409136       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0430 05:56:58.422135       1 timeout.go:142] post-timeout activity - time-elapsed: 63.102871ms, GET "/apis/shadow/v1alpha1" result: <nil>
E0430 05:56:58.440208       1 timeout.go:142] post-timeout activity - time-elapsed: 81.095062ms, GET "/apis/shadow/v1alpha1" result: <nil>
E0430 05:56:58.443085       1 writers.go:122] apiserver was unable to write a JSON response: http2: stream closed
E0430 05:56:58.443143       1 timeout.go:142] post-timeout activity - time-elapsed: 83.81419ms, GET "/apis/shadow/v1alpha1" result: <nil>
E0430 05:56:58.443298       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/shadow/v1alpha1" audit-ID="93f899e2-55b3-4255-9e21-c7c2e65322b5"
E0430 05:56:58.461930       1 timeout.go:142] post-timeout activity - time-elapsed: 102.552189ms, GET "/apis/shadow/v1alpha1" result: <nil>
E0430 05:56:58.462592       1 writers.go:122] apiserver was unable to write a JSON response: http2: stream closed
E0430 05:56:58.462633       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/shadow/v1alpha1" audit-ID="b9a97584-7ec5-416b-9054-edf2ea99fa21"                                                                          ~                                  

[TanyaSingh-espina]

Are there any special settings when you setup the Kubernetes cluster?
-- No, special settings, I created the cluster using kubeadm on azure vms, and calico as CNI.