divided resource to child cluster failed about clusternet

[dixudx]

@Sad-polar-bear Please specify the Kubernetes version, Clusternet version, and clusternet-agent parameters.

It will be better to have hub logs for further investigating.

[Sad-polar-bear]

@Sad-polar-bear Please specify the Kubernetes version, Clusternet version, and clusternet-agent parameters.

It will be better to have hub logs for further investigating.

I0909 02:56:28.142658 1 description.go:299] start processing Description "clusternet-6phvq/app-demo-generic"
I0909 02:56:28.142661 1 description.go:299] start processing Description "clusternet-6phvq/app-demo-generic"
I0909 02:56:28.142674 1 description.go:272] successfully synced Description "clusternet-6phvq/app-demo-generic"
I0909 02:56:28.142676 1 base.go:273] successfully synced Base "clusternet-6phvq/app-demo"
I0909 02:56:28.142719 1 base.go:300] start processing Base "clusternet-6phvq/app-demo"
E0909 02:56:28.199450 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.199451 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.204895 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.205514 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.209844 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.210949 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.215334 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.215784 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.220864 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.221341 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.228016 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.228018 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.274766 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.274766 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.279825 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.280915 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.284823 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.285804 1 memcache.go:179] couldn't get current server API group list: Unauthorized
I0909 02:56:28.328628 1 request.go:600] Waited for 585.961744ms due to client-side throttling, not priority and fairness, request: PATCH:https://172.18.0.1:443/apis/apps.clusternet.io/v1alpha1/namespaces/clusternet-reserved/manifests/deployments-foo-my-nginx
I0909 02:56:28.528606 1 request.go:600] Waited for 785.925299ms due to client-side throttling, not priority and fairness, request: PATCH:https://172.18.0.1:443/apis/apps.clusternet.io/v1alpha1/namespaces/clusternet-reserved/manifests/namespaces-foo
E0909 02:56:28.541513 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.545129 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.547233 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.550021 1 memcache.go:179] couldn't get current server API group list: Unauthorized
E0909 02:56:28.557486 1 memcache.go:179] couldn't get current server API group list: Unauthorized

[Sad-polar-bear]

clusternet-hub configuration:

• /usr/local/bin/clusternet-hub
  - --secure-port=443
  - --feature-gates=SocketConnection=true,Deployer=true,ShadowAPI=true,FeedInUseProtection=true
  - -v=4

clusternet-agent configuration:

• /usr/local/bin/clusternet-agent
  - --cluster-reg-token=$(REG_TOKEN)
  - --cluster-reg-parent-url=$(PARENT_URL)
  - --cluster-sync-mode=Dual
  - --feature-gates=SocketConnection=true,AppPusher=true
  - -v=4
  - --cluster-reg-name=liuer-test

[dixudx]

couldn't get current server API group list

This is usually caused by api listing, i.e., https://10.96.0.1:443/api?timeout=32s.

@Sad-polar-bear Since clusternet-agent is running in-cluster, now please do below check,

$ kubectl exec -it -n clusternet-system clusternet-agent-xxxx-xxx sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
~ # env | grep KUBERNETES_SERVICE_HOST
KUBERNETES_SERVICE_HOST=10.96.0.1

You may get a different value for KUBERNETES_SERVICE_HOST. Now please check whether this address, such as https://10.96.0.1:443, is accessible in this agent container.

~ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
v3.13.6-10-gf6674f13d1 [https://dl-cdn.alpinelinux.org/alpine/v3.13/main]
v3.13.6-12-g1c57206c4d [https://dl-cdn.alpinelinux.org/alpine/v3.13/community]
OK: 13895 distinct packages available
~ # apk add curl
(1/5) Installing ca-certificates (20191127-r5)
(2/5) Installing brotli-libs (1.0.9-r3)
(3/5) Installing nghttp2-libs (1.42.0-r1)
(4/5) Installing libcurl (7.78.0-r0)
(5/5) Installing curl (7.78.0-r0)
Executing busybox-1.32.1-r6.trigger
Executing ca-certificates-20191127-r5.trigger
OK: 8 MiB in 20 packages
~ # curl -k https://10.96.0.1:443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}

It is desired to get above 403 status code.

Here you can use a mirror repo for faster installation in China. Please follow this guide.

[Sad-polar-bear]

curl -k https://

[dixudx]

@Sad-polar-bear Would please have a check on ManagedCluster.

$ kubectl get mcls -n clusternet-xxxx clusternet-cluster-xxxx -o yaml

[Sad-polar-bear]

@Sad-polar-bear Would please have a check on ManagedCluster.

$ kubectl get mcls -n clusternet-xxxx clusternet-cluster-xxxx -o yaml

➜  ~ kubectl get mcls -n clusternet-6phvq liuer-vc -oyaml
apiVersion: clusters.clusternet.io/v1beta1
kind: ManagedCluster
metadata:
  creationTimestamp: "2021-09-09T02:45:03Z"
  generation: 1
  labels:
    clusternet.io/created-by: clusternet-agent
    clusters.clusternet.io/cluster-id: 00216258-eaaa-4f1c-a810-8e5c58f9ca9e
    clusters.clusternet.io/cluster-name: liuer-vc
  managedFields:
  - apiVersion: clusters.clusternet.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:clusternet.io/created-by: {}
          f:clusters.clusternet.io/cluster-id: {}
          f:clusters.clusternet.io/cluster-name: {}
      f:spec:
        .: {}
        f:clusterId: {}
        f:clusterType: {}
        f:syncMode: {}
    manager: clusternet-hub
    operation: Update
    time: "2021-09-09T02:45:03Z"
  - apiVersion: clusters.clusternet.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        .: {}
        f:allocatable:
          .: {}
          f:cpu: {}
          f:memory: {}
        f:apiserverURL: {}
        f:appPusher: {}
        f:capacity:
          .: {}
          f:cpu: {}
          f:memory: {}
        f:healthz: {}
        f:k8sVersion: {}
        f:lastObservedTime: {}
        f:livez: {}
        f:nodeStatistics:
          .: {}
          f:readyNodes: {}
        f:parentAPIServerURL: {}
        f:platform: {}
        f:readyz: {}
        f:useSocket: {}
    manager: clusternet-agent
    operation: Update
    time: "2021-09-10T03:09:51Z"
  name: liuer-vc
  namespace: clusternet-6phvq
  resourceVersion: "7929832888"
  selfLink: /apis/clusters.clusternet.io/v1beta1/namespaces/clusternet-6phvq/managedclusters/liuer-vc
  uid: f8adc0c5-8f25-4658-97e9-873bf9823fe3
spec:
  clusterId: 00216258-eaaa-4f1c-a810-8e5c58f9ca9e
  clusterType: EdgeCluster
  syncMode: Dual
status:
  allocatable:
    cpu: 31640m
    memory: 58909160Ki
  apiserverURL: https://172.18.0.1:443
  appPusher: true
  capacity:
    cpu: "32"
    memory: 64643560Ki
  healthz: true
  k8sVersion: v1.18.4-tke.12
  lastObservedTime: "2021-09-10T03:09:31Z"
  livez: true
  nodeStatistics:
    readyNodes: 4
  parentAPIServerURL: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  platform: linux/amd64
  readyz: true
  useSocket: true

[dixudx]

And have you deployed all the needed roles and rolebings in child clusters?

$ kubectl get clusterrole | grep clusternet
$ kubectl get clusterrolebing | grep clusternet
$ kubectl get role -n clusternet-system | grep clusternet
$ kubectl get rolebinding -n clusternet-system | grep clusternet

[Sad-polar-bear]

kubectl get rolebinding -n clusternet-system | grep clusternet

oh, and how can I deploy the clusterrolebing?

[dixudx]

And check the credentials in your hub cluster.

$ kubectl get secret child-cluster-deployer -n clusternet-xxxx

Whether this secret is created and check the data service-account.name and service-account.uid in this secret. Both values are base64 encoded, you can decode it and check this ServiceAccount in your child cluster.

[dixudx]

how can I deploy the clusterrolebing?

Just follow this guide https://github.com/clusternet/clusternet#deploying-clusternet-agent-in-child-cluster

[Sad-polar-bear]

And check the credentials in your hub cluster.

$ kubectl get secret child-cluster-deployer -n clusternet-xxxx

Whether this secret is created and check the data service-account.name and service-account.uid in this secret. Both values are base64 encoded, you can decode it and check this ServiceAccount in your child cluster.

decode result:
service-account.name: clusternet-app-deployer
service-account.uid: bfad57cc-f889-4bd2-9db6-6b51c354415e

[Sad-polar-bear]

@Sad-polar-bear Please specify the Kubernetes version, Clusternet version, and clusternet-agent parameters.

It will be better to have hub logs for further investigating.

k8s version: v1.18.4-tke.12
clusternet: 0.4.0

[dixudx]

After debugging, we found v1.18.4-tke.12 had set extra restrictions on authentication and authorization, which did not allow anonymous accessing. @Sad-polar-bear Please try to use other Kubernetes distros, such as official release Kubernetes.

@huxiaoliang @DanielXLee Please keep an eye on this.

[huxiaoliang]

@Sad-polar-bear please remove --basic-auth-file from api-server and try it again

[Sad-polar-bear]

remove --basic-auth-file and working

[DanielXLee]

@Sad-polar-bear I also encountered this problem, I try to remove --basic-auth-file from the master CR, and delete the old api-server pod, seems not to work. Did you have any other steps?

[dixudx]

Please remove --basic-auth-file from apiserver running in the parent cluster.