crd permission fail about multus-cni

[Elegant996]

Could you elaborate a bit more on what permissions you needed to change and where? Thanks!

[snowmansora]

@kfox1111 Thank you for creating this issue and sharing your solution. I encountered this issue as well, and I spent hours figuring out why, but failed (I think there is no related error/warning log...)

@Elegant996
When I try to run below on one of my kube nodes:

export KUBECONFIG=<kubeconfig in cni-conf.json>; kubectl get crd

It returns:

Error from server (Forbidden): customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:node:<node>" cannot list customresourcedefinitions.apiextensions.k8s.io at the cluster scope

My lazy-solution / quick-workaround is to run the following command for all my kube nodes:

kubectl create clusterrolebinding cluster-admin-binding-<node> --clusterrole=cluster-admin --user=system:node:<node>

I believe there is a much better solution than granting cluster admin permission like above... but I don't know how to configure that since my knowledge on Kubernetes security is very limited.
p.s. since I granted the cluster admin permission, addition permissions may be required if someone want to configure it properly

I +1 on the behaviour where the pod should be updated to fail in this case.

[kfox1111]

@Elegant996 I was in the same boat as @snowmansora when I tested things. Things were broken until I made the node an admin. (Totally the wrong thing to do) That is a separate issue. How to properly configure RBAC for Multus? But this issue is about the behavior where it just did the wrong thing when the permissions were not correct rather then erroring the pod setup.

[dougbtv]

I believe that this might be a documentation issue? There's a possibility that the readme here might need to suggest how one might add the proper RBAC for Multus to access the API.

I'd be happy to contribute a section on this, however I'd definitely like some input on proper permissions for the correct endpoints instead of an over-arching and overly permissive RBAC.

Currently the solution I've been using personally looks rather similar, for example, I create a cluster role such as this, from this source:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: multus-crd-overpowered
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'

And then create a binding such as:

kubectl create clusterrolebinding multus-node-thehostname
    --clusterrole=multus-crd-overpowered
    --user=system:node:thehostname

For each node in the cluster (including the master). As can be seen in these ansible plays.

[snowmansora]

I think a documentation update on a proper RBAC setup for Multus will certainly benefits others.

I also think a behaviour update as suggested by @kfox1111 also make sense.
To clarify, I specified a CRD annotation of what networks I want to use for Multus on my pod's definition, the pod created should match my definition, if not, it should fail. (It is like if I specified some memory limit for my pod, if that cannot be met, pod should fail.)

@dougbtv
FYI, I am using your Ansible playbook to play around with K8s and Multus. (By the way, thank you so much for sharing your playbook, it has been great trying stuffs with it).
The reason I encountered the permission problem even with your "multus-crd-overpowered" is because I used IP in my Ansible inventory, hence "inventory_hostname" in multus-crd/tasks/main.yml is IP. But K8s is expecting hostname, so after I replaced "inventory_hostname" with "ansible_nodename", it works.

[uabfra]

Back to the original issue for this, If something goes wrong with the kubeconfig file, access to crd, annotation for networks that dont exists. Dont fallback to the default network, because it is not what the person wanted for the pod, I have created an issue #45 with a patch that fixes this.

[dougbtv]

I believe this will be fixed with #100