postfinance / kubelet-csr-approver Goto Github PK

Hey, I checked images on dockerhub but there is no tag information. I think a tag should exist to use a specific version in values.YAML. What can we do about it?

It would be handy if the helm chart contained an optional network policy to restrict what this container can access over the network.

I'm experiencing troubles with kubelet-csr-approver without passing the option --bypass-dns-resolution=true

"Denying kubelet-serving CSR. DNS checks failed. Reason:The SAN DNS Name could not be resolved, denying the CSR"

$ dig A k8snode03u @10.96.0.10
$ dig A k8snode03u.cluster.local @10.96.0.10

Doesn't resolve to the node IP address.

Am I doing something wrong on my cluster?
Is having core-dns resolving the node's hostname a usually expected behavior?
How do you enable such feature on your cluster?

Since Docker Hub has been applying rate limits on image pulls for a while now, and their more recent move to scrap free teams I feel that Docker Hub is becoming more and more hostile towards the OSS/hobbyist community.

I would like to propose providing GitHub Container Registry as an alternative (or completely switching over to it) for the project.
Would you be open to this?

Thank you for considering!

kubeadm registers Kubernetes nodes by their canonical hostname instead of the FQDN, making it hard to find a regex that universally matches and provides more security than .*. Would it be an option to verify legitimate certificate requests against a list of allowed IP subnets?

The CSR contains the node IP address:

$ kubectl get csr csr-5pq5z -o jsonpath='{.spec.request}' |base64 -d |openssl req -noout -text
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: O = system:nodes, CN = system:node:n0218
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:5e:a7:2e:34:53:bf:88:71:f7:e5:f1:26:63:57:
                    be:0b:db:d1:68:87:ae:17:a5:e7:5d:45:16:e6:f7:
                    df:7f:64:d6:bc:b1:e9:c7:a5:4c:5b:c9:91:31:89:
                    26:82:8c:cc:70:f2:1c:ec:45:8f:e6:a2:81:b8:f2:
                    80:ec:09:1e:8b
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        Attributes:
        Requested Extensions:
            X509v3 Subject Alternative Name: 
                DNS:n0218, IP Address:192.168.8.58
    Signature Algorithm: ecdsa-with-SHA256
         30:45:02:21:00:d5:17:3e:d4:56:04:c4:4c:b5:ff:47:3c:1e:
         98:92:d1:20:e3:52:39:28:34:2a:eb:c3:53:93:b6:34:0d:93:
         4b:02:20:0c:d4:a2:68:f0:6e:22:e8:ca:f6:c9:27:78:55:bb:
         7c:5a:4c:01:af:78:42:1f:2e:b7:5a:39:a9:b5:36:93:85

Hi & thanks for the project.
I've seen it’s possible to set additional Annotations to pods via podAnnotations.
Would be great if this was also the case for labels – eg podLabels.
Thanks again for the work.

Add support for providerIpPrefixes to the Helm chart.

See #28 (comment)

helm install kubelet-csr-approver kubelet-csr-approver/kubelet-csr-approver -n kube-system \
  --set providerRegex='^cn-[a-z]+\.192\.168\.((2(5[0-5]|[0-4]\d))|[0-1]?\d{1,2})\.((2(5[0-5]|[0-4]\d))|[0-1]?\d{1,2})$' \
  --set providerIpPrefixes='192.168.0.0/16' \
  --set maxExpirationSeconds='86400'

return results: Error: INSTALLATION FAILED: failed parsing --set data: error parsing index: strconv.Atoi: parsing "0-5": invalid syntax

I want to match: cn-xxxxx.192.168.xxx.xxx

It is important to understand that the node DNS name needs to be resolvable for the kubelet-csr-approver to work properly. If this is an issue for you, please file an issue and I'll add a flag to disable this validation.

This is an issue for me :)

Kubernetes node names don't have to be resolvable, and mine are not.

Fwiw, I use /etc/machine-id as the k8s node name (via systemd %m substitution), uname -n returns localhost on all my hosts, and none of my node IPs are in DNS (nor even have static IPs across reboots). About the only thing they have is a TLS private key, and a cert that links that to their k8s node name - they use their client cert to update their Node entry with their current IP - and apiserver->node always uses IP anyway.

In the Readme it is mentioned :
It is important to understand that the node DNS name needs to be resolvable for the kubelet-csr-approver to work properly. If this is an issue for you, please file an issue and I'll add a flag to disable this validation.

For some situations like local clusters where there is no DNS server to map node to ip, this type of requirement makes CSR leave Denied. So adding a related flag to disable this validation makes sense.

With the change from master to control-plane Kubernetes 1.25 entirely removes the taint node-role.kubernetes.io/master (see CHANGELOG).

Acceptance criteria:

• Remove node-role.kubernetes.io/master taint

We explicity change the hostname of the box to a different naming convention the certificates get rejected because of this reason.

The SAN DNS Name in the x509 CSR is not prefixed by the node name (hostname)

Hello,

I am trying to use a chart called stackgres on the same cluster as kubelet-csr-approver, but it fails to install because kubelet-csr-approver automatically denies its certificate request.

Revelant log line:

2022-06-11T13:51:08.348Z        INFO    controller.certificatesigningrequest    controller/csr_controller.go:107        Denying kubelet-serving CSR. Reason:CSR Spec.Username is not prefixed with system:node: {"reconciler group": "certificates.k8s.io", "reconciler kind": "CertificateSigningRequest", "name": "stackgres-operator", "namespace": ""}

Would it possible to have an option to only process CSR with a given prefix, like system:node:, and leave others untreated as they may have a different usage pattern?

Thank you

Hello! I'd really appreciate it if you'd add a flag to disable the DNS name resolvability validation.
RegExp alone is probably good enough for the testing needs in absence of properly configured DNS server.

:)

Hi, for some odd reason I have several kubelet-csr-approver pods in not running state. This only happens for its deployment, all other deployments do not have this behaviour.

Any idea what may be causing this ?

Grabbing https://github.com/postfinance/kubelet-csr-approver/blob/v1.0.4/deploy/k8s/deployment.yaml (note: v1.0.4) will run the image postfinance/kubelet-csr-approver:v1.0.1

The Helm chart still contains the pre-release version v0.2.0-1 instead of the final v0.2.0:

version: v0.2.0-1

When pulling this chart from a Git Repo (for situations where a Helm Repo is not available), the Chart.yaml always shows version and appVersion as 0.0.1-dummy instead of i.e. 1.0.7

Could Chart.yaml be updated at release times to correctly state the version?

• (Implement and) document checks on the .spec.expirationSeconds field for a kubelet CertificateSigningRequest
  • for example, only allow certificates to be issued with a duration of 86400 seconds (24 hours) by default
  • maybe a command line option to allow issuing certificates that last longer than this?

https://github.com/postfinance/kubelet-csr-approver/blob/main/deploy/k8s/deployment.yaml

│   Normal   Pulling    102s (x4 over 3m9s)  kubelet            Pulling image "pf/kubelet-csr-approver:0.1.0"                                                              │
│   Warning  Failed     101s (x4 over 3m7s)  kubelet            Failed to pull image "pf/kubelet-csr-approver:0.1.0": rpc error: code = Unknown desc = Error response from │
│  daemon: pull access denied for pf/kubelet-csr-approver, repository does not exist or may require 'docker login'                                                         │
│   Warning  Failed     101s (x4 over 3m7s)  kubelet            Error: ErrImagePull                                                                                        │
│   Warning  Failed     86s (x6 over 3m7s)   kubelet            Error: ImagePullBackOff                                                                                    │
│   Normal   BackOff    72s (x7 over 3m7s)   kubelet            Back-off pulling image "pf/kubelet-csr-approver:0.1.0"  

Hi,

could you please add an option to bypass or tune SAN validation?
Currently CSR are being denied with the following reason:

> kubectl get csr csr-zwzjd -ojson | jq '.status.conditions[] | {message,reason,type}' 
{
  "message": "CSR not complying with kubelet-csr-approver validation process. Reason: The x509 Cert Request contains more than 1 DNS name",
  "reason": "kubelet-serving cert denied",
  "type": "Denied"
}

The SAN part of the request looks like:

                X509v3 Subject Alternative Name: 
                    DNS:ec2-1-2-3-4.eu-central-1.compute.amazonaws.com, DNS:ip-192-168-5-6.eu-central-1.compute.internal, IP Address:192.168.5.6, IP Address:1.2.3.4

I am not aware of any way to influence that, even though in my case public names/IPs are not needed. The easiest would be to have an option to bypass this check or tune the allowed value, ie. making it greater than 1.

helm chart version: 1.0.3

I enabled the serviceMonitor in helm values:

metrics:
  serviceMonitor:
    enabled: true

But when I upgrade the helm release it failed due to invalid syntax in servicemonitor template:

Error from server (BadRequest): error when creating "servicemonitor.yaml": ServiceMonitor in version "v1" cannot be handled as a ServiceMonitor: strict decoding error: unknown field "spec.selector.app.kubernetes.io/instance", unknown field "spec.selector.app.kubernetes.io/name"

The ServiceMonitor "release-name-kubelet-csr-approver" is invalid: spec.endpoints[0].port: Invalid value: "integer": spec.endpoints[0].port in body must be of type string: "integer"

When BypassDNSResolution == true The DNSCheck function seems to bail out before iterating over the CSR DNSNames, completely skipping the regex checks too.

Shouldn't this flag only skip the r.DNSResolver.LookupHost step?

More details here: kubernetes-sigs/controller-runtime#1442

It seems a race condition exists within controller-runtime. It might be important to have a look at it and determine if this could impact us, and if yes, whether we should change our approach.

Also, this breaks tests running with the -race argument, disabling it for the moment.

Hi !

I would like to know if there is a solution to solve the problem "The SAN DNS Name could not be resolved" without using the bypassDnsResolution argument.

This is my entire error : certificate request was not signed: cannot watch on the certificate signing request: certificate signing request is denied, reason: kubelet-serving cert denied, message: CSR not complying with kubelet-csr-approver validation process. Reason: The SAN DNS Name could not be resolved, denying the CSR

Thanks in advance !

What should the feature do:
Link the kubelet-csr-approver helmchart to the central registry under https://artifacthub.io/

Should be not much effort: https://artifacthub.io/docs/topics/repositories/#helm-charts-repositories

What is use case behind this feature:
More visibility

Hi, would it be possible to add a nodeselector to make the deployment to run on the master nodes just like Kubelet Rubber Stamp?
Thanks!

It'd be nice to add support for v1beta1 API version. Currently project only works with v1 version.

See #45 (comment) for details.

hey,

if my assumption is correct after I read the source code, it is possible to add extra SANs to request and only the first one will be verified, technically I can still add auth.company.com into the list of SANs per threat model, amirite?

cheers

Thank you. Great work. I found a problem with helm. Even I try 'helm repo update'.

Error: chart "kubelet-csr-approver" matching 0.2.6 not found in kubelet-csr-approver index. (try 'helm repo update'): no chart version found for kubelet-csr-approver-0.2.6
Error: chart "kubelet-csr-approver" matching 0.2.5 not found in kubelet-csr-approver index. (try 'helm repo update'): no chart version found for kubelet-csr-approver-0.2.5

We have a Kubernetes integration with Datadog 7.44.0 on AWS EKS clusters, to monitor resources and performance of different applications. Everything was working fine in EKS version 1.23 and earlier. However, after upgrading one of the clusters (dev) to EKS version 1.24, the integration stopped working.

Upon investigation, we realized that the problem seemed to be with the certificates generated by Datadog. EKS does not recognize their origin, and as a result, the certificates are left in a "pending" state. Being a development environment, daily it is turned off at night and turned on in the morning, by doing this the integration with Datadog is lost, and the command kubectl get csr | awk '{print $1}' | xargs kubectl certificate approve to manually approve the certificates and restore the integration.

AWS in its documentation indicates that a driver must be developed in order to automate the approval of certificates.
https://docs.aws.amazon.com/eks/latest/userguide/cert-signing.html#csr-considerations

Kubernetes recommends in its documentation to use a ready-built controller.
Third party custom controllers can be used: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubelet-serving-certs

1. First Experimentation:
   On the development ('dev') cluster, we had a node with a CSR in a 'Pending' state.

Then when installing the driver, the certificate is approved and issued

2. Second Experimentation:
   However, if the kubelet-csr-approver controller is already installed on the 'dev' cluster and then a new node is added to the cluster, the controller is able to approve the CSR of the new node, but does not issue the corresponding certificate. In this case, the CSR remains in the 'Approved' state, but does not change to 'Approved, Issued', this causes the integration of the new node with datadog to be unsuccessful.

We appreciate any help

I'm using the default options in deploy/k8s and only overriding the KUBERNETES_SERVICE_{HOST,PORT} environment variables but getting the following error about a minute or so after the pod starts:

E1211 19:49:34.608041       1 leaderelection.go:332]
{
"level":"ERROR",
"ts":"2023-12-11T19:44:49.789Z",
"logger":"controller-runtime.source.EventHandler",
"caller":"source/kind.go:68",
"msg":"failed to get informer from cache",
"error":"failed to get API group resources: 
      unable to retrieve the complete list of server APIs: 
          certificates.k8s.io/v1: 
              Get \"https://API_SERVER:6443/apis/certificates.k8s.io/v1\": 
                 dial tcp: lookup API_SERVER: i/o timeout"
}

I know that pods are able to talk to the api server because I have a running deployment of kube-state-metrics that also overrides the same env vars with the same values.

I'm running kubernetes 1.28.3 if that's helpful.

Enable to set log level in values.yml. In this way, we can see more detailed log.

After updating to 1.2.0, the following error message appeared and CSRs were not approved.

Denying kubelet-serving CSR. DNS checks failed. Reason:One of the SAN IP addresses, xxx.xxx.xxx.xxx, is not contained in the set of resolved IP addresses, denying the CSR.

In my environment, the node name cannot be resolved by DNS, so I specify --bypass-dns-resolution.

I checked the source, and the IP addresses of the SAN are checked against resolvedIPSet here, however
resolvedIPSet contains only the IP addresses resolved here by DNS.

So, I think the check will always fail if --bypass-dns-resolution is specified.

Thanks for the custom controller to auto approve the csr's. The current functionality of the project doesn't support older version of K8's v 1.18) as it leverages /v1beta1 k8's apis.

Therefore, I would like to request for an enhancement to optimize the current functionality to support v1beta1 apis.