Kong pods get replaced on every update to helm release when admission controller is enabled about charts

[hbagdi]

It would be nice to have an option to provide your own CA/cert for the admission webhook to prevent this unnecessary Kong restart.

This was possible before and it should be possible to provide your own cert/key. If not, that is a bug.

[rainest]

This is more or less the same as our issue with the Postgres password secret. The cert is randomly generated, and Helm generates random resources every time the template is rendered. We do need to replace the pods when this occurs, as otherwise they'll try to unsuccessfully use the old cert.

Historically, using a pre-install hook to generate the cert Secret was the only workaround for this, but that comes with the caveat that the Secret is no longer part of the release and doesn't get cleaned up if the release is deleted. The proposed solution is to add functionality that ignores resource changes if the resource has some annotation, but there's no work in progress for that: helm/helm#5290 (comment)

https://github.com/helm/charts/issues/5167#issuecomment-619137759 indicates that Helm 3.1's lookup function provides a means to work around this, but using this would break Helm 2.x compatibility. We do plan to drop support for Helm 2.x, but don't yet have a planned timeline (probably some time this year, but it depends on when other Kong releases come out).

@hbagdi you can specify your own certificate, but the chart will always add the checksum annotation regardless. We could add a check to see if the certificate environment variables are present once controller 0.9 is out, though that seems a bit hacky, and adds some extra user work. I recommend we wait until we remove Helm 2.x support and then implement the lookup workaround (which, granted, is also a bit hacky): this is annoying, but not critical to fix. Does that sound reasonable?

[hbagdi]

you can specify your own certificate, but the chart will always add the checksum annotation regardless.

In that case, the cert/key will not change every-release and this will not result in any change in the checksum, meaning the pods will not roll. Is my understanding correct?

helm/charts#5167 (comment) indicates that Helm 3.1's lookup function provides a means to work around this, but using this would break Helm 2.x compatibility.

Yes, we can adopt this later on.

Meta comment:
Regarding the new label "2.0", can we change it to "Chart 2.0"?
There are two upgrades coming in: Helm 3 upgrade, Chart 2.0, and it is important to differentiate them.

[agaffney]

How do you specify your own cert/key for the admission controller? The template for the admission webhook config seems to unconditionally generate a random one.

https://github.com/Kong/charts/blob/master/charts/kong/templates/admission-webhook.yaml#L2-L4

[rainest]

In that case, the cert/key will not change every-release and this will not result in any change in the checksum, meaning the pods will not roll. Is my understanding correct?

Not quite: the checksum is based off the certificate installed at the default location, which is the random certificate. The volume mount uses the random certificate and can't be configured to use your own secret, so it's always rotated.

We can make that configurable, but it will need to take the environment-based certificate handling from 0.9 into account. That has a secondary issue where generating a checksum would also require the 3.x lookup function, though we could disable the annotation if it's present and trust users to restart their own pods when it changes.

[hbagdi]

https://github.com/Kong/charts/blob/master/charts/kong/templates/admission-webhook.yaml#L2-L4

We need to fix this. It should be possible for user to supply certs.

We can make that configurable, but it will need to take the environment-based certificate handling from 0.9 into account.

though we could disable the annotation if it's present and trust users to restart their own pods when it changes.

Not ideal but that's totally acceptable.

[mflendrich]

I understand that:

• if the user provides a custom webhook cert, then the annotation changes causing an unnecessary rolling restart of pods. Filed #252 to handle this.
• if the user provides no webhook cert, auto-generation should happen only once. Filed #253

I understand that these are two independent issues that should be handled separately. Closing this issue to allow more focused execution on those two. Thanks @agaffney for reporting this.