Namespace deletion hanging about helm-controller

[toms-place]

What needs to be done, so we can deploy a Kind HelmChart in other namespaces than kube-system?
Not every User is able to access this namespace..

[mcandio]

@cubinet-code Thank you so much, I really appreciate your answer, this solves a lot of things in our company.

[piotrminkina]

I have the same problem. We implement HelmChart to the same namespace on which we implement Helm Charts. This makes it easier to deploy using the GitOps method with Anthos Config Manager.

Does it say somewhere in the documentation that you have to apply HelmChart to kube-system? I don't remember that.

There is a script for fast patching:

namespace=hanging-namespace
kubectl --namespace="$namespace" get helmchart --output=name \
	| xargs -rI{} kubectl patch --namespace="$namespace" {} --type=merge --patch='{"metadata": {"finalizers": []}}'

[brandond]

There is no requirement that you deploy charts to any particular namespace. I'm not sure why the finalizer would be getting stuck when deploying to different namespaces.

[piotrminkina]

@brandond I checked what the issue might be. I created a namespace hec-local where Helm Controller in version 0.15.4 is installed. There are no problems in the logs:

[helm-controller] time="2023-12-04T09:08:36Z" level=info msg="Applying CRD helmcharts.helm.cattle.io"
[helm-controller] time="2023-12-04T09:08:36Z" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io"
[helm-controller] I1204 09:08:36.463283       1 controllers.go:93] Starting helm controller with 3 threads
[helm-controller] I1204 09:08:36.463336       1 controllers.go:98] Starting helm controller in namespace hec-stubs
[helm-controller] I1204 09:08:36.463412       1 leaderelection.go:248] attempting to acquire leader lease hec-stubs/helm-controller-lock...
[helm-controller] I1204 09:08:36.534016       1 leaderelection.go:258] successfully acquired lease hec-stubs/helm-controller-lock
[helm-controller] time="2023-12-04T09:08:37Z" level=info msg="Starting /v1, Kind=ConfigMap controller"
[helm-controller] time="2023-12-04T09:08:37Z" level=info msg="Starting /v1, Kind=Secret controller"
[helm-controller] time="2023-12-04T09:08:37Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
[helm-controller] time="2023-12-04T09:08:37Z" level=info msg="Starting batch/v1, Kind=Job controller"
[helm-controller] time="2023-12-04T09:08:37Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
[helm-controller] time="2023-12-04T09:08:37Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller"
[helm-controller] I1204 09:08:37.030812       1 controllers.go:105] All controllers have been started
[helm-controller] time="2023-12-04T09:08:37Z" level=info msg="Starting /v1, Kind=ServiceAccount controller"

Then I started namespace hec-stubs, in which I implemented HelmChart podinfo. Here also in the logs clean.

[helm-controller] time="2023-12-04T09:08:37Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"hec-stubs\", Name:\"podinfo\", UID:\"fe95d22a-343b-4068-b0ee-d0eefb71d177\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"10898\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job hec-stubs/helm-install-podinfo"
[helm-controller] time="2023-12-04T09:08:37Z" level=error msg="error syncing 'hec-stubs/podinfo': handler dep-hec-helm-controller-chart-registration: helmcharts.helm.cattle.io \"podinfo\" not found, requeuing"
[helm-controller] time="2023-12-04T09:08:37Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"hec-stubs\", Name:\"podinfo\", UID:\"fe95d22a-343b-4068-b0ee-d0eefb71d177\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"10898\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job hec-stubs/helm-install-podinfo"
[helm-controller] time="2023-12-04T09:08:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"hec-stubs\", Name:\"podinfo\", UID:\"fe95d22a-343b-4068-b0ee-d0eefb71d177\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"10909\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job hec-stubs/helm-install-podinfo"
[helm-controller] time="2023-12-04T09:08:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"hec-stubs\", Name:\"podinfo\", UID:\"fe95d22a-343b-4068-b0ee-d0eefb71d177\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"10909\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job hec-stubs/helm-install-podinfo"
[helm-controller] time="2023-12-04T09:08:42Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"hec-stubs\", Name:\"podinfo\", UID:\"fe95d22a-343b-4068-b0ee-d0eefb71d177\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"10909\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job hec-stubs/helm-install-podinfo"
[helm-controller] time="2023-12-04T09:08:42Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"hec-stubs\", Name:\"podinfo\", UID:\"fe95d22a-343b-4068-b0ee-d0eefb71d177\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"10909\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job hec-stubs/helm-install-podinfo"

I then wanted to delete the entire namespace hec-stubs with the kubectl delete ns hec-stubs command, but you can see there is a problem with that. In the logs quite a few errors appeared:

[helm-controller] time="2023-12-04T09:09:56Z" level=error msg="error syncing 'hec-stubs/podinfo': handler on-helm-chart-remove: failed to create hec-stubs/chart-values-podinfo /v1, Kind=Secret for helm-chart-registration hec-stubs/podinfo: secrets \"chart-values-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/chart-content-podinfo /v1, Kind=ConfigMap for helm-chart-registration hec-stubs/podinfo: configmaps \"chart-content-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/helm-podinfo /v1, Kind=ServiceAccount for helm-chart-registration hec-stubs/podinfo: serviceaccounts \"helm-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/helm-delete-podinfo batch/v1, Kind=Job for helm-chart-registration hec-stubs/podinfo: jobs.batch \"helm-delete-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, requeuing"
[... above 4 times ...]
[helm-controller] E1204 09:09:57.139671       1 leaderelection.go:334] error initially creating leader election record: configmaps "helm-controller-lock" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated
[helm-controller] time="2023-12-04T09:09:57Z" level=error msg="error syncing 'hec-stubs/podinfo': handler on-helm-chart-remove: failed to create hec-stubs/chart-values-podinfo /v1, Kind=Secret for helm-chart-registration hec-stubs/podinfo: secrets \"chart-values-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/chart-content-podinfo /v1, Kind=ConfigMap for helm-chart-registration hec-stubs/podinfo: configmaps \"chart-content-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/helm-podinfo /v1, Kind=ServiceAccount for helm-chart-registration hec-stubs/podinfo: serviceaccounts \"helm-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/helm-delete-podinfo batch/v1, Kind=Job for helm-chart-registration hec-stubs/podinfo: jobs.batch \"helm-delete-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, requeuing"
[... above 4 times ...]
[helm-controller] E1204 09:09:59.166354       1 leaderelection.go:334] error initially creating leader election record: configmaps "helm-controller-lock" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated
[... above 3 times ...]
[helm-controller] time="2023-12-04T09:10:03Z" level=error msg="error syncing 'hec-stubs/podinfo': handler on-helm-chart-remove: failed to create hec-stubs/chart-values-podinfo /v1, Kind=Secret for helm-chart-registration hec-stubs/podinfo: secrets \"chart-values-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/chart-content-podinfo /v1, Kind=ConfigMap for helm-chart-registration hec-stubs/podinfo: configmaps \"chart-content-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/helm-podinfo /v1, Kind=ServiceAccount for helm-chart-registration hec-stubs/podinfo: serviceaccounts \"helm-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/helm-delete-podinfo batch/v1, Kind=Job for helm-chart-registration hec-stubs/podinfo: jobs.batch \"helm-delete-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, requeuing"
[helm-controller] E1204 09:10:05.167275       1 leaderelection.go:334] error initially creating leader election record: configmaps "helm-controller-lock" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated
[... above 5 times ...]
[helm-controller] time="2023-12-04T09:10:14Z" level=error msg="error syncing 'hec-stubs/podinfo': handler on-helm-chart-remove: failed to create hec-stubs/chart-values-podinfo /v1, Kind=Secret for helm-chart-registration hec-stubs/podinfo: secrets \"chart-values-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/chart-content-podinfo /v1, Kind=ConfigMap for helm-chart-registration hec-stubs/podinfo: configmaps \"chart-content-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/helm-podinfo /v1, Kind=ServiceAccount for helm-chart-registration hec-stubs/podinfo: serviceaccounts \"helm-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, failed to create hec-stubs/helm-delete-podinfo batch/v1, Kind=Job for helm-chart-registration hec-stubs/podinfo: jobs.batch \"helm-delete-podinfo\" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated, requeuing"
[helm-controller] E1204 09:10:15.162280       1 leaderelection.go:334] error initially creating leader election record: configmaps "helm-controller-lock" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated
[... above 6 times ...]
[helm-controller] I1204 09:10:27.118320       1 leaderelection.go:283] failed to renew lease hec-stubs/helm-controller-lock: timed out waiting for the condition
[helm-controller] E1204 09:10:27.118405       1 leaderelection.go:306] Failed to release lock: resource name may not be empty
[helm-controller] time="2023-12-04T09:10:27Z" level=fatal msg="leaderelection lost for helm-controller-lock"
INFO[0118] Streaming logs from pod: dep-hec-helm-controller-9998876f6-nxx4c container: helm-controller  subtask=-1 task=DevLoop
[helm-controller] time="2023-12-04T09:10:28Z" level=info msg="Applying CRD helmcharts.helm.cattle.io"
[helm-controller] time="2023-12-04T09:10:28Z" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io"
[helm-controller] I1204 09:10:28.258992       1 controllers.go:93] Starting helm controller with 3 threads
[helm-controller] I1204 09:10:28.259005       1 controllers.go:98] Starting helm controller in namespace hec-stubs
[helm-controller] I1204 09:10:28.259020       1 leaderelection.go:248] attempting to acquire leader lease hec-stubs/helm-controller-lock...
[helm-controller] E1204 09:10:28.262808       1 leaderelection.go:334] error initially creating leader election record: configmaps "helm-controller-lock" is forbidden: unable to create new content in namespace hec-stubs because it is being terminated
[... further just repeated the above error message, and so on endlessly ...]

I applied the patch from my previous post, then the namespace was removed.

For further testing I reinstalled namespace hec-stubs, then tried again to remove namespace, but this time just before removing namespace I ran a command that first removed the HelmChart resource, i.e. kubectl delete helmchart podinfo, which produced the following logs:

[helm-controller] time="2023-12-04T09:18:45Z" level=error msg="error syncing 'hec-stubs/podinfo': handler on-helm-chart-remove: waiting for delete of helm chart for hec-stubs/podinfo by helm-delete-podinfo, requeuing"
[helm-controller] time="2023-12-04T09:18:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"hec-stubs\", Name:\"podinfo\", UID:\"c6735340-a323-4626-baff-ebeede84edc6\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"11690\", FieldPath:\"\"}): type: 'Normal' reason: 'RemoveJob' Uninstalled HelmChart using Job hec-stubs/helm-delete-podinfo, removing resources"
[helm-controller] time="2023-12-04T09:18:52Z" level=error msg="error syncing 'hec-stubs/podinfo': handler on-helm-chart-remove: waiting for delete of helm chart for hec-stubs/podinfo by helm-delete-podinfo, requeuing"
[helm-controller] time="2023-12-04T09:18:55Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"hec-stubs\", Name:\"podinfo\", UID:\"c6735340-a323-4626-baff-ebeede84edc6\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"11723\", FieldPath:\"\"}): type: 'Normal' reason: 'RemoveJob' Uninstalled HelmChart using Job hec-stubs/helm-delete-podinfo, removing resources"

And finally, I removed the namespace with the kubectl delete ns hec-stubs command, which removed the namespace, but produced the following logs in the Helm Controller logs:

[helm-controller] E1204 09:19:50.749821       1 leaderelection.go:334] error initially creating leader election record: namespaces "hec-stubs" not found
[... further just repeated the above error message ...]
[helm-controller] I1204 09:20:20.689513       1 leaderelection.go:283] failed to renew lease hec-stubs/helm-controller-lock: timed out waiting for the condition
[helm-controller] E1204 09:20:20.689995       1 leaderelection.go:306] Failed to release lock: resource name may not be empty
[helm-controller] time="2023-12-04T09:20:20Z" level=fatal msg="leaderelection lost for helm-controller-lock"
INFO[0290] Streaming logs from pod: dep-hec-helm-controller-857679455b-nbzfk container: helm-controller  subtask=-1 task=DevLoop
[helm-controller] time="2023-12-04T09:20:21Z" level=info msg="Applying CRD helmcharts.helm.cattle.io"
[helm-controller] time="2023-12-04T09:20:21Z" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io"
[helm-controller] I1204 09:20:21.658102       1 controllers.go:93] Starting helm controller with 3 threads
[helm-controller] I1204 09:20:21.658143       1 controllers.go:98] Starting helm controller in namespace hec-stubs
[helm-controller] I1204 09:20:21.658199       1 leaderelection.go:248] attempting to acquire leader lease hec-stubs/helm-controller-lock...
[helm-controller] E1204 09:20:21.778432       1 leaderelection.go:334] error initially creating leader election record: namespaces "hec-stubs" not found
[... further just repeated the above error message, and so on endlessly ...]

It seems that before deleting namespace you must first delete HelmChart resources, although despite this order Helm Controller still doesn't quite get it right.

[jan-hudec]

There is no requirement that you deploy charts to any particular namespace. I'm not sure why the finalizer would be getting stuck when deploying to different namespaces.

The problem is not having the HelmChart in another namespace, the problem is deleting HelmChart resources implicitly by deleting the namespace that contains them. In that case the controller tries to uninstall the helm release, but it can't create the job (or something else it needs; I didn't check the exact sequence of events) to do it, because the namespace is in Terminating state and nothing can be created in it.

In the common case not uninstalling with helm won't matter, because the resources will also be deleted with the namespace, but in case helm installed some non-namespaced resources or installed resources explicitly in other namespaces, they will remain, so this is not ideal.

Unfortunately helm does not force annotating the resources it creates in any way, so they can't be deleted without running helm. So simply ignoring the error if the namespace is in Terminating state is the only thing the controller could do to avoid this issue.

[cubinet-code]

The Helmchart resource should be in the kube-system namespace, as otherwise the finalizers hang.
You then specify the namespace you want the deployment to happen in the spec.targetNamespace:

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: rabbitmq
  namespace: kube-system
spec:
  chart: https://charts.bitnami.com/bitnami/rabbitmq-8.18.0.tgz
  targetNamespace: candio-helm-controller-issue
  valuesContent: |-

To release the broken HelmChart resource in the wrong namespace you can manually patch the finalizers on the HelmChart resource. This will release the namespace:

  finalizers:
    - wrangler.cattle.io/helm-controller

to:

  finalizers: []