fluxcd / helm-controller Goto Github PK
View Code? Open in Web Editor NEWThe GitOps Toolkit Helm reconciler, for declarative Helming
Home Page: https://fluxcd.io
License: Apache License 2.0
helm-controller's People
Contributors
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Stargazers
Watchers
Forkers
seaneagan stealthybox idvoretskyi phillebaba smorimoto mewzherder relu devopstoday11 rmccright-ms3 jaganikuman sa-spag juozasget mccodeman rossywhite vijaysamanuri clix-dev-llc marvel-works raffis gtracer alex-berger kingdonb willejs doytsujin makkes somtochiama dmccaffery dan-slinky-ckpd filintod bluepeople1 siegenthalerroger samueltorres johngmyers scottrigby laozc pthomison darkowlzz nuriel77 or-shachar mmartyn adamkorcz vespian tabramczyk thorstenhuhn cebernardi mdnfiras pjbgf aryan9600 janeliul yiannistri mvoitko cbuto tehlers320 liuli01 trekawek morpheusph yuriolisa mikeyrcamp ilanpillemer slimm609 nstogner jont828 steffenmllr alexandermarston isabella232 oeular hiddeco poor12 sbernheim 0x00d0 nuclearwu foxandducksoftware souleb cclank eljulians sachinphogat jtyr mac-chaffee santosh1176 chengleqi josephprem testingiacwithnewaccount jseiser nilesh892003 jsam samgiles nab-gha zaldnoay guobingithub stoetti dippynark mipeho polikeiji rollandf uya116 rannox tanujd11 yraoumbe bholland-bh bignuoli imduffy15helm-controller's Issues
Support Multiple valuesFile in HelmRelease
Original discussion see:
I'm creating this issue on @arbourd's behalf because it seems he didn't create this issue after the discussion.
In short, it would be nice to support passing multiple values files in the HelmRelease. Expected behaviour:
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: helloworld
namespace: default
spec:
chart:
spec:
chart: ./helloworld
sourceRef:
kind: GitRepository
name: MyGitRepo
valuesFile:
- values.yaml
- values.production.yaml
- values.somechange.yaml
interval: 1m0sso that it will have a consistent user experience with helm.
By default, when you do helm install xxx -f values.production.yaml, the values.yaml will always be picked up first, then what's inside values.production.yaml will overwrite the defined values from values.yaml.
In fact, the first element in the valuesFile array in the example above should be omitted too but picked up anyway, just as helm does.
In fact, passing multiple values file is already supported by flux cli, but not supported by using YAML, which isn't user-friendly to GitOps users.
Discussion: Action trigger APIs
The OnCondition fields currently in #2 seem error prone as there are many non-sensical condition states to use as triggers for certain actions e.g.:
- install failure triggering a rollback
- test failure (even after install) triggering a rollback
- install/upgrade/test success triggering an uninstall/rollback.
- reconciliation success/failure triggering anything
To stoke discussion, here is a sketch of an API for specifying triggers for actions which I think matches use cases more closely, with default values shown:
spec:
install:
# remediation strategy for install is always uninstall
maxRetries: 5 # each retry implies remediation
remediateLastRetry: false # set to false to leave last retry in place for debugging purposes
upgrade:
remediationStrategy: rollback # uninstall also supported
maxRetries: 0
remediateLastRetry: false
test:
enable: true
interval: null
# takes into account preceding install / upgrade remediation config
remediateFailures: true
remediateDelayedFailures: false # failures due to interval or e.g. `helm.fluxcd.io/testAt` annotation
rollback: # how, not when
timeout: 300
uninstall: # how, not when
timeout: 300Providing values from ConfigMap and Secret resources
To be able to provide values by other means than in-lining them in the HelmRelease resource. We want to provide users with the option to reference to values in ConfigMap and Secret resources.
Spec changes
diff --git a/docs/spec/v2alpha1/helmreleases.md b/docs/spec/v2alpha1/helmreleases.md
index 5c13e45..086f624 100644
--- a/docs/spec/v2alpha1/helmreleases.md
+++ b/docs/spec/v2alpha1/helmreleases.md
@@ -70,6 +70,10 @@ type HelmReleaseSpec struct {
// +optional
Uninstall Uninstall `json:"uninstall,omitempty"`
+ // ValuesFrom holds references to values in Secrets and ConfigMaps.
+ // +optional
+ ValuesFrom []ValuesReference `json:"valuesFrom,omitempty"`
+
// Values holds the values for this Helm release.
// +optional
Values apiextensionsv1.JSON `json:"values,omitempty"`
@@ -96,6 +100,30 @@ type HelmChartTemplate struct {
Interval *metav1.Duration `json:"interval,omitempty"`
}
+// ValuesReference contains enough information to let you locate the
+// values reference at namespace level, and where the (sub)set of
+// values should be merged at.
+type ValuesReference struct {
+ // Kind of the values referent.
+ // +kubebuilder:validation:Enum=Secret,ConfigMap
+ // +required
+ Kind string `json:"kind"`
+
+ // Name of the values referent.
+ // +required
+ Name string `json:"name"`
+
+ // ValuesKey is the key in the referent the values can be found at.
+ // Defaults to 'values.yaml'.
+ // +optional
+ ValuesKey string `json:"valuesKey,omitempty"`
+
+ // TargetPath is the YAML dot notation path the values should be merged at.
+ // Defaults to 'None', which results in the values getting merged at the root.
+ // +optional
+ TargetPath string `json:"targetPath,omitempty"`
+}
+
// Install holds the configuration for Helm install actions.
type Install struct {
// Timeout is the time to wait for any individual Kubernetes operation (like JobsChanges vs Helm Operator's valuesFrom
-
Values from external sources, for example an URL, are no longer supported.
This choice was made because you can not reliably restore the cluster state from Git repositories if the configuration of a service relies on some URL being available. In addition to this, having the
optionalparam is even more problematic, as the cluster state will change if the URL is briefly unavailable during reconciliation. -
Chart file references are no longer supported, this will instead be part of the
HelmChartAPI. #4 -
It is possible to define a
targetPathwhere the values should be merged at. This works the same as--set <YAML dot notation path>=<value>. With this, the following Helm action values:helm install \ --set-file global.identityTrustAnchorsPEM=ca.crt \ --set-file identity.issuer.tls.crtPEM=issuer.crt \ --set-file identity.issuer.tls.keyPEM=issuer.key \ --set identity.issuer.crtExpiry=$(date -d '+8760 hour' +"%Y-%m-%dT%H:%M:%SZ") \ linkerd/linkerd2Can be created as a secret using
kubectl:kubectl create secret generic cert-values \ --from-file=ca.crt=./certs/ca.crt \ --from-file=issuer.crt=./certs/issuer.crt \ --from-file=issuer.key=./certs/issuer.key \ --from-literal=crtExpiry=$(date -d '+8760 hour' +"%Y-%m-%dT%H:%M:%SZ")And then be declaratively defined in the
HelmReleaseas:spec: valuesFrom: - kind: Secret name: cert-values valuesKey: ca.crt targetPath: global.identityTrustAnchorsPEM - kind: Secret name: cert-values valuesKey: issuer.crt targetPath: identity.issuer.tls.crtPEM - kind: Secret name: cert-values valuesKey: issuer.key targetPath: identity.issuer.tls.keyPEM - kind: Secret name: cert-values valuesKey: crtExpiry targetPath: identity.issuer.crtExpiry
Behavioral notes
- Referenced values should be watched, and changes should trigger a reconciliation of the
HelmRelease. This may require some documentation around the fact that this may need some coordination during changes to multiple referenced resources, either by:- A pattern of creating new
ConfigMapandSecretresources each time using a hash in the name. - Combining values in the same file that are likely to receive updates at the same time.
- A pattern of creating new
Tasks
- Support referring to
ConfigMapandSecretresources - Support merging at
TargetPathlevel - Watch referred resources
Discussions / references
Typo in API docs
See
Reconciliation can be suspended by setting spec.susped to true.
in section https://github.com/fluxcd/helm-controller/blob/main/docs/spec/v2beta1/helmreleases.md#reconciliation
Reconciliation error when release name is auto-generated
If a release doesn't contain spec.releaseName, after install, at first reconciliation, it fails with:
{"level":"error","ts":"2020-07-21T10:49:09.883Z","logger":"controller-runtime.controller","msg":"Reconciler error","controller":"helmrelease","name":"podinfo","namespace":"gitops-system","error":"cannot re-use a name that is still in use","errorVerbose":"cannot re-use a name that is still in use\nhelm.sh/helm/v3/pkg/action.(*Install).availableName\n\t/go/pkg/mod/helm.sh/helm/[email protected]/pkg/action/install.go:428\nhelm.sh/helm/v3/pkg/action.(*Install).Run\n\t/go/pkg/mod/helm.sh/helm/[email protected]/pkg/action/install.go:179\ngithub.com/fluxcd/helm-controller/controllers.install\n\t/workspace/controllers/helmrelease_controller.go:516\ngithub.com/fluxcd/helm-controller/controllers.(*HelmReleaseReconciler).release\n\t/workspace/controllers/helmrelease_controller.go:309\ngithub.com/fluxcd/helm-controller/controllers.(*HelmReleaseReconciler).Reconcile\n\t/workspace/controllers/helmrelease_controller.go:183\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}
Providing Values via chartFileRef
Per the roadmap git chart sources are a non goal.
Migrate users that are using Helm charts from Git
In deprecating this we also will be losing the the chartFileRef option for providing values.
Values from chart files are only supported for charts from a Git repository.
Some users (like myself) have used git chart sources specifically to enable this despite agreeing that chart repository is a better solution. Many public charts like redis are published with this method for differentiating environments. While it may be messy, It would be possible to include this functionality for chart repository sources as helm does download the tar.
How the specify dependencies in HelmRelease?
In Flux1 it was possible to override the dependencies of a chart like this
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
name: kube-prometheus-stack
spec:
chart:
repository: my-internal-repo
name: kube-prometheus-stack
version: 1.2.3
dependencies:
- condition: kubeStateMetrics.enabled
name: kube-state-metrics
repository: my-internal-repo
version: 3.2.1
But there is nothing about dependencies mention in de Flux2 HelmRelease CRD. Should the repository tag be replaced by a sourceRef or is this not possible in Flux2?
Use existing HelmChart resource in a HelmRelease
Each HelmRelease automatically creates a corresponding HelmChart object and it looks like a separate tarball gets downloaded for each HelmChart object. This seems like an inefficient use of storage if multiple HelmReleases use the same chart & version because multiple identical tarballs are being downloaded.
Is there a way to manually create a shared HelmChart object and reference it from multiple HelmReleases?
Chart update fails
When changing the chart version, HC can't update the HelmChart:
helmcharts.source.fluxcd.io "gitops-system-podinfo" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update
scheme "oci" not supported
I wanted to use AWS ECR as a helm repository (see https://docs.aws.amazon.com/AmazonECR/latest/userguide/push-oci-artifact.html).
I pushed my chart there. Then I wanted to specify this as a subchart with the oci:// schema (see https://helm.sh/docs/topics/registries/#specifying-dependencies).
I even tried to add the HELM_EXPERIMENTAL_OCI=1 as env for the helm controller with no help.
All I got was the error message saying scheme "oci" not supported
Sporadically incorrect remediation actions
I have noticed that one of our end-to-end tests is flaky (install-test-fail), and sometimes acts in such a way that the failures counter suddenly resets and the wrong remediation strategy is taken.
From a first observation of the logs this problem does not seem to just exist for the end-to-end tests, but may also occur in real environments. I have a suspicion that it may have to do with the fact that we do not retry failed status updates, and seem to be running into the object has been modified; please apply your changes to the latest version and try again errors quote often, especially for the install-test-fail scenario.
This may be an indication that:
- Multiple processes are touching the same resource
- The (during reconciliation) modified object is not passed on correctly everywhere, and we have a split brain problem (aggravated by the fact that we do not perform retries on failed updates)
Support Helm PostRender Functionality
Describe the feature
It might be a little harder given the toolkit's new structure, but it'd be great to see support for Helm's Post Render functionality.
This allows for customization of out-of-the box charts without having to manually untar and manipulate the base chart.
Alternatives
Probably an advanced feature, but being able to chain Helm and Kustomize controllers would be another way to attack this.
Conditional remediation on Helm actions
The idea is to add <Action>Remediation structs to the API, to be implemented by the Helm actions that should offer a remediation strategy in case they fail (Install and Upgrade).
Spec changes
Install
diff --git a/docs/spec/v2alpha1/helmreleases.md b/docs/spec/v2alpha1/helmreleases.md
index bf6010d..127c196 100644
--- a/docs/spec/v2alpha1/helmreleases.md
+++ b/docs/spec/v2alpha1/helmreleases.md
@@ -96,6 +96,21 @@ type HelmChartTemplate struct {
Interval *metav1.Duration `json:"interval,omitempty"`
}
+type InstallRemediation struct {
+ // Retries is the number of retries that should be attempted on failures before
+ // bailing. Defaults to '0', a negative integer equals to unlimited retries.
+ // +optional
+ Retries int `json:"retries,omitempty"`
+
+ // IgnoreTestFailures tells the controller to skip remediation when
+ // the Helm tests are run after an install action but fail.
+ // Defaults to 'Test.IgnoreTestFailures'.
+ // +optional
+ IgnoreTestFailures bool `json:"ignoreTestFailures,omitempty"`
+
+ // SkipLastRemediation tells the controller to skip remediation when
+ // there are no retries left so the release can be debugged.
+ // +optional
+ SkipLastRemediation bool `json:"skipLastRemediation,omitempty"`
+}diff --git a/docs/spec/v2alpha1/helmreleases.md b/docs/spec/v2alpha1/helmreleases.md
index bf6010d..127c196 100644
// Install holds the configuration for Helm install actions.
type Install struct {
// Timeout is the time to wait for any individual Kubernetes operation (like Jobs
@@ -104,6 +119,12 @@ type Install struct {
// +optional
Timeout *metav1.Duration `json:"timeout,omitempty"`
+ // Remediation holds the remediation configuration for when the
+ // Helm install action fails. The default install failure
+ // remediation is an uninstall action.
+ // +optional
+ Remediation InstallRemediation `json:"remediation,omitempty"`
+
// DisableWait disables the waiting for resources to be ready after a
// Helm install has been performed.
// +optionalUpgrade
diff --git a/docs/spec/v2alpha1/helmreleases.md b/docs/spec/v2alpha1/helmreleases.md
index bf6010d..127c196 100644
--- a/docs/spec/v2alpha1/helmreleases.md
+++ b/docs/spec/v2alpha1/helmreleases.md
@@ -96,6 +96,21 @@ type HelmChartTemplate struct {
Interval *metav1.Duration `json:"interval,omitempty"`
}
+type UpgradeRemediation struct {
+ // Retries is the number of retries that should be attempted on failures before
+ // bailing. Defaults to '0', a negative integer equals to unlimited retries.
+ // +optional
+ Retries int `json:"retries,omitempty"`
+
+ // Strategy to use for the remediation of a failed action.
+ // +kubebuilder:validation:Enum=rollback,uninstall
+ // +optional
+ Strategy string `json:"strategy,omitempty"`
+
+ // IgnoreTestFailures tells the controller to skip remediation when
+ // the Helm tests are run after an upgrade action but fail.
+ // Defaults to 'Test.IgnoreTestFailures'.
+ // +optional
+ IgnoreTestFailures bool `json:"ignoreTestFailures,omitempty"`
+
+ // SkipLastRemediation tells the controller to skip remediation when
+ // there are no retries left so the release can be debugged.
+ // +optional
+ SkipLastRemediation bool `json:"skipLastRemediation,omitempty"`
+}diff --git a/docs/spec/v2alpha1/helmreleases.md b/docs/spec/v2alpha1/helmreleases.md
index bf6010d..127c196 100644
@@ -137,10 +158,11 @@ type Upgrade struct {
// +optional
Timeout *metav1.Duration `json:"timeout,omitempty"`
- // MaxRetries is the number of retries that should be attempted on failures before
- // bailing. Defaults to '0', a negative integer equals to unlimited retries.
+ // Remediation holds the remediation configuration for when the
+ // Helm upgrade action for the HelmRelease fails. The default
+ // upgrade failure remediation is to not perform any action.
// +optional
- MaxRetries int `json:"maxRetries,omitempty"`
+ Remediation UpgradeRemediation `json:"remediation,omitempty"`
// DisableWait disables the waiting for resources to be ready after a
// Helm upgrade has been performed.Test
diff --git a/docs/spec/v2alpha1/helmreleases.md b/docs/spec/v2alpha1/helmreleases.md
index bf6010d..127c196 100644
@@ -179,6 +201,12 @@ type Test struct {
// +optional
Enable bool `json:"enable,omitempty"`
+ // IgnoreTestFailures tells the controller to skip remediation when
+ // the Helm tests are run but fail.
+ // Can be overwritten for tests run after install or upgrade actions
+ // in 'Install.IgnoreTestFailures' and 'Upgrade.IgnoreTestFailures'.
+ // +optional
+ IgnoreTestFailures bool `json:"ignoreTestFailures,omitempty"`
// Timeout is the time to wait for any individual Kubernetes operation
// during the performance of a Helm test action. Defaults to
// 'HelmReleaseSpec.Timeout'.Rollback
diff --git a/docs/spec/v2alpha1/helmreleases.md b/docs/spec/v2alpha1/helmreleases.md
index bf6010d..127c196 100644
@@ -188,11 +216,6 @@ type Test struct {
// Rollback holds the configuration for Helm rollback actions.
type Rollback struct {
- // Enable enables Helm rollback actions for this release after an
- // Helm install or upgrade action failure.
- // +optional
- Enable bool `json:"enable,omitempty"`
-
// Timeout is the time to wait for any individual Kubernetes operation (like Jobs
// for hooks) during the performance of a Helm rollback action. Defaults to
// 'HelmReleaseSpec.Timeout'.Behavioral notes
- Currently test failures are ignored in the POC and result in
Ready==True, this behavior changes due to theIgnoreTestFailuresflags. - With the
Upgrade.Remediation.Strategyvalue ofuninstall, the retry will be an install. This means that after a successful uninstall, the install remediation configuration takes over.
Discussions / references
Unable to deploy a chart with an included library chart
Hello!
I am on version 0.2.4 of flux2 and am having issues deploying a chart. We're working on creating a library chart for some of the charts we host. This chart has a common library chart.
How to replicate
Apply the HelmRepository
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: k8s-at-home-charts
namespace: flux-system
spec:
interval: 10m
url: https://k8s-at-home.com/charts/
timeout: 3mApply the HelmRelease
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: jackett
namespace: media
spec:
interval: 5m
chart:
spec:
chart: jackett
version: 5.2.0
sourceRef:
kind: HelmRepository
name: k8s-at-home
namespace: flux-system
interval: 1m
values:
controllerType: deploymentErrors in logs
helmrelease.helm.toolkit.fluxcd.io/jackett False Helm install failed: template: jackett/templates/common.yaml:1:3: executing "jackett/templates/common.yaml" at <include "common.all" .>: error calling include: template: jackett/charts/common/templates/_all.tpl:8:16: executing "common.all" at <.Values.addons.vpn.enabled>: nil pointer evaluating interface {}.vpn 12m
helmrelease.helm.toolkit.fluxcd.io/jackett False Helm install failed: template: jackett/templates/common.yaml:1:3: executing "jackett/templates/common.yaml" at <include "common.all" .>: error calling include: template: jackett/charts/common/templates/_all.tpl:21:5: executing "common.all" at <include "common.service" .>: error calling include: template: jackett/charts/common/templates/_service.tpl:3:8: executing "common.service" at <include "common.classes.service" .>: error calling include: template: jackett/charts/common/templates/classes/_service.tpl:23:14: executing "common.classes.service" at <eq $svcType "ClusterIP">: error calling eq: incompatible types for comparison
Debugging
I am able to run helm template over this and it generated the manifests correctly
# values.yaml
---
controllerType: deploymenthelm template \
https://github.com/k8s-at-home/charts/releases/download/jackett-5.2.0/jackett-5.2.0.tgz \
--values=./values.yaml \
--output-dir=./ \
--name-template=jackett \
--version=5.2.0 \
--namespace media
I have also noticed this helm chart works fine in Flux v1.
I have also made a post in slack if that's easier to help debugging.
https://cloud-native.slack.com/archives/CLAJ40HV3/p1605148705143800
Thanks again for all your amazing work on this tool!
Impersonate service accounts
To make use of helm-controller in a multi-tenant cluster, the HelmRelease API could have a ServiceAccountName field similar to the Kustomization API. When a HelmRelease has a ServiceAccountName specified, helm-controller should impersonate that service account when installing or upgrading the Helm release.
Helm Controller fails to process helmreleases
I have noticed this scenario repeatedly on a number of different clusters, Helm Controller seems to stop doing periodic reconcilations. Deleting the pod fixes this.
After upgrading to v0.3.0 I noted that some helmreleases were reporting issues
$ kubectl get -A helmreleases.helm.toolkit.fluxcd.io
NAMESPACE NAME READY STATUS AGE
apps microservice-1 True Release reconciliation succeeded 20h
base microservice-1 False HelmChart 'gotk-system/base-microservice-1' is not ready 20h
bootstrap integration-test True Release reconciliation succeeded 20h
bootstrap microservice-1 True Release reconciliation succeeded 20h
bootstrap microservice-2 True Release reconciliation succeeded 20h
cluster-addons pr172922-aks-helmchart-cert-manager True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-cp-poll True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-datadog False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-helmchart-datadog' is not ready 10d
cluster-addons pr172922-aks-helmchart-disable-updates True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-externaldns False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-helmchart-externaldns' is not ready 10d
cluster-addons pr172922-aks-helmchart-nginx-ingress-controller True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-rolessetup False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-helmchart-rolessetup' is not ready 10d
cluster-addons pr172922-aks-helmchart-tenancyoperator False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-helmchart-tenancyoperator' is not ready 10d
cluster-addons pr172922-aks-helmchart-vaultregistration False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-helmchart-vaultregistration' is not ready 10d
cluster-addons pr172922-aks-kubestatemetrics False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-kubestatemetrics' is not ready 10d
mgmt microservice-1 True Release reconciliation succeeded 20h
vault-injector-webhook vault-injector-webhook False HelmChart 'cluster-addons/vault-injector-webhook-vault-injector-webhook' is not ready 10d
Note it has been over three minutes since last reconcile yet HelmRelease intervals are set to 1 or 3 minute.
So I deleted the pod
a669981@vc2crtp2473106n:~$ kubectl -n gotk-system delete pod helm-controller-6b8747c4dc-shkdr
pod "helm-controller-6b8747c4dc-shkdr" deleted
a669981@vc2crtp2473106n:~$ kubectl get -A helmreleases.helm.toolkit.fluxcd.io
NAMESPACE NAME READY STATUS AGE
apps microservice-1 True Release reconciliation succeeded 20h
base microservice-1 False HelmChart 'gotk-system/base-microservice-1' is not ready 20h
bootstrap integration-test True Release reconciliation succeeded 20h
bootstrap microservice-1 True Release reconciliation succeeded 20h
bootstrap microservice-2 True Release reconciliation succeeded 20h
cluster-addons pr172922-aks-helmchart-cert-manager True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-cp-poll True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-datadog False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-helmchart-datadog' is not ready 10d
cluster-addons pr172922-aks-helmchart-disable-updates True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-externaldns False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-helmchart-externaldns' is not ready 10d
cluster-addons pr172922-aks-helmchart-nginx-ingress-controller True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-rolessetup False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-helmchart-rolessetup' is not ready 10d
cluster-addons pr172922-aks-helmchart-tenancyoperator False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-helmchart-tenancyoperator' is not ready 10d
cluster-addons pr172922-aks-helmchart-vaultregistration False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-helmchart-vaultregistration' is not ready 10d
cluster-addons pr172922-aks-kubestatemetrics False HelmChart 'cluster-addons/cluster-addons-pr172922-aks-kubestatemetrics' is not ready 10d
mgmt microservice-1 True Release reconciliation succeeded 20h
vault-injector-webhook vault-injector-webhook False HelmChart 'cluster-addons/vault-injector-webhook-vault-injector-webhook' is not ready 10d
$ kubectl get -A helmreleases.helm.toolkit.fluxcd.io
NAMESPACE NAME READY STATUS AGE
apps microservice-1 True Release reconciliation succeeded 20h
base microservice-1 True Release reconciliation succeeded 20h
bootstrap integration-test True Release reconciliation succeeded 20h
bootstrap microservice-1 True Release reconciliation succeeded 20h
bootstrap microservice-2 True Release reconciliation succeeded 20h
cluster-addons pr172922-aks-helmchart-cert-manager True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-cp-poll True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-datadog True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-disable-updates True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-externaldns True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-nginx-ingress-controller True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-rolessetup True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-tenancyoperator True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-helmchart-vaultregistration True Release reconciliation succeeded 10d
cluster-addons pr172922-aks-kubestatemetrics True Release reconciliation succeeded 10d
mgmt microservice-1 True Release reconciliation succeeded 20h
vault-injector-webhook vault-injector-webhook True Release reconciliation succeeded 10d
logs : https://gist.github.com/paulcarlton-ww/2f22967692e2ab25a5c8f0a47435b058
Freeze installed chart version in HelmRelease
I think this is not yet possible so here is my use case:
I'd like to install a couple of HelmReleases with a semver expr of * or any other version selector.
Meaning I'd like to install the latest published chart.
However I need a flag to freeze the version once selected and installed because I want avoid automated updates of these releases if it gets reconciled.
In other words I want to install the latest version but avoid further automated updates.
Basically a flag would be required on the HelmRelease, something like freezeVersion bool.
Happy to create a pr for this if you can imagine merging this.
Cluster-state manifest drift correction
Suppose HelmRelease object deploys a helm chart that contains a deployment. The pod resource can now be updated by the user. Until there is a chart version update in HelmRelease object, the pod will continue to have the manually updated setting. I'm looking for a way to force reconcile the resources deployed by the HelmRelease after every x minutes. Does helm-controller support this?
There are a few other intervals like one used by HelmRepository and HelmRelease but they don't seem to reconcile a previous succeeded release unless there is a new version in the HelmRelease. requeueDependency interval is only responsible for reconcile if the dependencies for that helmrelease failed. These seem to be unrelated to my scenario.
post-install hooks are not executed after resources were posted
If a Helm-chart defines a post-install hook the controller does not execute the job after the resources were loaded into kubernetes. Instead the controller waits for all resources to be available (e.g. all pods of the deployments are ready) until it executes the post-install jobs.
The chart I am using needs the post-install jobs to run in order for the pods to successfully start. For my case a workaround by setting disableWait to true which tells the controller to execute the jobs.
From my understanding the specification of the chart hook "post-install" states that it shall be executed after the resources were loaded and not after they become ready?
How to deploy toolkit components programatically i.e. using "helm install" ?
I'm following your excellent guide at https://toolkit.fluxcd.io/guides/installation/ and have managed to get up and running with helm-controller and other toolkit components.
This is fantastic and quick on my laptop / dev environment but i couldn't find any documentation on how to roll these out into non-dev clusters such as for prod?
I guess it is indeed a catch 22 because we want to install helm-controller that handles helm release using helm - but that's the whole point of helm-controller so i guess that is why the flux cli is there for. We typically do not use a cli to release out to prod so was looking to see if there's other ways to do this.
I see flux install --export is available which does indeed show me the manifests used?
Watch ConfigMaps/Secrets referenced in HelmReleases
Unless I'm mistaken the helm controller does not trigger reconciliations of helm releases when ConfiMaps and Secrets used as valuesFrom are updated.
If it would reconciliate on such events it would be quite useful.
unable to merge value from key 'sharedSecret' in Secret 'pomerium/pomerium-config-secrets' into target path 'config.sharedSecret': unable to parse key: assignment to entry in nil map
I have this HelmRelease:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: pomerium
namespace: pomerium
selfLink: /apis/helm.toolkit.fluxcd.io/v2beta1/namespaces/pomerium/helmreleases/pomerium
spec:
chart:
spec:
chart: pomerium
sourceRef:
kind: HelmRepository
name: pomerium
namespace: gotk-system
version: 13.0.0
interval: 10m0s
values:
config:
policy:
- allowed_users:
- github/stuartpb
- [email protected]
from: https://alertmanager.horn.horse
to: http://kps-alertmanager.prometheus.svc.cluster.local:9093
- allowed_users:
- github/stuartpb
- [email protected]
from: https://prometheus.horn.horse
to: http://kps-prometheus.prometheus.svc.cluster.local:9090
- allowed_users:
- github/stuartpb
- [email protected]
from: https://grafana.horn.horse
pass_identity_headers: true
to: http://kube-prometheus-stack-grafana.prometheus.svc.cluster.local
- allowed_users:
- github/stuartpb
- [email protected]
from: https://rook-ceph.horn.horse
to: http://rook-ceph-mgr-dashboard.rook-ceph.svc.cluster.local:7000
- allowed_users:
- github/stuartpb
- [email protected]
from: https://kubernetes-dashboard.horn.horse
tls_skip_verify: true
to: https://kubernetes-dashboard.kubernetes-dashboard.svc.cluster.local
rootDomain: horn.horse
extraEnvFrom:
- secretRef:
name: hornhorse-github
forwardAuth:
enabled: true
internal: true
ingress:
annotations:
cert-manager.io/cluster-issuer: stuartpb-letsencrypt-staging
ingress.kubernetes.io/force-ssl-redirect: "true"
kubernetes.io/ingress.class: internal
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
secret:
name: pomerium-hornhorse-cert
nodeSelector:
kubernetes.io/arch: amd64
redis:
enabled: true
image:
repository: library/redis
master:
persistence:
storageClass: throwaway
slave:
persistence:
storageClass: throwaway
valuesFrom:
- kind: Secret
name: pomerium-config-secrets
targetPath: config.sharedSecret
valuesKey: sharedSecret
- kind: Secret
name: pomerium-config-secrets
targetPath: config.cookieSecret
valuesKey: cookieSecret
status:
conditions:
- lastTransitionTime: "2020-10-21T11:33:39Z"
message: 'unable to merge value from key ''sharedSecret'' in Secret ''pomerium/pomerium-config-secrets''
into target path ''config.sharedSecret'': unable to parse key: assignment to
entry in nil map'
reason: InitFailed
status: "False"
type: Ready
failures: 1
helmChart: gotk-system/pomerium-pomerium
lastAttemptedRevision: 13.0.2
lastAttemptedValuesChecksum: 5b8d69560c6da71246c86fa2402a934632dd96e1
lastHandledReconcileAt: "2020-10-21T04:33:31.530090319-07:00"
lastReleaseRevision: 3
observedGeneration: 12And this Secret (:shushing_face:)
apiVersion: v1
data:
cookieSecret: YW83K2tmdTBvR0RPNWFQWjQvWUQ2Y2NrR2dDbi9Sb1gyWkRwTTUyVQ==
sharedSecret: YmJlK1BXNlBWTnd1STNYaUp1RStEUXh3eEgvVVB6ZU5QYi9WSGpTYQ==
kind: Secret
metadata:
annotations:
secret-generator.v1.mittwald.de/autogenerate: sharedSecret,cookieSecret
secret-generator.v1.mittwald.de/autogenerate-generated-at: "2020-10-21T11:21:26Z"
secret-generator.v1.mittwald.de/secure: "yes"
secret-generator.v1.mittwald.de/type: string
name: pomerium-config-secrets
namespace: pomerium
selfLink: /api/v1/namespaces/pomerium/secrets/pomerium-config-secrets
type: Opaque
As you can see, the Helm chart isn't reconciling, as every attempt to do so fails with unable to merge value from key ''sharedSecret'' in Secret ''pomerium/pomerium-config-secrets'' into target path ''config.sharedSecret'': unable to parse key: assignment to entry in nil map.
This Secret was originally created with the data fields missing (they were created by the secret-generator operator), so I thought it might be possible that it's trying to pull its values from the empty version, but I tried restarting the Helm Controller pod and it's still happening, so I've ruled that out.
Changing targetNamespace causes orphaned HelmRelease
Describe the bug
If after deploying a HelmRelease object the spec.targetNamespace is later changed, the flux controller will create a new deployment, etc in the newly specified namespace. However, it fails to delete the old components. If the HelmRelease is later deleted, Flux orphans the original deployment and does not delete it.
To Reproduce
Steps to reproduce the behaviour:
Install a HelmRelease:
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: myapp
namespace: flux-system
spec:
chart:
spec:
chart: myapp
sourceRef:
kind: HelmRepository
name: myrepo
version: 0.1.0
interval: 1m0s
targetNamespace: mynamespace
Commit the config to git and let Flux deploy.
Later, change the spec.targetNamespace value:
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: myapp
namespace: flux-system
spec:
chart:
spec:
chart: myapp
sourceRef:
kind: HelmRepository
name: myrepo
version: 0.1.0
interval: 1m0s
targetNamespace: mynewnamespace
Commit to git or apply ad-hoc to the cluster.
Expected behavior
Flux should create the deployment components in the new namespace, but it should also delete the components from the old namespace.
Additional context
I first hit this with v0.6.1, but I upgraded to v0.9.0 and the issue still exists.
- Kubernetes version: 1.17.15
- Git provider: Github
- Container registry provider: GCR
Below please provide the output of the following commands:
flux --version
flux check
kubectl -n <namespace> get all
kubectl -n <namespace> logs deploy/source-controller
kubectl -n <namespace> logs deploy/kustomize-controller
% flux --version
flux version 0.9.0
% flux check
► checking prerequisites
✔ kubectl 1.19.3 >=1.18.0-0
✔ Kubernetes 1.17.15-gke.800 >=1.16.0-0
► checking controllers
✔ helm-controller: healthy
► ghcr.io/fluxcd/helm-controller:v0.8.0
✔ kustomize-controller: healthy
► ghcr.io/fluxcd/kustomize-controller:v0.9.1
✔ notification-controller: healthy
► ghcr.io/fluxcd/notification-controller:v0.9.0
✔ source-controller: healthy
► ghcr.io/fluxcd/source-controller:v0.9.0
✔ all checks passed
possible feature-request: allow for targetNamespace to be ignored
I believe it would be beneficial to have an arg that allows the helm-controller to ignore the targetNamespace: field.
Consider a scenario where a tenant in a multi-tenant cluster is restricted to create resources in a single namespace. Assuming a central helm-controller is monitoring all namespaces (with appropriate clustertrole), any tenant could create a HelmRelease resource in their namespace, but install arbitrary resources in another namespace, effectively rendering the namespace restriction of their serviceaccount moot because they can leverage the helm-controller's permissions to action on their behalf.
While there are certainly some configurations that can benefit from targetNamespace, being able to add a flag to have the controller ignore this field would be helpful in these situations so tenants can use the controller, but only be able to affect their own namespace.
Besides having a separate helm-controller per workspace, is there another way to solve this?
Emit events
Helm controller should emit events whenever a release status changes and integrate with notification controller.
"Revision" field disambiguation
Suggestions:
LastAttemptedRevision > LastAttemptedChartRevision
LastAppliedRevision >: LastSuccessfulChartRevision
LastReleaseRevision > no change needed
"ChartRevision" to me helps disambiguate from the "ReleaseRevision" in "LastReleaseRevision', which represents a different kind of revision (chart vs release). I think "Successful" is more accurate than "Applied" since even a failed attempt could be considered applied unless/until it is uninstalled or rolled back.
HelmRelease, GitRepository and applied revisions
When using a GitRepository as chart source for a HelmRelease the spec.Chart.spec.Version of the HelmRelease is not honoured.
This should be reflected in the Status of the HelmRelease, i.e., Last Applied Revision and Last Attempted Revision should point to the git revision of the source instead of the Version value of the Chart.yaml which can be outdated.
Release does not recover from failure
If the chart artifact becomes unavailable for a short period of time, the release doesn't recover from this failure as the status ready condition is not updated on no-op upgrades.
Helm release fails with 'unable to parse key: assignment to entry in nil map'
We are trying to deploy linkerd with the following HelmRelease
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: linkerd
namespace: default
spec:
chart:
spec:
chart: linkerd2
sourceRef:
kind: HelmRepository
name: linkerd
interval: 1m0s
releaseName: linkerd
targetNamespace: default
valuesFrom:
- kind: Secret
name: linkerd-ca
valuesKey: ca.crt
targetPath: global.identityTrustAnchorsPEM
- kind: Secret
name: linkerd-issuer
valuesKey: issuer.crt
targetPath: identity.issuer.tls.crtPEM
- kind: Secret
name: linkerd-issuer
valuesKey: issuer.key
targetPath: identity.issuer.tls.keyPEMWe get the following error message when we run gotk get hr -n default
unable to merge value from key 'ca.crt' in Secret 'default/linkerd-ca' into target path 'global.identityTrustAnchorsPEM': unable to parse key: assignment to entry in nil map
We have noticed that the error is happening at this line:
as
result is still nil.is valuesFile in a helm release spec also working with a git source?
I am trying to use valuesFile in a helm release definition using other values file for staging env but this doesn't seem to work with a git source? Where is he expecting the file? Do I have to give a path relative to the root of the git source or relative to the chart path wich I specify? Tried all but nothing worked...
Helm install failed: timed out waiting for the condition
Context
HelmRelease :
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: ingress
namespace: default
spec:
interval: 5m0s
chart:
spec:
chart: ingress-nginx
sourceRef:
kind: HelmRepository
name: ingress-nginx
namespace: flux-system
interval: 5m0s
Helm Repository:
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: ingress-nginx
namespace: flux-system
spec:
interval: 1m0s
url: https://kubernetes.github.io/ingress-nginx
Issue
Kubectl says that the pod is deployed and working fine. But flux says otherwise...
Also tried increasing the interval, install retries. But nothing worked.
Also manual install with helm works just fine.
Helm test doesn't use Zap logger
Helm test logs write to stdout directly, we should use Zap as the test runner logger.
Pod frontend-podinfo-grpc-test-d0acu pending
Pod frontend-podinfo-grpc-test-d0acu running
{"level":"info","ts":1594408916.3492382,"logger":"controllers.HelmRelease","msg":"Add/Modify event for frontend-podinfo-grpc-test-d0acu: MODIFIED","helmrelease":"helm-system/frontend"}
{"level":"info","ts":1594408917.3560367,"logger":"controllers.HelmRelease","msg":"Add/Modify event for frontend-podinfo-grpc-test-d0acu: MODIFIED","helmrelease":"helm-system/frontend"}
u
Can i use HelmRelease CRD to reconcile Helm Controller itself ?
I am not sure whether this is possible or not currently. But i am just curious about this possibility :
Can i use HelmRelease, HelmChart & HelmRepository CRDs for installing and reconciling Helm controller itself.
I just checked the generation manifests in the fleet repo and found that Kustomization CRD yaml is being used to install & reconcile all gotk components.
Can i change it to HelmRelease for Helm controller (assume i have a helm chart) and get rid of bootstrap upgrade process ?
Unable to send event on NotificationController
Since the last update, I have sometimes (mostly in error case) this log in the HelmController
{"level":"error","ts":"2021-04-16T16:07:56.881Z","logger":"controller.helmrelease","msg":"unable to send event","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"app","namespace":"default","error":"POST http://notification-controller/ giving up after 5 attempt(s)"}
The NotificationController is up and running, I see successful log on it from ImageAutomation.
{"level":"info","ts":"2021-04-16T16:46:45.501Z","logger":"event-server","msg":"Discarding event, no alerts found for the involved object","reconciler kind":"HelmChart","name":"app","namespace":"default"}
I'm running the image ghcr.io/fluxcd/helm-controller:v0.9.0 on an arm64 architecture. Is this something known on your side?
I don't know if you need extra logs or config, feel free to ask me.
Coordination of HelmChart and HelmRelease updates
With HelmRelease.chart content being extracted into source-controller HelmChart CRD, if there are changes to e.g. HelmRelease.values and e.g. HelmChart.version, these two updates are no longer atomic (separate resources), so seems like whichever update happens first may not be successful if it depends on the other. Seems like to rectify this one would need to treat HelmCharts as immutable and rather than update them, instead create and reference new ones (and ideally delete old ones) when using a new chart version. Is there a way to automate this workflow? Similar problem exists with e.g. ConfigMaps and Secrets referenced by PodTemplates, what solutions exist there?
ambiguous error message: install retries exhausted, the object has been modified
I have tried to install cert-manager using flux v2 the installation fails but there is no clear error message what went wrong:
ubectl logs helm-controller-7fc55767cc-zgzv4 -n flux-system | grep cert-manager
{"level":"info","ts":"2020-11-13T21:37:07.958Z","logger":"controllers.HelmRelease","msg":"HelmChart 'cert-manager/cert-manager-cert-manager' is not ready","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"info","ts":"2020-11-13T21:37:07.968Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 37.242494ms","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"info","ts":"2020-11-13T21:37:07.969Z","logger":"controllers.HelmRelease","msg":"HelmChart 'cert-manager/cert-manager-cert-manager' is not ready","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:37:07.974Z","logger":"controllers.HelmRelease","msg":"unable to update status after reconciliation","controller":"helmrelease","request":"cert-manager/cert-manager","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"cert-manager\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2020-11-13T21:37:07.974Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"cert-manager\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"info","ts":"2020-11-13T21:37:07.980Z","logger":"controllers.HelmRelease","msg":"HelmChart 'cert-manager/cert-manager-cert-manager' is not ready","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"info","ts":"2020-11-13T21:37:07.987Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 7.947928ms","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"info","ts":"2020-11-13T21:37:09.229Z","logger":"controllers.HelmRelease","msg":"requesting reconciliation due to GitRepository revision change","helmrelease":"cert-manager/cert-manager","revision":"v1.0.4"}
{"level":"info","ts":"2020-11-13T21:42:11.559Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 5m2.329587331s, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:11.559Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"Helm install failed: timed out waiting for the condition"}
{"level":"info","ts":"2020-11-13T21:42:11.653Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 86.44247ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:11.653Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:11.736Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 72.946477ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:11.736Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:11.836Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 79.143911ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:11.836Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:12.631Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 71.894153ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:12.631Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:12.804Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 93.343503ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:12.804Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:13.050Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 83.851501ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:13.050Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:13.458Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 87.45015ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:13.458Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:14.184Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 85.678056ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:14.184Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:15.608Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 143.716893ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:15.608Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:18.288Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 119.868292ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:18.288Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:23.564Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 155.566888ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:23.564Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:33.923Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 117.852392ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:33.923Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:42:54.533Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 130.255424ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:42:54.533Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:43:35.591Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 96.492659ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:43:35.591Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:44:57.629Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 117.566774ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:44:57.629Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"install retries exhausted"}
{"level":"info","ts":"2020-11-13T21:52:12.013Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 5m3.49529022s, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:52:12.013Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"Helm upgrade failed: timed out waiting for the condition"}
{"level":"error","ts":"2020-11-13T21:52:12.218Z","logger":"controllers.HelmRelease","msg":"unable to update status after state update","controller":"helmrelease","request":"cert-manager/cert-manager","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"cert-manager\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2020-11-13T21:52:12.222Z","logger":"controllers.HelmRelease","msg":"unable to update status after reconciliation","controller":"helmrelease","request":"cert-manager/cert-manager","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"cert-manager\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2020-11-13T21:52:12.222Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"cert-manager\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"info","ts":"2020-11-13T21:57:39.921Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 227.09063ms, next run in 30m0s","controller":"helmrelease","request":"cert-manager/cert-manager"}
{"level":"error","ts":"2020-11-13T21:57:39.921Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"cert-manager","namespace":"cert-manager","error":"upgrade retries exhausted"}
I also do not understand the message" the object has been modified; please apply your changes to the latest version and try again" because I have not changed anything after my commit.
The cert-manager helm chart seems to be installed but does not work:
~/.../clusters/virtnuc1 >>> kubectl get all -n cert-manager
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cert-manager ClusterIP 10.43.190.102 <none> 9402/TCP 35m
service/cert-manager-webhook ClusterIP 10.43.5.80 <none> 443/TCP 35m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cert-manager 0/1 0 0 35m
deployment.apps/cert-manager-cainjector 0/1 0 0 35m
deployment.apps/cert-manager-webhook 0/1 0 0 35m
NAME DESIRED CURRENT READY AGE
replicaset.apps/cert-manager-cainjector-6d5b55cf4b 1 0 0 35m
replicaset.apps/cert-manager-webhook-56f89f4db5 1 0 0 35m
replicaset.apps/cert-manager-78844b6566 1 0 0 35m
~/.../clusters/virtnuc1 >>> helm ls -n cert-manager STATUS CHART APP VERSION
cert-manager cert-manager 2 2020-11-13 21:47:08.906544449 +0000 UTC failed cert-manager-v1.0.4 v1.0.4
Release stuck on HelmChart '...' is not ready
I'm not entirely sure what is happening, but I'd guess a race condition between the chart being ready and the release being notified and reconciliating...?
Let me know what any other info may be helpful
kubectl describe helmreleases jam01 -n postgresql
Name: jam01
Namespace: postgresql
Labels: <none>
Annotations: API Version: helm.toolkit.fluxcd.io/v2beta1
Kind: HelmRelease
Metadata:
Creation Timestamp: 2020-11-20T18:15:31Z
Finalizers:
finalizers.fluxcd.io
Generation: 2
Resource Version: 22905
Self Link: /apis/helm.toolkit.fluxcd.io/v2beta1/namespaces/postgresql/helmreleases/jam01
UID: ecb9d7db-780c-4aca-88f9-cd906067ee6f
Spec:
Chart:
Spec:
Chart: postgresql
Source Ref:
Kind: HelmRepository
Name: bitnami
Namespace: flux-system
Version: 9.8.11
Interval: 30m0s
Release Name: jam01
Values:
Existing Secret: config-secret
Postgresql Database: postgres
Status:
Conditions:
Last Transition Time: 2020-11-20T18:15:33Z
Message: Reconciliation in progress
Reason: Progressing
Status: Unknown
Type: Ready
Helm Chart: flux-system/postgresql-jam01
Last Attempted Revision: 9.8.11
Last Attempted Values Checksum: 26b52db786511f9fefc219d38376f0b8cc69e562
Observed Generation: 2
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal info 2m3s (x2 over 2m3s) helm-controller HelmChart 'flux-system/postgresql-jam01' is not ready
kubectl describe helmchart postgresql-jam01 -n flux-system
Name: postgresql-jam01
Namespace: flux-system
Labels: <none>
Annotations: <none>
API Version: source.toolkit.fluxcd.io/v1beta1
Kind: HelmChart
Metadata:
Creation Timestamp: 2020-11-20T18:15:31Z
Finalizers:
finalizers.fluxcd.io
Generation: 1
Resource Version: 23652
Self Link: /apis/source.toolkit.fluxcd.io/v1beta1/namespaces/flux-system/helmcharts/postgresql-jam01
UID: 461a8165-5036-4b88-bcde-ae5b1f556b59
Spec:
Chart: postgresql
Interval: 30m0s
Source Ref:
Kind: HelmRepository
Name: bitnami
Version: 9.8.11
Status:
Artifact:
Checksum: 568832cdc691436f1f8ca9bd46bba1bc0eab8e17
Last Update Time: 2020-11-20T18:17:56Z
Path: helmchart/flux-system/postgresql-jam01/postgresql-9.8.11.tgz
Revision: 9.8.11
URL: http://source-controller.flux-system/helmchart/flux-system/postgresql-jam01/postgresql-9.8.11.tgz
Conditions:
Last Transition Time: 2020-11-20T18:15:33Z
Message: Fetched revision: 9.8.11
Reason: ChartPullSucceeded
Status: True
Type: Ready
Observed Generation: 1
URL: http://source-controller.flux-system/helmchart/flux-system/postgresql-jam01/postgresql-latest.tgz
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal info 5m35s source-controller Fetched revision: 9.8.11
Allow configurable use of --create-namespace
I am re-opening here the issue fluxcd/helm-operator#488 since helm-operator is now in maintenance mode and this issue was getting some attention.
I do believe having the option to explicitly control the --create-namespace flag via the HelmRelease could be extremely beneficial for certain use-cases, such as the one mentioned by @voron:
Helm2 has "create namespace" by default, helm3 re-invented this feature starting from 3.2.
I'm here because I want to create all HelmRelease in single namespace and helm-operator with clusterRole will create all the required namespaces based on targetNamespace then.
With this new setup and the new HelmRelease CRD, implementing this should be as easy as adding a field to the Install configuration which in turn would control whether this flag is set or not when calling helm install commands (defaulting to false).
helm-controller Install fails, but works fine with standard Helm install
When attempting to make a release of this Kong Chart the helm-controller errors out when running a _helpers template. However, this Chart installs just fine via standard Helm commands.
The error:
helm-controller Helm install failed: template: kong/templates/migrations.yaml:41:10: executing "kong/templates/migrations.yaml" at <include "kong.wait-for-postgres" .>: error calling include: template: kong/templates/_helpers.tpl: │ │ 630:6: executing "kong.wait-for-postgres" at <include "kong.no_daemon_env" .>: error calling include: template: kong/templates/_helpers.tpl:599:15: executing "kong.renderEnv" at <.>: wrong type for value; expected map[string]interface {}; got string
The Source/HelmRelease used:
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: konghq
namespace: flux-system
spec:
interval: 1m
url: https://charts.konghq.com
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: kong
namespace: kong
spec:
targetNamespace: kong
interval: 1m
chart:
spec:
#https://github.com/Kong/charts/tree/main/charts/kong
chart: kong
version: 1.11.0
sourceRef:
kind: HelmRepository
name: konghq
namespace: flux-system
install:
remediation:
retries: 5
upgrade:
remediation:
retries: 5
Helm upgrade failed: another operation (install/upgrade/rollback) is in progress
Sometimes helm releases are not installed because of this error:
{"level":"info","ts":"2020-11-19T15:41:11.273Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 50.12655ms, next run in 9m0s","controller":"helmrelease","request":"traefik/traefik"}
{"level":"error","ts":"2020-11-19T15:41:11.274Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"traefik","namespace":"traefik","error":"Helm upgrade failed: another operation (install/upgrade/rollback) is in progress"}
{"level":"info","ts":"2020-11-19T15:43:19.310Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 69.439664ms, next run in 9m0s","controller":"helmrelease","request":"traefik/traefik"}
{"level":"error","ts":"2020-11-19T15:43:19.310Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"traefik","namespace":"traefik","error":"Helm upgrade failed: another operation (install/upgrade/rollback) is in progress"}
{"level":"info","ts":"2020-11-19T15:52:42.524Z","logger":"controllers.HelmRelease","msg":"reconcilation finished in 69.944579ms, next run in 9m0s","controller":"helmrelease","request":"traefik/traefik"}
{"level":"error","ts":"2020-11-19T15:52:42.525Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"helm.toolkit.fluxcd.io","reconcilerKind":"HelmRelease","controller":"helmrelease","name":"traefik","namespace":"traefik","error":"Helm upgrade failed: another operation (install/upgrade/rollback) is in progress"}
In this case the helm release is stuck in pending status.
We have not found any corresponding log entry of the actual installation. Is this some concurrency bug?
Capture and expose debug (log) information on wait timeout
If a helm release has wait enabled, the default setting, there is a chance of failure due to something obscure and hard to detect which is only logged in debug mode. Like when a service: LoadBalancer is used on a home lab cluster that doesn't enable LB, opting instead for HostNetworking ingress. Helm will wait for the LoadBalancer service to become ready, which it will never do, and the error message will be context deadline exceeded or some fairly unhelpful message that is not by itself enough to trace back to the source of the problem. You have to enable debug mode.
This is not strictly a Helm Controller bug, since HC faithfully reproduces the behavior of Helm upstream in this regard, in debug operating mode, before a timeout is reached, Helm fails reconciling with this error (and so does helm-controller):
client.go:464: [debug] Looks like there are no changes for Ingress "controller-api-server-ingress-http"
wait.go:53: [debug] beginning wait for 59 resources with timeout of 30s
wait.go:225: [debug] Service does not have load balancer ingress IP address: deis/deis-builder
wait.go:225: [debug] Service does not have load balancer ingress IP address: deis/deis-builder
...
# on like this until "context deadline exceeded"
The helpful information is hidden behind the --debug gateway, which is possible to access in helm-controller now by setting --log-level=debug on the entire helm controller.
So, filed under DX/UX note, I thought wouldn't it be great if with my HelmRelease that fails to reconcile, there was a debug flag in the API that I can use on a per-HelmRelease basis, to enable emitting debug entries into (at least the helm-controller logs?)
In thinking this through, it occurs to me now too that perhaps the most useful place to have the debug logs would be in the kubernetes event stream for the HelmRelease, and not just in the helm-controller logs... I have not tried to enable the log-level debug yet, not sure which of these it does right now. But either way, I can see wanting to enable --debug logs from Helm on just a particular HelmRelease vs doing so on every helm release that hc reconciles across the entire cluster at once.
Related conversation from CNCF slack: https://cloud-native.slack.com/archives/CLAJ40HV3/p1613417136364700?thread_ts=1613335794.255500&cid=CLAJ40HV3
Release stuck on HelmChart is not ready
When updating the spec.interval the release gets stuck.
Create release:
gotk create source helm bitnami \
--interval=10m \
--url=https://charts.bitnami.com/bitnami
kubectl create ns contour
gotk create helmrelease contour \
--interval=1h \
--target-namespace=contour \
--release-name=contour \
--source=HelmRepository/bitnami \
--chart=contour \
--chart-version=">2.0.0 <3.0.0"Update release:
$ gotk create helmrelease contour \
> --interval=1m \
> --target-namespace=contour \
> --release-name=contour \
> --source=HelmRepository/bitnami \
> --chart=contour \
> --chart-version=">2.0.0 <3.0.0"
✚ generating release
► applying release
✔ release updated
◎ waiting for reconciliation
✗ HelmChart is not readyRelease is stuck even if the chart is ready:
$ gotk get helmreleases
NAME REVISION SUSPENDED READY MESSAGE
contour 2.3.2 False False HelmChart is not ready
$ kubectl -n gotk-system get helmcharts.source.toolkit.fluxcd.io
NAME CHART VERSION SOURCE KIND SOURCE NAME READY STATUS AGE
gotk-system-contour contour >2.0.0 <3.0.0 HelmRepository bitnami True Fetched revision: 2.3.2 88mIt recovers after a manual reconciliation:
$ gotk reconcile helmrelease contour
► annotating HelmRelease contour in gotk-system namespace
✔ HelmRelease annotated
◎ waiting for HelmRelease reconciliation
✔ HelmRelease reconciliation completed
✔ reconciled revision 2.3.2Ready status when single pod can't start
I have found some weird behavior when testing the status behavior of HelmRelease.
The following setup should deploy the Helm charts podinfo and redis, both of which should fail as the tag foo does not exist for any of the images.
apiVersion: source.toolkit.fluxcd.io/v1alpha1
kind: HelmRepository
metadata:
name: podinfo
namespace: gitops-system
spec:
url: https://stefanprodan.github.io/podinfo
interval: 10m
---
apiVersion: helm.toolkit.fluxcd.io/v2alpha1
kind: HelmRelease
metadata:
name: frontend
namespace: gitops-system
spec:
targetNamespace: webapp
interval: 5m
chart:
spec:
chart: podinfo
version: '>=4.0.0 <5.0.0'
sourceRef:
kind: HelmRepository
name: podinfo
interval: 1m
values:
image:
tag: foo
---
apiVersion: source.toolkit.fluxcd.io/v1alpha1
kind: HelmRepository
metadata:
name: stable
namespace: gitops-system
spec:
url: https://kubernetes-charts.storage.googleapis.com/
interval: 10m
---
apiVersion: helm.toolkit.fluxcd.io/v2alpha1
kind: HelmRelease
metadata:
name: redis
namespace: gitops-system
spec:
targetNamespace: webapp
interval: 5m
chart:
spec:
chart: redis
sourceRef:
kind: HelmRepository
name: stable
interval: 1m
values:
image:
tag: fooBoth result in pods in a ImagePullBackOff state.
NAME READY STATUS RESTARTS AGE
webapp-frontend-podinfo-6694fbcbc4-rvjcn 0/1 ImagePullBackOff 0 6m32s
webapp-redis-master-0 0/1 ImagePullBackOff 0 4m59s
webapp-redis-slave-0 0/1 ImagePullBackOff 0 4m59sYet the podinfo HelmRelease ends up in a ready state which redis does not.
NAME READY STATUS AGE
frontend True release reconciliation succeeded 7m15s
redis False Helm install failed: timed out waiting for the condition 5m45sI would expect both HelmReleases to not be in a ready state.
HelmRelease targetNamespace not honored by helm release secret
I have noticed a strange behavior when using the targetNamespace feature of the HelmRelease.
My understanding was that installing a HelmRelease resource in targetNs or installing it in hrNs with targetNamespace targetNs should result in the same behavior from helm's point of view, with the only change being in where the helm-controller stores metadata to manage the reconciliation.
However I am seeing that when installing a HelmRelease in hrNs with targetNamespace targetNs, helm is somehow set up to store the release's secrets in hrNs rather than targetNs even though the release indeed targets targetNs.
This results in the weird behavior of that helm release not showing up when running helm list in targetNs but showing up with a different namespace when running helm list in hrNs as follows (output is simplified for better understanding):
$ helm list -n targetNs
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
$ helm list -n hrNs
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
helmRelease hrNs 6 2020-12-28 16:47:12.134388 -0600 CST deployed helmRelease-0.0.12
helmRelease2 hrNs 4 2020-12-22 15:31:46.448803 -0600 CST deployed helmRelease2-0.1.0
fluxRelease targetNs 14 2020-12-28 21:56:58.786390024 +0000 UTC deployed fluxRelease-0.7.4
In the previous example, the two helmRelease helm releases are manually installed via helm while fluxRelease has been installed by the helm-controller.
This behavior also brings as consequence the fact that if fluxRelease had already been manually installed via helm and/or previously managed by helm-operator, when helm-controller starts the reconciliation process for the first time it believes that it is not yet installed (since the secret would live in targetNs and therefore it tries to do an install which might leave everything in a weird state considering the release already exists.
This behavior, as far as I have found, is also hard to reproduce using helm directly, as I don't see any option to install a release storing its secret in a different namespace, which might beg the question whether it will actually even remain supported by helm.
I think the reason for this is the initialization of the helm client here using the namespace of the HelmRelease rather than the targetNamespace.
Is this behavior intended? If so, is it officially supported by helm and do you have any advice on how to migrate a helm release already installed without having to purge it and re-install it?
Support cross namespace references in DependsOn field
It would be good to declare cross helm release dependencies that may be deployed into separate namespaces. I attempted to prefix with namespace/chart-name however that did not work. This seems like a fairly popular use case and would be good to setup.
// DependsOn may contain a list of HelmReleases that must be ready before this
// HelmRelease can be reconciled.
// +optional
DependsOn []string `json:"dependsOn,omitempty"`
I also tried bundling all the releases in a single namespace and using targetNamespace however the valuesFrom using either config maps or secrets then no longer reference correctly.
Helm install failed: rendered manifests contain a resource that already exists.
I tried to migrate from helm-operator to helm-controller, but failed.
How to reproduce it
software versions
$ kind version
kind v0.10.0 go1.15.7 darwin/amd64
$ helm version
version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.16"}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-12T01:09:16Z", GoVersion:"go1.15.4", Compiler:"gc", Platform:"darwin/amd64"}
$ flux -v
flux version 0.9.1
create kind cluster
kind.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: kindest/node:v1.19.7@sha256:a70639454e97a4b733f9d9b67e12c01f6b0297449d5b9cbbef87473458e26dca
- role: worker
image: kindest/node:v1.19.7@sha256:a70639454e97a4b733f9d9b67e12c01f6b0297449d5b9cbbef87473458e26dca
$ kind create cluster --name kind --config kind.yaml
deploy helm-operator
$ kubectl apply -f https://raw.githubusercontent.com/fluxcd/helm-operator/1.2.0/deploy/crds.yaml
$ kubectl create ns flux
$ helm upgrade -i helm-operator fluxcd/helm-operator \
--namespace flux \
--set helm.versions=v3
deploy strimzi (or other software)
strimzi.yaml
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
name: strimzi
namespace: strimzi
spec:
chart:
git: https://github.com/strimzi/strimzi-kafka-operator
path: helm-charts/helm3/strimzi-kafka-operator
ref: 0.21.1
$ kubectl create ns strimzi
$ kubectl apply -f strimzi.yaml
bootstrap flux V2
$ export GITHUB_TOKEN=<my-token>
$ export GITHUB_USER=<my-username>
$ flux bootstrap github \
--owner=$GITHUB_USER \
--repository=<my-repository> \
--branch=main \
--path=./ \
--personal
try to migrate to helm-controller
strimzi-helm-controller.yaml
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: strimzi
namespace: strimzi
spec:
interval: 10m
url: https://github.com/strimzi/strimzi-kafka-operator
ref:
tag: 0.21.1
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: strimzi
namespace: strimzi
spec:
interval: 5m
releaseName: strimzi
chart:
spec:
chart: helm-charts/helm3/strimzi-kafka-operator
sourceRef:
kind: GitRepository
name: strimzi
interval: 10m
$ kubectl scale deployment helm-operator --replicas=0 -n flux
$ kubectl apply -f strimzi-helm-controller.yaml
helm-controller log
{"level":"info","ts":"2021-03-12T08:44:37.942Z","logger":"controller.helmrelease","msg":"HelmChart 'strimzi/strimzi-strimzi' is not ready","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"info","ts":"2021-03-12T08:44:37.957Z","logger":"controller.helmrelease","msg":"reconcilation finished in 107.2335ms, next run in 10m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"info","ts":"2021-03-12T08:44:37.961Z","logger":"controller.helmrelease","msg":"HelmChart 'strimzi/strimzi-strimzi' is not ready","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"info","ts":"2021-03-12T08:44:37.977Z","logger":"controller.helmrelease","msg":"reconcilation finished in 18.0793ms, next run in 10m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
W0312 08:44:42.865352 8 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0312 08:44:43.006744 8 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0312 08:44:43.157140 8 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0312 08:44:43.213875 8 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0312 08:44:43.305576 8 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0312 08:44:43.450567 8 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0312 08:44:43.555201 8 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0312 08:44:43.613961 8 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0312 08:44:43.778626 8 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0312 08:44:43.858419 8 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
{"level":"info","ts":"2021-03-12T08:44:44.246Z","logger":"controller.helmrelease","msg":"reconcilation finished in 1.8566583s, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:44.246Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"Helm install failed: rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount \"strimzi-cluster-operator\" in namespace \"strimzi\" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key \"app.kubernetes.io/managed-by\": must be set to \"Helm\"; annotation validation error: missing key \"meta.helm.sh/release-name\": must be set to \"strimzi\"; annotation validation error: missing key \"meta.helm.sh/release-namespace\": must be set to \"strimzi\""}
{"level":"info","ts":"2021-03-12T08:44:44.290Z","logger":"controller.helmrelease","msg":"reconcilation finished in 38.2572ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:44.290Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
{"level":"info","ts":"2021-03-12T08:44:44.350Z","logger":"controller.helmrelease","msg":"reconcilation finished in 46.9777ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:44.350Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
{"level":"info","ts":"2021-03-12T08:44:44.436Z","logger":"controller.helmrelease","msg":"reconcilation finished in 65.3234ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:44.436Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
{"level":"info","ts":"2021-03-12T08:44:44.503Z","logger":"controller.helmrelease","msg":"reconcilation finished in 24.848ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:44.503Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
{"level":"info","ts":"2021-03-12T08:44:44.632Z","logger":"controller.helmrelease","msg":"reconcilation finished in 48.3117ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:44.632Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
{"level":"info","ts":"2021-03-12T08:44:44.861Z","logger":"controller.helmrelease","msg":"reconcilation finished in 68.2977ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:44.862Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
{"level":"info","ts":"2021-03-12T08:44:45.221Z","logger":"controller.helmrelease","msg":"reconcilation finished in 33.556ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:45.221Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
{"level":"info","ts":"2021-03-12T08:44:45.886Z","logger":"controller.helmrelease","msg":"reconcilation finished in 24.5398ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:45.886Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
{"level":"info","ts":"2021-03-12T08:44:47.210Z","logger":"controller.helmrelease","msg":"reconcilation finished in 34.0933ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:47.210Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
{"level":"info","ts":"2021-03-12T08:44:49.800Z","logger":"controller.helmrelease","msg":"reconcilation finished in 29.1911ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:49.800Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
{"level":"info","ts":"2021-03-12T08:44:54.948Z","logger":"controller.helmrelease","msg":"reconcilation finished in 27.4958ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:44:54.949Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
{"level":"info","ts":"2021-03-12T08:45:05.215Z","logger":"controller.helmrelease","msg":"reconcilation finished in 59.6533ms, next run in 5m0s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi"}
{"level":"error","ts":"2021-03-12T08:45:05.215Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"strimzi","namespace":"strimzi","error":"install retries exhausted"}
Maybe this is due to difference in helm versions of helm-operator and helm-controller.
helm/helm#7649 (comment)
Broadcast events for Helm actions
The controller should emit events for each Helm action it took, e.g. install/uninstall/rollback/test.
Support GitRepositories in valuesFrom
I'm working to build a multi-tenant setup in which each tenant will be using the same Helm chart to deploy their apps, but with different values. As a platform team, we're trying to minimize the amount of stuff each of them need to know, especially when it comes to Kubernetes manifests. So, we're looking at the ability to tell a team "Here's a YAML document, update here to deploy your new version." Each of the teams have their own repos with this values file.
In case it helps provide additional context, we're working to standardize our platform efforts around the Compose spec. We've built a Helm chart that consumes a Compose file as its values.yaml and creates the appropriate manifests. We also support various extension fields for additional support around ingress, cert management, secrets injection, etc. This gives us the flexibility to change our mind down the road if we want to leave Kubernetes and use something else... just re-implement the deploy tool. We'd also love to support the ability for anyone to simply run docker-compose -f my-compose-file.yaml up and spin up a very similar stack locally (obviously some of the extensions won't work).
Feature Request
It would be great to be able to define a HelmRelease in which the values come from a specific file(s) located in a GitRepository. I'd like the ability to specify a specific GitRepository and then the specific file within it.
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: backend
namespace: default
spec:
chart:
# omitted
valuesFrom:
- kind: GitRepository
name: tenant-repo
path: ./values.yamlCurrent Workarounds
I've thought of two different approaches to work around this, each with their drawbacks.
- Each team would define their own
HelmReleaseobject with the values embedded in it. This has the downside of now each team is in charge of that object and is exposed to the fact we're using Kubernetes. We also then don't have a file that is Compose spec-compatible. - We could use a Kustomize controller to take the values file and convert it into a
ConfigMap, which then is referenced in thevaluesFrom. It just feels like overkill and would require each repo to have akustomization.yaml. Not end of the world, but feels hacky if we can't reference the file directly.
Between the two workarounds, I'd prefer the later, but it would be great to be able to reference the values file directly. Feedback welcome!
HelmRelease pods being created in `gotk-system` instead of targetNamespace in 0.1.2
I've just been trying to upgrade to gotk version 0.1.2. My configuration was working in alpha 1.
I'm trying to deploy nginx-ingress Helm chart into targetNamespace: foo, however, controller and default-backend the pods are being created in the gotk-system ns.
I've created a sample repo for repro.
Below is what I see in terminal:
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.