rancher / helm-locker

helm-locker

Helm Locker is a Kubernetes operator that prevents resource drift on (i.e. "locks") Kubernetes objects that are tracked by Helm 3 releases.

Once installed, a user can create a HelmRelease CR in the Helm Release Registration Namespace (default: cattle-helm-system) by providing:

1. The name of a Helm 3 release
2. The namespace that contains the Helm Release Secret (supplied as --namespace on the helm install command that created the release)

Once created, the Helm Locker controllers will watch all resources tracked by the Helm Release Secret and automatically revert any changes to the persisted resources that were not made through Helm (e.g. changes that were directly applied via kubectl or other controllers).

Getting Started

For more information, see the Getting Started guide.

Who needs Helm Locker?

Anyone who would like to declaratively manage resources deployed by existing Helm chart releases.

How is this different from projects like fluxcd/helm-controller?

Projects like fluxcd/helm-controller allow users to declaratively manage Helm charts from deployment to release, whereas this project only allows you lock an existing Helm chart release; as a result, the scope of this project is much more narrow than what is offered by fluxcd/helm-controller and should be integrable with any solution that produces Helm releases.

If you are looking for a larger, more opinionated solution that also has features around how Helm charts should be deployed onto a cluster (e.g. from a GitRepository or Bucket or HelmRepository), this is not the project for you.

However, if you are looking for something light-weight that simply guarentees that Helm is the only way to modify resources tracked by Helm releases, this is a good solution to use.

How does Helm Locker know whether a release was changed by Helm or by another source?

In order to prevent multiple Helm instances from performing the same upgrade at the same time, Helm 3 will always first update the info.status field on a Helm Release Secret from deployed to another state (e.g. pending-upgrade, pending-install, uninstalling, etc.) before performing the operation; once the operation is complete, the Helm Release Secret is expected to be reverted back to deployed.

Therefore, if Helm Locker observes a Helm Release Secret tied to a HelmRelease has been updated, it will check to see what the current status of the release is; if the release is anything but deployed, Helm Locker will not perform any operations on the resources tracked by this release, which will allow upgrades to occur as expected.

However, once a release is deployed, if what is tracked in the Helm secret is different than what is currently installed onto the cluster, Helm Locker will revert all resources back to what was tracked by the Helm release (in case a change was made to the resource tracked by the Helm Release while the release was being modified).

Developing

Which branch do I make changes on?

Helm Locker is built and released off the contents of the main branch. To make a contribution, open up a PR to the main branch.

For more information, see the Developing guide.

Debugging

How do I manually inspect the content of the Helm Release Secret to debug a possible Helm Locker issue?

Identify the release namespace (RELEASE_NAMESPACE), release name (RELEASE_NAME), and release version (RELEASE_VERSION) that identifies the Secret used by Helm to store the release data. Then, with access to your Kubernetes cluster via kubectl, run the following command (e.g. run base64 decode, base64 decode, gzip decompress the .data.release of the Secret):

RELEASE_NAMESPACE=default
RELEASE_NAME=test
RELEASE_VERSION=v1

# Magic one-liner! jq call is optional...
kubectl get secrets -n ${RELEASE_NAMESPACE} sh.helm.release.v1.${RELEASE_NAME}.${RELEASE_VERSION} -o=jsonpath='{ .data.release }' | base64 -d | base64 -d | gunzip -c | jq -r '.'

Building

make

Running

./bin/helm-locker

License

Copyright (c) 2020 Rancher Labs, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

helm-locker's Issues

Add automated testing suite to Helm Locker to prevent regressions across environments

Need unit, integration, and validation tests to be written in this repository to ensure that regressions can be caught as part of a CI process.

Creating an issue should automatically add a team/area3 label

Testing whether this works

Support suspending Helm Locker from reconciliation temporarily via spec fields

Need to add the ability to keep a HelmRelease unlocked after deployment

Add support for HelmLockerPolicy resources to allow admin to specify changes that are allowed

If you have a Helm release deployed but you don't want Helm Locker to reconcile changes to certain resources (identified by GVK), certain subsets of the spec of those resources (e.g. .spec.containers to allow something like Istio to inject a sidecar), or you want certain patches to be applied (e.g. adding nodeSelectors and tolerations to all Workloads deployed by the Helm chart), you need the ability to specify these configurations and persist them to the cluster for Helm Locker to use.

HelmLockerPolicy is a proposed resource that would select a list of HelmReleases (by label selector) and modify the apply call made to the ObjectSet on behalf of the HelmRelease to add an apply.Patcher to it that does custom sanitization based on the reconcile action provided.

The example manifest deploying a HelmLockerPolicy would look something like this:

kind: HelmLockerPolicy
apiVersion: helm.cattle.io/v1alpha1
metadata:
  name: allow-partial-mutation-on-configmaps
  namespace: cattle-helm-system
spec:
  releaseSelector: # required, identifies which HelmRelease objects are selected by metav1.labelSelector. If not provided, this policy is a noop. This HelmLockerPolicy will only apply to HelmRelease CRs in the same namespace as itself.
    matchLabels:
      helm.cattle.io/configmaps: partially-mutable # example label to use
  policies:
  - apiVersion: v1 # required
    kind: ConfigMap # required
    name: my-config-map # optional, mutually exclusive with  .spec.policies[i].selector. If both are provided, this will be chosen.
    selector: # optional, mutually exclusive with .spec.policies[i].name
      matchLabels:
        myworkload: true
    apply: # optional, the map provided here will override the values provided in the manifest for all targeted resources before they are parsed into an ObjectSet. It will not be validated that these patches are valid on the targeted resource, but if they are invalid the HelmRelease that selects this will be marked with the status FailedToApplyPolicy. If multiple selected patchers try to modify the same field, the HelmRelease will also be marked FailedToApplyPolicy. If you need to modify a list, provide _i (e.g. _1, _2, etc.) as the key to the map to indicate changes to resources in that index.
      data:
        config: "hello: world"
    ignorePaths: # paths that should be ignored on reconciling changes
    - '.data.config' # marks a path on which changes made are ignored. Providing `.` here will mark all paths as ignored and will allow changes to occur, although the resource will still be watched and recreated on deletion

Add ValidatingWebhookConfiguration endpoint to prevent changes to locked resources before attempts are made

Add ability to attach a ValidatingWebhookConfiguration to a /validate endpoint of Helm Locker that denies changes to objects tracked by an ObjectSet before they are persisted.

This is a safer option that overwriting changes that are introduced.

helm-locker TLS handshake timeout - failed to update lock

INFO[48281] applied cattle-monitoring-system/rancher-monitoring
E0415 08:42:35.218052 4898 leaderelection.go:367] Failed to update lock: Put "https://ronhead2.qa.rancher.space/k8s/clusters/c-k56pq/api/v1/namespaces/cattle-helm-system/configmaps/helm-locker-lock": net/http: TLS handshake timeout

Saw this this morning.

---

Note: cross-posted from https://github.com/aiyengar2/helm-locker/issues/7 on behalf of @ronhorton