equinor / radix-flux Goto Github PK

• Review Epic notes
• Upgrade

• Review Epic notes
• Upgrade

equinor/radix-platform#1007

Change the configuration for cert issuer

Dev: no
Playground: no
Platform: yes
c2: yes

Strategy to decide:

• Nodeselector (simple)
• affinity / antiaffinity

Components

• azure-service-operator
• blob-csi-driver
• cert-manager
• external-secrets-operator
• grafana
• ingress-nginx
• keda
• kube-prometheus-stack
• kubernetes-replicator
• kured
• (radix)prometheus-guard DEV #2169
• (radix)prometheus-guard All envs
• equinor/radix-acr-cleanup#83
• equinor/radix-cicd-canary#205
• radix-cluster-cleanup
• radix-cost-allocation
• radix-operator
• radix-pipeline-runner
• radix-tekton
• radix-image-builder (ACR tasks)
• radix-builder with buildah (build-kit)
• radix-vulnerability-scanner
  • fix database workflow service principals - equinor/radix-platform#1421
• tekton-pipeline-runs
• velero, radix-velero-plugin
• workload-identity-webhook

• Review Epic notes
• Upgrade

Now replaced by a common structure for both Lets Encrypt and DigiCert

(Cleanup after implementation of new automatic certificate feature)

Upgrade to newer version of Flux

Fix issue on clusters:

• c2-prod-34
• eu-34
• playground-07

Containers shouldn't run as root users in your Kubernetes cluster. Running a process as the root user inside a container runs it as root on the host. If there's a compromise, an attacker has root in the container, and any misconfigurations become easier to exploit.

• Fix the security context for Tekton init container
• Fix or make exception for Velero

Manual remediation:

1. From the Unhealthy resources tab, select the cluster. Defender for Cloud lists the relevant pods.
2. For these pods, ensure the runAsUser property is set to a non-zero value or set property runAsNonRoot=true.
3. After making your changes, redeploy the pod with the updated rule.

• Review Epic notes
• Upgrade

Add configuration for

• dev
• playground
• platform
• c2
• extmon

Note that v0.40.0 contained breaking changes
https://github.com/fluxcd/flux2/releases/tag/v0.40.0

The autologin flags (--aws-autologin-for-ecr, --gcp-autologin-for-gcr and --azure-autologin-for-acr) have been deprecated to bring the Image API closer to the Source API, where cloud provider contextual login is configured at object level with .spec.provider. Usage of these flags will result in a logged error. Please update all the ImageRepository manifests that require contextual login with the new field .spec.provider and the appropriate cloud provider value; aws, gcp, or azure. Refer the docs for more details and examples.

Investigate what settings (https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/) can reduce the high memory usage without breaking anything, like large requests and responses

https://github.com/fluxcd/flux2/releases

Running containers as root user should be avoided
Containers shouldn't run as root users in your Kubernetes cluster. Running a process as the root user inside a container runs it as root on the host. If there's a compromise, an attacker has root in the container, and any misconfigurations become easier to exploit.
#velero

Containers should run with a read only root file system in your Kubernetes cluster. Immutable filesystem protects containers from changes at run-time with malicious binaries being added to PATH.

Components with issues

• cert-manager #2007 (newest release uses read only root by default)
• grafana #2008 #2012
• kube-state-metrics #2010
• velero/velero #2011
• ingress-nginx-controller Blocked: https://github.com/kubernetes/ingress-nginx/ Issue 9504
• tekton
  • Dev: #2029
  • Platform, Playground, C2: #2049
• kubernetes replicator #2047 (needs runAsNonroot too)

Upgrade to same version as the other clusters.

Ref. https://github.com/equinor/radix/issues/151#issuecomment-2284110860

example on slow action run: move custom ingress run 23
example on unreachable cluster: move custom ingress run 20

• Check if monitor addon is enabled before enabling
• Exit action if cluster is unreachable
• Exit action if kubelogin failed
• fix Waiting for AAD role to propagate

Perm:

• 2023-05-11T12:32:09.0652279Z Message: The client 'dd4dd75c-6e56-4c2b-9404-e76d2c29c67f' with object id 'dd4dd75c-6e56-4c2b-9404-e76d2c29c67f' does not have authorization or an ABAC condition not fulfilled to perform action 'Microsoft.Authorization/roleAssignments/write' over scope '/subscriptions/**_/resourceGroups/clusters/providers/Microsoft.ContainerService/managedClusters/weekly-19/providers/Microsoft.Authorization/roleAssignments/c25cdbcc-5afc-4a57-899f-01b1de71f50e' or the scope is invalid. If access was recently granted, please refresh your credentials.

Update done in Flux

radix-operator pr. cluster

Change the configuration for cert-manager

• dev
• playground
• platform
• c2
• extmon

API changes
https://github.com/fluxcd/flux2/releases/tag/v2.3.0

• #1857
• #1858
• #1859
• #1860
• #1861
• #2210

Features and improvements:
https://fluxcd.io/blog/2024/05/flux-v2.3.0/

Cool feature:
flux reconcile helmrelease <release> --reset
flux reconcile helmrelease <release> --force

Installing or upgrading Flux
To upgrade the APIs, make sure the new Custom Resource Definitions and controllers are deployed, and then change the manifests in Git:

1. Set apiVersion: helm.toolkit.fluxcd.io/v2beta2 in the YAML files that contain HelmRelease definitions.
2. Set apiVersion: notification.toolkit.fluxcd.io/v1beta3 in the YAML files that contain Alert and Provider definitions.
3. Set apiVersion: image.toolkit.fluxcd.io/v1beta2 in the TAML files that contain ImageRepository, ImagePolicy and ImageUpdateAutomation
4. Commit, push and reconcile the API version changes.

Bumping the APIs version in manifests can be done gradually. It is advised to not delay this procedure as the deprecated versions will be removed after 6 month

Kubernetes cluster pod security baseline standards reports pod running privileged
-kube-prometheus-stack
-kured

We should move these deployments to "kube-system" namespace instead of "Default"

• Review Epic notes
• Upgrade

• Add wildcard records. Cluster-specific, active-cluster and app alias
• Delete existing custom CNAMES, e.g. console.radix.equinor.com
• Patch radix-flux and remove external-dns
• Remove external-dns HelmRelease
• Remove existing external-dns managed records
• Remove unused external-dns manifests in radix-flux

Adding an extra issuer to be able to use DigiCert as cert issuer

• development
• playground
• platform
• c2

• use different branches for each environment
• Add environment to title

A fix for this issue is released in Blobfuse v2.1.1 or higher. However, this version of Blobfuse will only be available with a newer version of the Blob-CSI driver. We are currently waiting for this new release.

We are currently running:
blob-csi: 1.23.2
blobfuse2: 2.1.0

links:
blob-csi-driver blobfuse2 version: https://github.com/kubernetes-sigs/blob-csi-driver/blob/c53028ea6b024c4d1c1fea435edbbee3a83e9af3/deploy/csi-blob-node.yaml#L61
fix for truncated files: Azure/azure-storage-fuse#1142

grafana/grafana#91993

#################################################################################

BREAKING: The config values passed contained no longer accepted

options. See the messages below for more details.

To verify your updated config is accepted, you can use

the helm template command.

#################################################################################

ERROR: Please make .configuration.backupStorageLocation from map to slice

ERROR: Please make .configuration.volumeSnapshotLocation from map to slice

REMOVED: .configuration.provider has been removed, instead each backupStorageLocation and volumeSnapshotLocation has a provider configured

warning: Upgrade "velero" failed: post-upgrade hooks failed: unable to build kubernetes object for post-upgrade hook velero/templates/backupstoragelocation.yaml: error validating "": error validating data: [ValidationError(BackupStorageLocation.spec.objectStorage): missing required field "bucket" in io.velero.v1.BackupStorageLocation.spec.objectStorage, ValidationError(BackupStorageLocation.spec): missing required field "provider" in io.velero.v1.BackupStorageLocation.spec]