aws-observability / helm-charts Goto Github PK

I'm trying to use the latest release via https://aws-observability.github.io/helm-charts/index.yaml, but it's still referencing v1.6.0 as the latest.

I'm trying to deploy the Operator on Fargate, which has a fargate taint, but obviously the DaemonSets can't run on Fargate notes. To support this, the chart needs to be updated to handle different tolerations between operator and daemonsets

• Add default tolerations (#41)

This changed the functionality of the chart, and should have either defaulted to [] to retain existing functionality, or bumped the chart to 2.0.0 to indicate the breaking change.

I'm not sure what is the correct place to report this, please direct me if this is not the correct place.

Goal:
I want to have EKS cluster with working observability, Bottlerocket AMI and GPU-nodes (g5* instances)
I use this helm chart by enabling amazon-cloudwatch-observability EKS add-on for my cluster.

Steps to reproduce:

1. I create latest version of EKS, GPU nodes with the last version of Bottlerocket AMI.
2. I use the latest nvidia-device-plugin ( https://github.com/NVIDIA/k8s-device-plugin/releases/tag/v0.15.0 )
3. I enable the latest version of the amazon-cloudwatch-observability EKS add-on (dcgm-exporter image 602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/observability/dcgm-exporter:3.3.3-3.3.1-ubuntu22.04 is used )
4. All related daemonsets except for dcgm-exporter work well.
5. DCGM-exporter containers has this in output:

time="2024-05-08T11:22:02Z" level=info msg="Starting dcgm-exporter"
Error: Failed to initialize NVML
time="2024-05-08T11:22:02Z" level=fatal msg="Error starting nv-hostengine: DCGM initialization error"

I guess this is some version incompatibility issue for the DCGM and nvidia driver (being installed to nodes via k8s-device-plugin ).
What should I do to make DCGM exporter work?

Trying to run version 1.7.0 with the command

helm install --wait --create-namespace --namespace amazon-cloudwatch amazon-cloudwatch aws-observability/amazon-cloudwatch-observability --set clusterName=my-ns --set region=us-east-1 --dry-run --debug

We are getting an error:

INSTALLATION FAILED: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "AmazonCloudWatchAgent" in version "cloudwatch.aws.amazon.com/v1alpha1", unable to recognize "": no matches for kind "DcgmExporter" in version "cloudwatch.aws.amazon.com/v1alpha1", unable to recognize "": no matches for kind "NeuronMonitor" in version "cloudwatch.aws.amazon.com/v1alpha1"]

I have verified that none of these CRDs are currently in the cluster.