No exported_pod in metrics about dcgm-exporter

[Muscule]

Which version of dcgm-exporter can I use to see exported_pod?

[Muscule]

I use dcgm-exporter-2.4.5-2.6.7-ubuntu20.04

[Muscule]

2.2.9-2.4.0-ubuntu20.04 didn't help

Nvidia gpu-operator 1.11.0

[jojotong]

You can check your servicemonitor config. Do you add honorLabels: true ？

[Muscule]

You can check your servicemonitor config. Do you add honorLabels: true ？

Yes

[Muscule]

curl localhost:9400/metrics gives nothing with "exported"

[Muscule]

Is there any solution?

[wotkddl21]

@Muscule
The exported_pod label is made by the prometheus, which is called server-side labels. So it is right that 'curl localhost:9400/metrics gives nothing with "exported"'.
Let assume you set the honor_label to "false".
Then when the prometheus scrapes the metrics, there could be some rules to relabel the source labels.
If you are using the servicemonitor to scrape dcgm-exporter in the kubernetes, then the additional label "node", "pod" and "namespace" would be created by the prometheus.
So these new labels will crash with the source labels from the dcgm-exporter.
In this case, the prometheus adds the prefix "exported" to the labels from the scraped data.

But you set honor_label to true, then the prometheus won't make server-side labels, which is keeping the source labels.

Check the link below
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

# If honor_labels is set to "true", label conflicts are resolved by keeping label
# values from the scraped data and ignoring the conflicting server-side labels.
#
# If honor_labels is set to "false", label conflicts are resolved by renaming
# conflicting labels in the scraped data to "exported_<original-label>" (for
# example "exported_instance", "exported_job") and then attaching server-side
# labels. 

[mossad-zika]

source metrics don't show me the info

[mossad-zika]

SM also doesn't do any relabel by default

https://github.com/NVIDIA/dcgm-exporter/blob/main/deployment/templates/service-monitor.yaml

[mossad-zika]

extraEnv:
  - name: "DCGM_EXPORTER_KUBERNETES"
    value: "true"
  - name: "DCGM_EXPORTER_KUBERNETES_GPU_ID_TYPE"
    value: "uid"

seems working, but testing now

BTW: nvidia's products are awesome and docs are total shit