Policy reporter Grafana dashboards stopped working about policy-reporter

[fjogeleit]

Hey, thanks for the report, I will check an installation with the latest prometheus stack and check if anything changed

[fjogeleit]

In a fresh setup with the latest versions everything works as expected and I encounter no issues.

Are the Policy Reporter dashboards the only ones that are not working?

Can you check that your Prometheus still has Policy Reporter as working target configured.

[mceronja]

I checked several others and they all work including the Kyverno one: https://kyverno.io/docs/monitoring/bonus-grafana-dashboard/
Targets also seem ok:

[fjogeleit]

Can you query the metrics over prometheus? Like policy_report_result

Not sure if it has problems with the data of your metrics, maybe some illegal character in the labels I am not aware of or the Dashboard Configurations. The error message is not really clear about it. Do you maybe see some more details in Grafana logs?

[fjogeleit]

One thing you could try is tu reinstall policy-reporter. I am not sure but maybe the upgrade introduced some issues with the Policy Reporter config maps that are managed by Policy Reporter Helm Chart.

I think the unexpected identifier issue comes from the empty value selections but I don't know why they are empty. The default should be All. Maybe this Failed to call resource is the reason but I can't say why it is failing.

[mceronja]

For policy_report_result I got -> Load time: 139ms   Resolution: 14s   Result series: 3081
Basically I got alot of results.

This is from the grafana log after I try to open the dashboard:

level=error msg="Failed to call resource" error="query must contain metric name\n"
level=error msg="Request Completed" method=GET path=/api/datasources/2/resources/series status=500

[fjogeleit]

Do you get this error on the PolicyReports Dashboard? Do you get the same error for the PolicyReports Details Dashboard or another one?

[mceronja]

I did re-install nothing changed.
All 3 "Policy Reporter" Dashboards have the same behavior.

[fjogeleit]

hm okay, and they all have the same error log? query must contain metric name

[mceronja]

So it's like this:

clusterpolicyreport-details

level=error msg="Failed to call resource" error="parse error at line 1, col 1: syntax error: unexpected IDENTIFIER"

policyreport-details

level=error msg="Failed to call resource" error="parse error at line 1, col 1: syntax error: unexpected IDENTIFIER"

policyreports

level=error msg="Failed to call resource" error="query must contain metric name\n"

[fjogeleit]

Can oyu please try this Query in your prometheus and Grafana Explore:

label_values({__name__=~ "policy_report_result|cluster_policy_report_result", status=~"fail|error"}, policy)

On your details pages, do you have any possible selection for on of the filters? Like All or a concrete one.

[mceronja]

You mean these? Because it seems it can't find label_values? EDIT: Aha this is only valid for templating?

If you check the first picture I sent you can see there are 0 filters available.

[fjogeleit]

Hm yeah, my fault you can not check query functions in the explorer...

It seams that the issues all are related to the Variables.

Can you go to the "Dashboard Settings" of one of the Dashboards and check under "Variables" if they are working or returning some issues. Seems that the queries may not work for any reason.

[mceronja]

Could it be the data source maybe?
What is the data source for all of this?

I have 2 data sources in Grafana. Loki (default) and Prometheus.
When I edit a variable I see the datasource is Loki?

When I check the config maps with the dashboard json I see "datasource": "-- Grafana --"

[fjogeleit]

Hm could be, can you just try another one. In my case it uses Prometheus as source

[mceronja]

cat cluster-policy-reporter-details-dashboard.json | grep '"datasource":'
        "datasource": "-- Grafana --",

cat policy-reporter-details-dashboard.json | grep '"datasource":'
        "datasource": "-- Grafana --",

cat policy-reporter-dashboard.json | grep '"datasource":'
        "datasource": "-- Grafana --",
    "datasource": null,
    "datasource": null,
    "datasource": null,
    "datasource": null,
    "datasource": null,

This is from the json files on the Grafana pod.

When I edit a variable the datasource it set to Loki and I can't update it without an error. When I change data source to Prometheus and add the same Query I can update it with no problem.

So I guess the data source is the problem, but why did this start happening? And what should we do about it?

[fjogeleit]

I will create a new version of the Helm Chart with a new value for the datasource. Why this started now I don’t know but then you can simply change the source via the helm chart

[mceronja]

Yeah very strange... I can't remember was Loki always the default data source...

But this is definitely wrong right?

      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "P8E80F9AEF21F6940"
          },
          "expr": "sum(cluster_policy_report_result{policy=~\"$policy\", category=~\"$category\", severity=~\"$severity\", source=~\"$source\", kind=~\"$kind\", status=\"pass\"})",
          "instant": true,
          "interval": "",
          "legendFormat": "",
          "refId": "A"
        }
      ],

This is just a snippet from one of the dashboards.

[fjogeleit]

yes, it should be Prometheus

        "datasource": {
          "type": "prometheus",
          "uid": "prometheus"
        }

Thats how it looks like in my environment. I think because it is not declared in the ConfigMap it falls back to the default, Loki in your case.

[mceronja]

I checked these dashboards because they have different data sources as a good example: https://github.com/fluxcd/flux2/tree/main/manifests/monitoring/monitoring-config/dashboards

The dashboards work, the Prometheus ones and Loki ones.

So as you said maybe it would be good if this could be altered via the Helm chart? Because currently my only option is to change the default data source, but that might break something else...

[mceronja]

Also here is the Kyverno dashboard which works:
https://kyverno.io/docs/monitoring/bonus-grafana-dashboard/

it uses DS_PROMETHEUS_KYVERNO variable which is set to prometheus

[fjogeleit]

I will have a look and reach back as soon as I have a fixed version. Thanks for your help so far

[fjogeleit]

@mceronja I released v2.10.1 with a fixed monitoring subchart. Prometheus should now be your default datasource and you have a new select to change it.

Let me know if it works or if you still have issues.

[mceronja]

@fjogeleit upgraded to 2.10.1 and it works.

Thanks for the help!

[fjogeleit]

Great, thanks for your feedback