Comments (25)
Hey, thanks for the report, I will check an installation with the latest prometheus stack and check if anything changed
from policy-reporter.
In a fresh setup with the latest versions everything works as expected and I encounter no issues.
Are the Policy Reporter dashboards the only ones that are not working?
Can you check that your Prometheus still has Policy Reporter as working target configured.
from policy-reporter.
I checked several others and they all work including the Kyverno one: https://kyverno.io/docs/monitoring/bonus-grafana-dashboard/
Targets also seem ok:
from policy-reporter.
Can you query the metrics over prometheus? Like policy_report_result
Not sure if it has problems with the data of your metrics, maybe some illegal character in the labels I am not aware of or the Dashboard Configurations. The error message is not really clear about it. Do you maybe see some more details in Grafana logs?
from policy-reporter.
One thing you could try is tu reinstall policy-reporter. I am not sure but maybe the upgrade introduced some issues with the Policy Reporter config maps that are managed by Policy Reporter Helm Chart.
I think the unexpected identifier issue comes from the empty value selections but I don't know why they are empty. The default should be All
. Maybe this Failed to call resource
is the reason but I can't say why it is failing.
from policy-reporter.
For policy_report_result I got -> Load time: 139ms Resolution: 14s Result series: 3081
Basically I got alot of results.
This is from the grafana log after I try to open the dashboard:
level=error msg="Failed to call resource" error="query must contain metric name\n"
level=error msg="Request Completed" method=GET path=/api/datasources/2/resources/series status=500
from policy-reporter.
Do you get this error on the PolicyReports
Dashboard? Do you get the same error for the PolicyReports Details
Dashboard or another one?
from policy-reporter.
I did re-install nothing changed.
All 3 "Policy Reporter" Dashboards have the same behavior.
from policy-reporter.
hm okay, and they all have the same error log? query must contain metric name
from policy-reporter.
So it's like this:
clusterpolicyreport-details
level=error msg="Failed to call resource" error="parse error at line 1, col 1: syntax error: unexpected IDENTIFIER"
policyreport-details
level=error msg="Failed to call resource" error="parse error at line 1, col 1: syntax error: unexpected IDENTIFIER"
policyreports
level=error msg="Failed to call resource" error="query must contain metric name\n"
from policy-reporter.
Can oyu please try this Query in your prometheus and Grafana Explore:
label_values({__name__=~ "policy_report_result|cluster_policy_report_result", status=~"fail|error"}, policy)
On your details pages, do you have any possible selection for on of the filters? Like All
or a concrete one.
from policy-reporter.
You mean these? Because it seems it can't find label_values? EDIT: Aha this is only valid for templating?
If you check the first picture I sent you can see there are 0 filters available.
from policy-reporter.
Hm yeah, my fault you can not check query functions in the explorer...
It seams that the issues all are related to the Variables.
Can you go to the "Dashboard Settings" of one of the Dashboards and check under "Variables" if they are working or returning some issues. Seems that the queries may not work for any reason.
from policy-reporter.
Could it be the data source maybe?
What is the data source for all of this?
I have 2 data sources in Grafana. Loki (default) and Prometheus.
When I edit a variable I see the datasource is Loki?
When I check the config maps with the dashboard json I see "datasource": "-- Grafana --"
from policy-reporter.
Hm could be, can you just try another one. In my case it uses Prometheus as source
from policy-reporter.
cat cluster-policy-reporter-details-dashboard.json | grep '"datasource":'
"datasource": "-- Grafana --",
cat policy-reporter-details-dashboard.json | grep '"datasource":'
"datasource": "-- Grafana --",
cat policy-reporter-dashboard.json | grep '"datasource":'
"datasource": "-- Grafana --",
"datasource": null,
"datasource": null,
"datasource": null,
"datasource": null,
"datasource": null,
This is from the json files on the Grafana pod.
When I edit a variable the datasource it set to Loki and I can't update it without an error. When I change data source to Prometheus and add the same Query I can update it with no problem.
So I guess the data source is the problem, but why did this start happening? And what should we do about it?
from policy-reporter.
I will create a new version of the Helm Chart with a new value for the datasource. Why this started now I don’t know but then you can simply change the source via the helm chart
from policy-reporter.
Yeah very strange... I can't remember was Loki always the default data source...
But this is definitely wrong right?
"targets": [
{
"datasource": {
"type": "loki",
"uid": "P8E80F9AEF21F6940"
},
"expr": "sum(cluster_policy_report_result{policy=~\"$policy\", category=~\"$category\", severity=~\"$severity\", source=~\"$source\", kind=~\"$kind\", status=\"pass\"})",
"instant": true,
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
This is just a snippet from one of the dashboards.
from policy-reporter.
yes, it should be Prometheus
"datasource": {
"type": "prometheus",
"uid": "prometheus"
}
Thats how it looks like in my environment. I think because it is not declared in the ConfigMap it falls back to the default, Loki in your case.
from policy-reporter.
I checked these dashboards because they have different data sources as a good example: https://github.com/fluxcd/flux2/tree/main/manifests/monitoring/monitoring-config/dashboards
The dashboards work, the Prometheus ones and Loki ones.
So as you said maybe it would be good if this could be altered via the Helm chart? Because currently my only option is to change the default data source, but that might break something else...
from policy-reporter.
Also here is the Kyverno dashboard which works:
https://kyverno.io/docs/monitoring/bonus-grafana-dashboard/
it uses DS_PROMETHEUS_KYVERNO variable which is set to prometheus
from policy-reporter.
I will have a look and reach back as soon as I have a fixed version. Thanks for your help so far
from policy-reporter.
@mceronja I released v2.10.1 with a fixed monitoring subchart. Prometheus should now be your default datasource and you have a new select to change it.
Let me know if it works or if you still have issues.
from policy-reporter.
@fjogeleit upgraded to 2.10.1 and it works.
Thanks for the help!
from policy-reporter.
Great, thanks for your feedback
from policy-reporter.
Related Issues (20)
- Cant' access ui dashboard using ingress path /ui HOT 5
- [bug] missing name and kind in email violations report
- [Bug] .Values.target.webhook.headers ignored in values.yaml (and maybe other values) HOT 2
- Problem with sending messages to Telegram HOT 2
- [UI] Standalone UI HOT 56
- Add ability to set request headers in Loki target HOT 2
- Download report html via api HOT 6
- Policy Reporter no longer working after kyverno 1.11.0 upgrade HOT 2
- Cant work with Workload Identity HOT 15
- Splunk as a target HOT 1
- [DOCS] The external-cluster section needs to mention that rest.enabled: true is required HOT 1
- Policy Reporter Targets as CRDs HOT 4
- Include the timestamp HOT 7
- Policy Reporter not aligned with Kyverno HOT 5
- Helm chart won't start due to "Error: unknown flag: --template-dir" HOT 2
- Kyverno Policy Reporter writes extensively on node disk storage in certain ocassions. HOT 2
- strange S3 config error when using SecurityHub as target HOT 3
- Helm: Ingress template has wrong selection for pathType
- Allow for not rendering a Kubernetes Secret HOT 2
- policy-reporter presents incomplete results HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from policy-reporter.