Restart needed to remove old records for Pod (and maybe other) resources about policy-reporter

[fjogeleit]

Okay, I will try to reproduce and fix it.

[fjogeleit]

You are welcome, if the issue still exist it would be great if you can provide an example metric item that exists after the resource is removed.

The Helm Chart is up to date, I mean Policy reporter Helm Chart v2.6.0 I think Policy Reporter is on 2.3.0. Sorry for the confusion.

[fjogeleit]

I found the possible reason for your issue in the debouncer logic. I released v2.6.3 of the Helm Chart with a new version of Policy Reporter (v2.3.2) which should fix the issue. I hope this solves your problem. Please let me know if it still not work as expected.

[fjogeleit]

Hey, thanks for reporting. You face this issue in the UI, Metrics or in which situation? I will have a look if some remove logic does not work as expected.

Do you have some basic steps and policy (if its a custom one) to reproduce this issue?

[alen-z]

Hey, thx for the quick response. We collect metrics with Prometheus and visualize in Grafana (there I saw old records). We do not use Policy Reporter UI.

In terms of steps, we did not change the policy, only workload. After changing the workload, pass was registered and old Pods (that were terminated) were still showing as failing in Policy Reporter even though those Pods were replaced by new ones (did not exist in cluster).

[fjogeleit]

Okay, thanks. So the issue relates to the metrics provided by policy reporter? I will investigate into this issue.

One Thing, did I understand correctly that your policy validates on Pod level and not against a deployment / deamonset or something similar? Just in case that it matter.

[alen-z]

issue relates to the metrics provided by policy reporter

It seems since restart of the Policy Reporter helps.

policy validates on Pod level and not against a deployment / deamonset or something similar

I do not remember correctly, but I'm sure those were Pods as leftovers. Not sure if there were resources like Deployment passing and being removed from failing state. Sorry, can not confirm and it's best I can do at this point.

[fjogeleit]

I tried to reproduce it but it worked as expected. How so you encounter the issue? So the deleted resource is still visible in a Graph / Table or something similar in Grafana or do you see an item of the deleted resource in the metrics endpoint?

What I did to reproduce:

Installed Policy Reporter 2.6.0 and Kyverno 1.6.0

Start a simple nginx:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: test
  name: test
spec:
  containers:
  - image: nginx:alpine
    name: test
    resources: {}
status: {}

Which failed PSP disallow-capabilities-strict and get an item in the metrics:

policy_report_result{category="Pod Security Standards (Restricted)",kind="Deployment",name="test",namespace="policy-reporter",policy="disallow-capabilities-strict",report="polr-ns-policy-reporter",rule="autogen-require-drop-all",severity="medium",status="fail"} 1

I fixed it with

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: test
  name: test
spec:
  containers:
  - image: nginx:alpine
    name: test
    resources: {}
    securityContext:
      capabilities:
        drop:
          - "ALL"

It removed the old item and replaced it with:

policy_report_result{category="Pod Security Standards (Restricted)",kind="Pod",name="test",namespace="policy-reporter",policy="disallow-capabilities-strict",report="polr-ns-policy-reporter",rule="require-drop-all",severity="medium",status="fail"} 1

I tried the same with a Deployment and had the same result.

[alen-z]

We're on Policy Reporter 2.1.0. So based on your report we'll plan the move to 2.6.0. If there will be any further issues we'll reopen the issue. Feel free to close and thank you for quick response :)

Edit: Looking forward to updated Helm chart to use newest Policy Reporter version.