ckotzbauer / vulnerability-operator Goto Github PK

In a cluster (still) using dockershim, the image ID is prefixed with docker-pullable:// by the runtime. As a consequence, the comparison in kubernetes.go#L74 fails for all images.

As a quick workaround, I removed the prefix from c.ImageID:

--- a/internal/vuln/kubernetes/kubernetes.go
+++ b/internal/vuln/kubernetes/kubernetes.go
@@ -2,6 +2,7 @@ package kubernetes

 import (
        "context"
+       "strings"

        "github.com/sirupsen/logrus"
        corev1 "k8s.io/api/core/v1"
@@ -71,7 +72,8 @@ func (client *KubeClient) GetContainersWithImage(imageID string) ([]ContainerInf
                statuses = append(statuses, p.Status.EphemeralContainerStatuses...)

                for _, c := range statuses {
-                       if c.ImageID == imageID {
+                       fixedImageID := strings.ReplaceAll(c.ImageID, "docker-pullable://", "")
+                       if fixedImageID == imageID {
                                infos = append(infos, ContainerInfo{
                                        Namespace:     p.Namespace,
                                        PodName:       p.Name,

If something like this makes sense as a patch, I'd happily create a PR. But I don't know if k8s.io/client-go offers something more elegant.

While I was debugging an issue, I decided to enable the logging package in the grype module so that I could see if my sbom was actually being processed (I received no results initially, but resolved the issue -- see #386 for details). I wondered if you wanted me to submit a PR with this modification. I fed the verbosity argument into the grype logging module in order to do this. If this logging feature is desired, perhaps it would be better to have a new argument called --grype-verbosity or something similar because it is extremely verbose, and the logging may not be desired in most cases.

Let me know if:

1. you would like me to submit a PR with this modification
2. if "yes", do you want me to add a separate argument called --grype-verbosity or similar so that there is not a flood of output in the standard vulnerability-operator use case

Another thing is that I would like to attempt is to enable logging from the prometheus metrics module. I feel this will be useful to see when/if the metrics are being scraped and whatever other debug output the prometheus endpoint code can provide. Let me know if you think this would be something that you would want in the main code base. I can't guarantee this feature at this moment because I haven't looked into how easy or hard it is. The grype logging was non-trivial because of I've been learning Go along the way, and the Grype logging module makes you pass some "extra stuff" which turns out to be useless but necessary to have things operate properly.

BTW thanks for this project, it has filled a use case that I had initially started writing from scratch, but your features check a lot of boxes for what I want to accomplish, and after some tweaking it works beautifully. Also it has helped me to learn Go as I had to figure out why certain things were going wrong along the way to getting everything working :)

I think I understand the filter file conditions (because there is only way that makes sense to me), but I just wanted to confirm how it worked.

This audit and ignore are each basically a big OR (vulns/packages ORed), where each vuln/package is another OR (list of contexts ORed), where each context is an AND (image, namespace, etc ANDed).

For example, these two blocks are equivalent:

audit:
  - vulnerability: CVE-X
    context:
    - image: IMAGE1
      namespace: NAMESPACE1
  - vulnerability: CVE-X
    context:
    - image: IMAGE2
      namespace: NAMESPACE2

audit:
 - vulnerability: CVE-X
   context:
   - image: IMAGE1
     namespace: NAMESPACE1
   - image: IMAGE2
     namespace: NAMESPACE2

Correct?

Thanks

It might occur, that a CVE is a permanent false positive (e.g. rubygem rexml 3.2.3.1 in gitlab/gitlab-ce:15.5.6-ce.0) or a container is unaffected by a CVE.

In order to focus on the real issues, we'd like our view (Grafana consuming prometheus metric) to filter these CVEs. An idea was to have another label "audit_result" in the prometheus metric vuln_operator_cves.

How do you handle this?
In which ways could vulnerability_operator support audits?

cc/ @nicholasdille

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (github>ckotzbauer/renovate-config:default)

I was wondering what is the intended purpose of audit rules and ignore rules? Me and my team have guessed that ignore rules are for false positives. We have also guessed that audit rules are for CVEs that we have reviewed and determined are true positives, but we have justification or some other reason that we have determined are not a threat. Is this true?

Also, what is the purpose of grype-config-file and filter-config-file? They seem to have the same general function, except grype-config-file is only for ignores. Does grype-config-file have the same format documented at https://github.com/anchore/grype#specifying-matches-to-ignore and is simply passed directly to the grype library, whereas the filter-config-file is filtered after the fact by the vulenerability-operator itself? If this is try, is there a reason to use one over the other for ignore rules?

Sources

• Load SBOMs from Git-Repository (previously created from sbom-operator)
• Cron-Trigger (like sbom-operator)
• Webhook-Trigger (e.g. called from sbom-operator)

Targets

• Prometheus-Metrics (⚠️ needs more specification)
• Messaging (How to avoid sending the same messages for found CVEs on each scan?)
• Report generation
  • READMEs
  • Web-Report served from vulnerability-operator itself or uploaded to a destination
  • JSON-Report served from vulnerability-operator itself
• PolicyReport-CRDs (maybe there's a way to include this in Kyverno's Policy-Reporter)

Scanning

• Integrate grype-golang (https://github.com/anchore/grype/blob/v0.32.0/cmd/root.go)

CVE-Filtering-Options

• Only fixed
• Severity-Threshold
• Ignorelist

Build / Security

• GoReleaser
• Release-Pipeline from https://github.com/ckotzbauer/actions-toolkit/blob/main/.github/workflows/toolkit-release-goreleaser.yml
• OIDC-signed artifacts and images via cosign
• SBOMs
• SLSA provenance
• Docker-Image from scratch

Deployment

• Plain Kubernetes-YAMLs
• Helm-Chart
• Built-in (but optional) ServiceMonitor for Prometheus-Operator CRD

Preamble:

When I first tried out installing this app, I manually generated a single sbom and put it in a repo. I was getting an empty results.json file. I hacked the code and enabled debugging output from the grype library. I found from the grype debug output that it was in fact running on my sbom and finding vulnerabilities, so I figured that the results were being filtered out for some reason. I ran the code through a debugger, and I found the function which extracted the "ImageID" from the file path, which puzzled me at first. It would always return ., which wouldn't match any containers in kubernetes. Then when I was looking at the sbom-operator, and saw the directory structure that it used. It then dawned on me that this directory structure was a requirement for vulnerability-operator to function correctly. My initial thought was that I should modify the code to extract the imageID from the sbom, but given the different sbom types and even differences between schemas of the same type (syft for example), this may take some effort to ensure this is done in a robust way.
Also it was not clear to me from the documentation that the results are also filtered out if there are no matching containers in the cluster. It would be nice at some point that the scanning of sboms could be decoupled from actively running containers in a cluster. For example, if I want to just scan all of the images in my registry that I have created sboms, or even just sboms that I have generated from a list of images that I have a particular interest in. I may open an issue for this at some point, but for the time being the functionality of scanning my cluster images is a very nice start.

Request:

Document that required file structure. Perhaps even just a link to the sbom-operator README.md indicating that this is the required file structure.

Document the fact that there must be containers using your scanned images in order for vulnerabilities to show up in your reports/metrics.

I've been getting some incorrect results with filter files. Upon studying the code, I found that there are two key parts of the code which are flawed. I have generated my filter files according the the documentation and the answer here: #420. I have created a fix and will be submitting the pull request (edit: submitted, #445), but here I will document the issues I've been seeing.

The first issue originates here:

The error here is that there may be a filter in the array whose context array matches one container, and then another filter later in the array whose context array matches another container. However, the algorithm will shortcut out once it hits ANY filter which matches ANY container.

Secondly, there is a flaw here:

The code here is basically saying "for each context, add to the applied list any containers which match. add to the not applied list any containers which don't match". For certain filter files, you end up adding containers to both the applied and not applied lists. The correct logic should be "for each container, if it matches any context, add it to the applied list. if it matches no context, add it to the not applied list.

I wrote test cases to illustrate these issues here: samcornwell@d13d251. The current release of the vulnerability-operator fails these tests.

Here they are, reformatted to be a little easier to read:

{
    config: FilterConfig{
        Audit: []VulnerabilityFilter{
            {
                Vulnerability: "CVE-2023-0215", 
                Context: []FilterContext{
                    {
                        Namespace: "monitoring", 
                        Kind: "Deployment", 
                        Name: "grafana"
                    }
                }
            }, 
            {
                Vulnerability: "CVE-2023-0215", 
                Context: []FilterContext{
                    {
                        Namespace: "monitoring", 
                        Kind: "Deployment", 
                        Name: "kube-state-metrics"
                    }
                }
            }
        }
    },
    vulnList: []vuln.Vulnerability{
        libcryptoVulnMultiDeployment
    },
    audited: []vuln.Vulnerability{
        {
            ID: "CVE-2023-0215", 
            Package: "libcrypto3", 
            Severity: "High", 
            Type: "apk", 
            ImageID: "alpine:3.17", 
            Containers: []kubernetes.ContainerInfo{
                {
                    Namespace: "monitoring", 
                    OwnerName: "grafana", 
                    OwnerKind: "Deployment", 
                    PodName: "grafana-6d54bf9447-t7dfc"
                }, 
                {
                    Namespace: "monitoring", 
                    OwnerName: "kube-state-metrics", 
                    OwnerKind: "Deployment", 
                    PodName: "kube-state-metrics-6d54bf9447-t7dfc"
                }
            }
        }
    },
    found: []vuln.Vulnerability{}
},
{
    config: FilterConfig{
        Audit: []VulnerabilityFilter{
            {
                Vulnerability: "CVE-2023-0215", 
                Context: []FilterContext{
                    {
                        Namespace: "monitoring", 
                        Kind: "Deployment", 
                        Name: "grafana"
                    }, 
                    {
                        Namespace: "monitoring", 
                        Kind: "Deployment", 
                        Name: "kube-state-metrics"
                    }
                }
            }
        }
    },
    vulnList: []vuln.Vulnerability{
        libcryptoVulnMultiDeployment
    },
    audited: []vuln.Vulnerability{
        {
            ID: "CVE-2023-0215", 
            Package: "libcrypto3", 
            Severity: "High", 
            Type: "apk", 
            ImageID: "alpine:3.17", 
            Containers: []kubernetes.ContainerInfo{
                {
                    Namespace: "monitoring", 
                    OwnerName: "grafana", 
                    OwnerKind: "Deployment", 
                    PodName: "grafana-6d54bf9447-t7dfc"
                }, 
                {
                    Namespace: "monitoring", 
                    OwnerName: "kube-state-metrics", 
                    OwnerKind: "Deployment", 
                    PodName: "kube-state-metrics-6d54bf9447-t7dfc"
                }
            }
        }
    },
    found: []vuln.Vulnerability{}
}

Running go mod tidy after cloning repo complains about opentelemetry package not containing required module

go: downloading go.opentelemetry.io/otel/metric v0.30.0
github.com/ckotzbauer/vulnerability-operator/internal/vuln/grype imports
	github.com/anchore/grype/grype/pkg imports
	github.com/sigstore/cosign/pkg/signature imports
	github.com/sigstore/cosign/pkg/cosign imports
	github.com/sigstore/cosign/cmd/cosign/cli/fulcio/fulcioverifier/ctl imports
	github.com/google/certificate-transparency-go imports
	go.etcd.io/etcd/v3 imports
	go.etcd.io/etcd/tests/v3/integration imports
	go.etcd.io/etcd/server/v3/embed imports
	go.opentelemetry.io/otel/semconv: module go.opentelemetry.io/otel@latest found (v1.7.0), but does not contain package go.opentelemetry.io/otel/semconv
github.com/ckotzbauer/vulnerability-operator/internal/vuln/grype imports
	github.com/anchore/grype/grype/pkg imports
	github.com/sigstore/cosign/pkg/signature imports
	github.com/sigstore/cosign/pkg/cosign imports
	github.com/sigstore/cosign/cmd/cosign/cli/fulcio/fulcioverifier/ctl imports
	github.com/google/certificate-transparency-go imports
	go.etcd.io/etcd/v3 imports
	go.etcd.io/etcd/tests/v3/integration imports
	go.etcd.io/etcd/server/v3/embed imports
	go.opentelemetry.io/otel/exporters/otlp imports
	go.opentelemetry.io/otel/sdk/metric/controller/basic imports
	go.opentelemetry.io/otel/metric/registry: module go.opentelemetry.io/otel/metric@latest found (v0.30.0), but does not contain package go.opentelemetry.io/otel/metric/registry

Hi,
according to the documentation: This operator scans all SBOMs from a git-repository for vulnerabilities using Grype

The sbom-operator could generate a SBOM and store it into an OCI-Registry.

Do you think it is possible to support OCI Registry in vulnerability-operator

Add the possibility to use the vulnerability-operator standalone without the sbom-operator.