rode / grafeas-elasticsearch

An implementation of the Grafeas storage backend based on Elasticsearch

rode

\rōd\ - a line (as of rope or chain) used to attach an anchor to a boat

rode provides the collection, attestation and enforcement of policies in your software supply chain.

Why rode?

Enterprises require a secure and reliable software delivery lifecycle to meet the needs of audit and compliance. This has traditionaly been implemented by applying governance and additional process. rode aims to meet this need by enabling Automated Governance. Automated Governance allows us to move the existing change approval process left by automating stages in software delivery that may currently exist as manual activities. This is possible by building a codified system of trust and authority for the entire software lifecycle. rode facilitates the collection and organization of important software supply chain metadata and provides a method of Automated Governance via Policy as Code.

rode Architecture

The overall architecture of rode is built around bringing together tools built with the needs of governance in mind. The system of rode consists of Collectors, the rode API, Grafeas, and Open Policy Agent. We have extended the Grafeas storage backend to use Elasticsearch. These tools work together to enable Automated Governance.

Collectors

Collectors package the metadata in the form of an "occurrence". These occurrences represent verifiable, individual software delivery process events. Collectors provide an entrypoint to the rode system by helping standardize the way metadata is brought in. They will be "purpose built" to collect metadata from any of the tools you are using in your software delivery toolchain.

Grafeas

Grafeas is an open source project that provides an API and storage layer for artifact metadata.

Information that is gathered by collectors is ultimately stored within Grafeas as Occurrences. Occurrences represent a particular piece of metadata about an artifact, and they can be used by Rode when evaluating policies against artifacts. Information such as an artifact's vulnerabilities, how it was built, and the quality of the codebase that produced the artifact can be fed to a policy in order to determine whether that artifact meets a certain set of standards set by your organization.

The primary way that Grafeas adds value to the rode project is through its models around how artifact metadata should be tracked and stored. Over time, we plan to add new types of Occurrences to Grafeas to represent artifact metadata concepts that we believe are important, but aren't currently represented in the existing models.

We currently use a custom backend for Grafeas that's based on Elasticsearch.

From Grafeas docs:

Grafeas is an open-source artifact metadata API that provides a uniform way to audit and govern your software supply chain. Grafeas defines an API spec for managing metadata about software resources, such as container images, Virtual Machine (VM) images, JAR files, and scripts. You can use Grafeas to define and aggregate information about your project's components. Grafeas provides organizations with a central source of truth for tracking and enforcing policies across an ever growing set of software development teams and pipelines. Build, auditing, and compliance tools can use the Grafeas API to store, query, and retrieve comprehensive metadata on software components of all kinds.

Open Policy Agent

Open Policy Agent, or OPA is the open source standard for implementing Policy as Code.

rode uses OPA as a means to apply and validate policy via Rego policies stored in source control (Poicy as Code). By using the occurrence and attestation metadata stored in Grafeas as inputs for policy, all resources can be validated as needed. rode provides a method to bring the policy and metadata together as a means for enabling Automated Governance.

The Open Policy Agent (OPA, pronounced “oh-pa”) is an open source, general-purpose policy engine that unifies policy enforcement across the stack. OPA provides a high-level declarative language that lets you specify policy as code and simple APIs to offload policy decision-making from your software. You can use OPA to enforce policies in microservices, Kubernetes, CI/CD pipelines, API gateways, and more.

Policy Evaluation

..

Installation

Helm

Add Helm repositories

helm repo add rode https://rode.github.io/charts
helm repo add elastic https://helm.elastic.co
helm repo update 

Install Rode

helm install rode rode/rode --set grafeas-elasticsearch.grafeas.elasticsearch.username=grafeas --set grafeas-elasticsearch.grafeas.elasticsearch.password=BAD_PASSWORD

See Rode Helm chart for more details.

Documentation

• Development
• API Reference

grafeas-elasticsearch's Issues

Specify document ids up front

Currently we have Elasticsearch generate a document id whenever a new occurrence, note, or project is created and then retrieve a single document by doing a query against one of its unique fields (e.g., occurrenceName).

Instead we should set the document id up front, using that unique field value, which will allow for a get to retrieve based on the document id.

This is more performant for reads and may remove the need to do a refresh on every write.

It should also give us stronger uniqueness guarantees in the index itself, rather than trying to manage that it application code (like we currently do for note names).

There's a downside in that the Elasticsearch auto-generated IDs can skip the uniqueness check, so there's overhead when creating the document. We may want to add some basic benchmark tests to see how much of a performance impact it is.

`UpdateOccurrence` should check that the field mask is set

In Grafeas, the mask is plucked off the request and passed to the storage layer without validation, so it can be nil. That causes this panic when the mask is updated:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xfa4252]
grafeas_1        |
goroutine 32145 [running]:
github.com/rode/grafeas-elasticsearch/go/v1beta1/storage.(*ElasticsearchStorage).UpdateOccurrence(0xc0005a3840, 0x13b6280, 0xc0007ab0e0, 0xc000536409, 0x4, 0xc00053641a, 0x24, 0xc000117000, 0x0, 0x4, ...)
	/workspace/go/v1beta1/storage/elasticsearch.go:397 +0x952
github.com/grafeas/grafeas/go/v1beta1/api.(*API).UpdateOccurrence(0xc0000c60a0, 0x13b6280, 0xc0007ab0e0, 0xc000432a00, 0xc0000c60a0, 0xc0007ab0e0, 0xc00028cba0)
	/go/pkg/mod/github.com/grafeas/[email protected]/go/v1beta1/api/occurrence.go:220 +0x2a4
github.com/grafeas/grafeas/proto/v1beta1/grafeas_go_proto._GrafeasV1Beta1_UpdateOccurrence_Handler(0x11de320, 0xc0000c60a0, 0x13b6280, 0xc0007ab0e0, 0xc000155080, 0x0, 0x13b6280, 0xc0007ab0e0, 0xc00047c480, 0x21e)
	/go/pkg/mod/github.com/grafeas/[email protected]/proto/v1beta1/grafeas_go_proto/grafeas.pb.go:2267 +0x217
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00036d880, 0x13c67b8, 0xc000244d80, 0xc000111100, 0xc0002f6e10, 0x1c18378, 0x0, 0x0, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1210 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc00036d880, 0x13c67b8, 0xc000244d80, 0xc000111100, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1533 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0000b8d10, 0xc00036d880, 0x13c67b8, 0xc000244d80, 0xc000111100)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:871 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/pkg/mod/google.golang.org/[email protected]/server.go:869 +0x1fd

Either this function could return with an error or initialize an empty mask if it's not set. Either way, malformed requests shouldn't crash the entire server, see #99

Check for existence of Note before creating Occurrence?

I've just noticed that you can create an Occurrence even if the Note referred to in noteName does not exist.

Filtering: Support array filtering

Add support to filter occurrences by fields in array. This should also include wildcard expressions.

The primary use case is to match a resource name (partial URI) to build occurrences BuildArtifacts list.

Implement grafeas helper methods

Implement GetOccurrenceNote, ListNoteOccurrences, and GetVulnerabilityOccurrencesSummary

Add Documentation showing how to use the support filters

Updates to occurrence details require an invalid field mask

From this discussion: rode/collector-build#4 (comment)

Occurrence details vary depending on the kind and so are implemented using a oneof field in the protobuf definition

Generally, oneof fields aren't treated as part of the message and instead the actual name of the set field is used. For instance, there's no details field when using protojson to serialize an occurrence. They also shouldn't appear in field masks.

However, the fieldmask-utils library used in occurrence updates requires that to be passed, see this issue for more details:

https://github.com/mennanov/fieldmask-utils/pull/15

This is easy to work around when calling Grafeas using gRPC; however, it's more of an issue with HTTP requests. Because of the design of the Grafeas protos, it's not possible to pass the field mask in the request; instead gRPC gateway fills in the mask automatically based on the request body.

As a result it doesn't seem possible to update the details of an occurrence using HTTP.

In terms of fixing it there doesn't seem to be an official method for merging that takes into account the field mask. See this issue:

https://github.com/golang/protobuf/issues/1296

There's a proto.Merge function; however it doesn't take a field mask. Its behavior is also such that arrays are concatenated, rather than replaced and that's not configurable.

It might be easiest to use something like mergo to handle the actual merge and then see what options are available for filtering the request body using the field mask.

Check for project existence before creating notes/occurrences

CreateOccurrence, BatchCreateOccurrences, CreateNote, and BatchCreateNotes should all fail if the underlying project does not exist.

Filtering: Support ">", "<", ">=", and "<=" operators

These operators should only work on fields where these operators make sense. Specifically, we should be able to apply these operators to timestamp fields.

Extract esutil.Client and types

A lot of our recent activity to this repo has been extending or tweak esutil.Client. This makes the releases of grafeas-elasticsearch very noisy -- most of the time the only change was to update the library.

We may want to extract esutil.Client and its types into their own repository.

Panics shouldn't crash the entire server

See #98 for an example of a request the currently crashes grafeas-elasticsearch. In Rode, we use the grpc-recovery middleware to catch panics and return codes.Internal.

Grafeas doesn't appear to expose any options for configuring the gRPC server with additional interceptors, so we might have to wrap each storage method implementation with a function to recover from panics.

Extract Elasticsearch IndexManager and migrations into its own package

In #60 we added a mechanism for applying index mappings that includes a method for migrating data after schema changes.

In order to re-use this code in Rode, we need to do a few things:

• move the code into its own Git repository
• change the IndexManager interface and remove the kind-specific methods (e.g., ProjectsIndex)
• update IndexManager.LoadMappings to be agnostic to the document kind
• update the IndexManager.LoadMappings tests, use Go 1.16's MapFS to avoid changing internal state
• expose options/configuration for index prefix, reindex polling interval, and number of poll attempts

Order of operations in filtering

can we specify parentheses around a statement (like happening in searching for resources) so we can do filtering likename && (type1 ||type 2)

Allow for paging through more results than the limit set by `index.max_result_window`

Came out of this discussion.

The documentation that Elasticsearch provides on pagination makes it sound like there is a hard cap on the number of results than can be paged through using from and size:

By default, you cannot use from and size to page through more than 10,000 hits. This limit is a safeguard set by the index.max_result_window index setting. If you need to page through more than 10,000 hits, use the search_after parameter instead.

We need to determine if that's the case or not by loading a number of notes or occurrences greater than index.max_result_window and attempting to page through them.

If it is, we'll need to make some changes to grab the sort value from the last hit in the results, encode that in the page token, and send it along in future requests as the search_after parameter.

Implement UpdateNote

Implement the UpdateNote method. This method gets its own story because it isn’t part of the MVP.

move bulk response error checking to esutil package

Should this be moved to esutil at some point?

Originally posted by @alexashley in rode/rode#118 (comment)

For paginated results, the page token for the final page should be an empty string

Currently, the last set of results in a series of pages includes a page token with a PIT and from value. Since there are no further results, paging beyond the last page just returns an empty result set.

Clients can check for that and end the pagination requests, but it would be ideal if the last page token was empty as a way to indicate that there are no more results.

Add chart version pipeline

Add action workflow to update helm chart version on new release

Implement UpdateOccurrence

Implement the UpdateOccurrence method. This methods gets its own story since it isn’t a part of the MVP.

use counterfeiter for mocks over mockgen

Also we should probably make an issue to move everything over to counterfeiter in Rode and greafeas-elasticsearch

Originally posted by @alexashley in #76 (comment)

https://github.com/maxbrunsfeld/counterfeiter

Add support for self-signed TLS certs

I've just been testing against an Elastic instance, hosted by Kubernetes, that was created using the community elastic-operator. By default the instance only allows TLS access and it generates self-signed certs to support this. The code in the current createESClient function fails when trying to create a new ES client due to an x509 error caused by these self-signed certs, however, it was fairly straight-forward to handle this by changing the code in the function to alter the ES config and set InsecureSkipVerify to true.
What are your thoughts about a more permanent fix with a flag to enable/disable support for self-signed certs (default false)?

Elasticsearch mapping migrations

As we introduce more functionality into grafeas-elasticsearch, we may make changes to the index mappings that cannot be applied to existing indices (like changing a field to the nested type).

We need to provide a mechanism in grafeas-elasticsearch that will migrate documents to new indices with updated mappings

Port over ES types from rode

Port over any new elasticsearch types into the esutil package.
Already in progress and should be completed today

Filter question

I've started evaluating this storage backend for Grafeas and it looks very interesting, so thanks for making it available :-)
I have a question about the list filters that are supported....
I can see that something like kind=="BUILD" works as expected but what if I want to query a nested element such as resource.uri? I tried resource.name=="myimage" but that generated a parsing error. Am I right in thinking this isn't supported yet?