flipt-io / cup Goto Github PK

More often than not, controllers will be defined and scoped to a set of supported resource types.

For example, the Flipt controller handles Flipt and Segment resource types only.

It should therefor be possible to bundle both the Controller WASM binary and its support definitions in a single artefact. The most promising format is likely OCI.

With the https://github.com/oras-project/oras-go project we could add support for bundling WASM binaries with their associated resources type directly into cup and cupd. cupd could support sourcing controllers and definitions directly from OCI registries.

Here are some useful links for design ideas around packing:

• ORAS Artifact Concepts I think it might be best to store
• Helm Proposal for their OCI bundling support

Quick sketch of what a manifest might look like for cup:

{
  "schemaVersion": 2,
  "artifactType": "application/vnd.io.flipt.cup.controller+type",
  "config": {
    "mediaType": "application/vnd.oci.empty.v1+json",
    "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
    "size": 2
  },
  "layers": [
    {
      "mediaType": "application/vnd.io.flipt.cup.controller.content.v1.tar+gzip",
      "digest": "sha256:1b251d38cfe948dfc0a5745b7af5ca574ecb61e52aed10b19039db39af6e1617",
      "size": 7364
    },
    {
      "mediaType": "application/vnd.io.flipt.cup.resource.v1+json",
      "digest": "sha256:3e207b409db364b595ba862cdc12be96dcdad8e36c59a03b7b3b61c946a5741a",
      "size": 384
    }
  ]
}

cup subcommands

One of more new subcommand(s) would be useful for packing, publishing and pulling Cup controllers.

We will need to support users being able to:

• bundle together wasm controller binaries with resource definitions
• pushing named and tagged images to OCI registries
• (optional/stretch) pulling and unpacking / introspecting remote images

cupd controller definition

The Controller definition will need extending to support referencing a registry and tag.
The controller will need to:

• establish a local or inmemory oras store
• copy the target from the remote into the store (if local we could cache between starts of cup somewhere)
• configure the resource definitions found in the image
• compile and configure the controller with the wasm binary

We should update the description of the cli

workspace/cup - [main] » cup --help
NAME:
   cup - A new cli application

There are a number of pieces currently hard-coded that should be moved out as configuration or API parameters.

These include:

• commit and author signatures
• Target reference for reads and PRs (currently main)
• commit message templates
• PR title and body templates

This will be the first demonstration of how cup works.
We will use it to configure an instance of Flipt with the git backend type configured.

This runtime implementation will suport two kinds:

• Flag
• Segment

It should handle outputting the current Flipt (as of today v1.1 is in the pipeline) configuration format yaml.

If you run cup edit and open a resource in your editor, then immediately quit, cup still creates an empty PR

Currently, we clone source projects into memory.

This is ok, but wont scale well for larger projects.
We should support cloning to temporary or specified directories.
Some care will likely need to be taken for concurrent working directories al the git index.
Ideally, we can create a unique working index per request, but I need to do some more discovery into that (w.r.t go-git).

The sdk/go package for implementing controller runtimes will need a revamp for the new design.

I think this would be a useful isolated distributable to have as part of this project.
It should provide straightforward scaffolding for building a runtime implementation in Go.

We should use it to build a flipt.io implementation for Flags and Segments.

Build and publish a cup image to GHCR.
We currently publish one with Flipt controller built in, but we should do a vanilla Cup instance.

The purpose of this issue is to track the core Git implementation of the api.FilesystemStore interface.
This type handles obtaining a mountable implementation of the wazero filesystem abstraction.

While we have a fs.FS compatible implementation, we will start out with using their DirFS implementation which goes straight to the underlying host filesystem. Eventually, we may move to a virtual implementation when they make their internal interface public.

We will leverage go-git and explore supporting both in memory and on disk (/tmp) clones.
We need to support both reads and writes.

Reads should be in the form of a read-only snapshot of a given revision.
The write interface should be transactional and isolated.

Configuration

As with the local implementation, we should support a source type: git in configuration with relevant fields.

sources:
    flipt:
        type: git
        git:
            repository: https://github.com/flipt-io/flipt.git
    cup:
        type: git
        git:
            repository: https://github.com/flipt-io/cup.git

View

View requires the product of a simpler, read-only snapshot of the filesystem for a target revision.

The revision supplied could be a target SHA or a reference that needs resolving to the latest available SHA.

Once the provided function to View has returned, the implementation simply needs to propagate any error values forward.

Update

Update is more complicated than View. In particular it handles the update flow documented in the design document here: https://github.com/flipt-io/cup/blob/main/docs/DESIGN.md#flow.

This implementation is the heart of the contribution flow inside cup.
At a high-level for each Update it must at-least:

1. Resolve the requested source configuration
2. Checkout a new branch for the contribution
3. Create a blank worktree for the target revision
4. Build a controller.FSConfig to mount the new worktree
5. Invoke the provided function with the FSConfig
6. Add any changes to the worktree, commit and push them to the target upstream
7. Open a pull-request on the configured source target SCM

cupd will exist as the server binary for hosting and running controllers.

It will need some configuration for a number of concerns:

1. Identifying and configuring sources (local fs or git repositories)
2. Configuring server details (ports, connection details, etc)
3. Resource Definitions
4. Enabled source + resource definition combinations
5. Managing and configuring controller types (template, wazero etc).

Current idea: Grafana + Flipt combo with high throughput evaluator.

Demonstrate how evaluation results in the Grafana dashboard are effected as Flag state changes.
Drive change using cup.

The API Server will be the HTTP entrypoint to cup.

Initially, it will handle two core competancies:

1. List the available resource definitions per configured repository
2. Expose each available resource type as its own prefixed section of the API

Resource Definitions API

This section of the API is for discovery of which definitions have been registered with cup.
It should be possible to list all the groups, versions and kinds available per target repository.

All resource definitions across all sources:

/apis

All resources definitions for a particular source:

/apis/<repository>

Resource APIs

Each repo / resource definition combo will have its own prefix in the form:

/apis/<repository>/<group>/<version>/namespaces/<namespace>/<plural>

Within this prefix we expose the four initial operations per namespace:

Get a single instance:

GET /apis/<repository>/<group>/<version>/namespaces/<namespace>/<plural>/<name>

List multiple instances:

GET /apis/<repository>/<group>/<version>/namespaces/<namespace>/<plural>

Put a single instance:

PUT /apis/<repository>/<group>/<version>/namespaces/<namespace>/<plural>/<name>

Delete a single instance:

DELETE /apis/<repository>/<group>/<version>/namespaces/<namespace>/<plural>/<name>

Dependencies

The API server will depend on a number of abstractions on which to perform its job.

The controller will encapsulate away details on how a FilesystemStore chooses to share details on the undelying filesystem.
This will allow the implementations to switch between using fs.FS, a temporary real directory on disk and potentially the wazero filesystem abstraction in the future (when it becomes available).
The APIServer doesn't really care, it just need to make sure the right controller gets the appropraite configuration from the relevant store when requested.

package controller

type FSConfig struct {
    fs fs.FS
    dir *string
    // sysfs.FS for wazero
}

func WithFS(fs.FS) containers.Option[FSConfig] {}

func WithDir(string) containers.Option[FSConfig] {}

Filesystem Store

The purpose of this abstraction is to materialize the relevant filesystem for a particular resource controller to operate over.
There are two main methods for this abstraction. One for read operates and one transactional write operation method.

type FSFunc func(controller.FSConfig) error

type FilesystemStore interface {
    // View invokes the function provided with configuration for a controller
    // which leads to a read-only view when mounted as a filesystem
    View(_ context.Context, repository, revision string, fn FSFunc) error
    // Update invokes the function provided with configuration for a controller
    // which leads to a writeable filesystem being mounted and any changes
    // being packaged into either a pull request or an immediate local change (implementation dependent).
    Update(_ context.Context, repository, revision string, fn FSFunc) (*Result, error)
}

Resource Controller and Store

The ControllerStore abstraction encapsulates the details of sourcing a particular instance
of a Controller by group name.
The Controller abstracts away the details of any particular controller implementation itself from the APIServer.

type Controller interface {
    Get(_ context.Context, c *controller.GetRequest) (*cup.Resource, error)
    List(_ context.Context, c *controller.ListRequest) ([]*cup.Resource, error)
    Put(_ context.Context, c *controller.PutRequest) error
    Delete(_ context.Context, c *controller.DeleteRequest) error
}

type ControllerStore interface {
    Controller(_ context.Context, group string) (Controller, error)
}

Setup GH actions (+ possible Dagger) for

• linting
• unit tests
• integrations tests (if we have them)

Supports #2

This issue tracks implementing the Get function for the WASM/WASI Controller.

The role of this function is to source a single resource instance by namespace and name.

The controller needs to:

1. Build a wazero execution environment with the target FSConfig mounted at /
2. Invoke the wasm binary with the arguments ["get", kind, namespace, name]
3. Parse the response as a cup.Resource and return it

func (*Controller) Get(context.Context, *GetRequest) (*cup.Resource, error) { /**/ }

Success

The Controller can get individual resources

The package needs a bit of love (unit tests, comments etc.).

Once it is tidied up, we should tag a version of it for external use.
It would be nice to potentially move the Flipt controller into the Flipt codebase.
(doing so would be better served though if we also solved #33)

Controller will handle building and invoking implementations of controllers and it will live in pkg/controller.

This issue will acts as a parent issue for implementing each of the operations supported by the Controller.

• Get #3
• List #4
• Put #5
• Delete #6

The controller is reponsible for invoking the configured wasm binary accordingly for each of these operations.

Implementation Details

The initial lifecycle of the controller will likely involve:

1. Taking a resource definition path
2. Opening, parsing and validation the resource definition
3. Sourcing the associated controller binary

FS Abstractions

Each method on a controller will require access to a filesystem of sorts.
Currently, the plans for the shape of this interface is up in the air.
In the near term we can use gitfs for reads and an actual directory on disk for writes.
In the future we would like the use the (yet to be exported) Wazero abstractions.

So in the near-term, the controller package will abstract this details behind a struct with unexported fields.
Implementations of the FilesystemStore will decide what is appropriate based on either read (view) or write (update) level of requested access.

We will hide this details behind the following types and functions:

package controller

import "io/fs"

type FSConfig struct {
    fs fs.FS
    dir *string
}

func NewFSConfig(fs fs.FS) FSConfig { return FSConfig {fs: fs} }

func NewDirFSConfig(dir string) FSConfig { return FSConfig {dir: &dir} }

This will allow us to make adjustments over time without effecting the API server implementation.

Each Controller request structure will require a FSConfig as an argument.

Success Criteria

We have added support for each of the four core controller operations.

Wazero 1.4 will come with a new experimental FS interface.
This will allow for custom implementations to handle writeable filesystem operations.
I think we could leverage these capabilities to avoid having to always have write operations go the filesystem.
Perhaps doing everything in memory this way wont scale for certain operations. However, it may be nice to have the choice.

tetratelabs/wazero#1605

_FLI-570

We should probably mention how to contribute, help with Cup development, perhaps adding a CONTRIBUTING.md file? Here is the one from Flipt: https://github.com/flipt-io/flipt/blob/main/.github/contributing.md

Also perhaps adding a codeowners file to request reviews automatically?

The cup API should have some mechanism for tracking the state of resource change proposals.

When a Put or Delete action occurs, it results in a proposal to a target Source destination.
For the git type sources with a backing SCM, this manifests as a Pull or Merge Request.

We should tie the state of these proposals back into the resource API somehow.
For example, we could replicate the concept of the Status API in Kubernetes land.
Each resource could have a status on it. This status would contain entries for each currently proposed change.

Additionally, we should have an endpoint for all proposed changes. So that there is a central way to discover all open requests.
This is potentially useful for downstream tooling to present.

I suspect that for a first pass, we might be able to rely entirely on the SCM as the source of truth for this state. This would likely be enough to get to an MVP. Something which demonstrates the feature, so we can understand if it actually provides value.
This has the added benefit that it requires no extra dependencies to run cup. Meaning that, for example, just having a GitHub account and repository is enough to experiment with running it yourself.

Down the line, I suspect that this might come with its own scalability problems.
Each cup instance will have to do some tricks to stash relevant correlation state in corners of the SCM APIs.

Storing State

The kinds of questions we might want to ask with this data is:

• Does this resource have an open proposal?
• How many open proposals are there for a single resource?
• What is being proposed by each proposal?
  • Is just PUT vs DELETE enough?
• Is the proposal going through some CI process and what is the status of that?
• Has the proposal got requested changes on it?

Individual Resource State

In order to support e.g. a Status section for resources, we will need to be able to correlate open proposals (PRs, MRs etc) with a particular instance of a resource (kind, namespace and name). You could imagine e.g. an API like that of k8s status APIs:

GET /apis/{ group }/{ version }/namespaces/{ namespace }/{ kind.plural }/{ name }/status

The identifying metadata in this path would need to be correlated with some additional metadata in the proposal.

Some potential locations we can stash this kind of metadata is:

• PR title
• commit message (header and/or body)
• PR Description

The outcome of this decision also effects the next decision around listing all open proposals.

Listing all open proposal state

It will likely become valuable to list all open proposal statuses.
Depending on how we choose to stash state, the complexity of this operation changes.
If it is simply in the PR title, then we can rely solely the SCM list API.
However, if it is nested insider commit metadata or PR descriptions, then subsequent API requests are required to manifest this information.

The current thinking is to use cupd for the server itself.
Then reserve cup as the name for a local CLI for interacting with an instance of cupd.

The following are some ideas for what this local CLI could support.

cup config

Much like kubectl config we could have some local configuration management.
For example, setting a local context.

Much like how kubectl has a cluster + namespace in its context, we could have a server + source + namespace context.

cup ctl

cup ctl could be considered much like kubectl. It would handle introspection into the resources available and access to the resources themseves. A reflection of the API servers capabilities for the command line.

• cup ctl resources
• cup ctl get flags
• cup ctl list flipt.io/segments
• cat resource.yaml | cup ctl put

cup build

This is a long way off idea 💡

cup could support packaging Controller implementations.
It could take a Resource Definition file and the identified target WASM binary and package them into an OCI image.
This image could be loaded into some kind of local store to be consumed directly by cup.
It could also be pushed to some target upstream OCI registry for distribution.

The README will need a lot of love based on all the new proposed design for cup.

Here are some thoughts on what needs addressing:

• Overview (links to design etc.)
• Requirements
• Building
• Running
• Targets (cupd vs cup)
• License

We need to enforce the JSON schema is valid for resources on PUT.
This should happen in the *api.Server.

Open consideration is; how can we support more than just spec as the schema validated field?

This issues tracks the work required to build a local FilesystemStore for the pkg/api package.

This source is primarily useful for local evaluation / development experience.
This storage implementation works directly on the underlying target filesystem.
There are no proposals made during an update, just direct and immediate effect change.

The implementation should be super straight-forward and we can simply just pass the path to the location on the actual filesystem.

The store should be configurable such that, in theory, multiple directories could be named in configuration.

sources:
  flipt:
    type: local
    local:
      path: "/projects/flipt"
  cup:
    type: local
    local:
      path: "/projects/cup"

Ultimately, these manifest as API Server sources:

/apis/flipt/...

/apis/cup/...

Build a test harness around the core behaviours of cup.

Some initial back of napkin behaviours to assess:

1. Mounting and listing new resource types
2. Getting resources
3. Listing resources
4. Putting new resources and asserting PRs are created
5. Putting existing resources and asserting PRs are created
6. Deleting existing resoulrves and asserting PRs are created

Likely will leverage both Dagger and Gitea to make an end to end experience.

Supports #2

This issue tracks implementing the Put function for the WASM/WASI Controller.

The role of this function is to invoke the put <kind> over the wasm binary for the provided FS mounted at root.

The controller needs to:

1. Build a wazero execution environment with the target FS mounted at /
2. Validate the proposed resource matches the desired resource definition schema
3. Invoke the wasm binary with the arguments ["put", kind] and the desired resource marshalled on STDIN

func (*Controller) Put(context.Context, *PutRequest) error { /**/ }

Success

The Controller can adjust the state of the described target filesystem based on the new state of the request resource.

Add support for GitHub as a target SCM.

Supports #2

This issue tracks implementing the Delete function for the WASM/WASI Controller.

The role of this function is to support invoking the delete target on the underlying wasm binary for the request resource.

The controller needs to:

1. Build a wazero execution environment with the target FS mounted at /
2. Invoke the wasm binary with the arguments ["delete", kind, namespace, name]

func (*Controller) Delete(context.Context, *DeleteRequest) error { /**/ }

Success

The Controller can delete existing resources from the target FS represented by the FSConfig

Supports #2

This issue tracks implementing the List function for the WASM/WASI Controller.

The role of this function is to list multiple instances of a resources by namespace and optional label key/pairs.

The controller needs to:

1. Build a wazero execution environment with the target FSConfig mounted at /
2. Invoke the wasm binary with the arguments ["list", kind, namespace, ...(k/v pairs)]
3. Parse the response as a []*cup.Resource and return them

func (*Controller) List(context.Context, *ListRequest) ([]*cup.Resource, error) { /**/ }

Success

The Controler can list and filter multiple resources for a given namespace