cnabio / duffle Goto Github PK

Duffle currently needs a home directory configured with a few things like a credentials directory.

We should probably create a duffle init or something similar, and have that scaffold out the $HOME/.duffle directory.

TODO: Should we name this duffle init? Or is there a better name? Or do we do things from an external tool?

We need a way to generate credentialsets. I am guessing we want a suite of commands under duffle credentials. However, I don't have a strong set of opinions about what these tools should look like.

Probably, as a starting point, we should do duffle credentials list, which would just list the different credentialset names:

$ duffle credentials list
production
staging
development
personal

Right now, pointing which bundle to install is done by referencing the container image of the bundle:

$ duffle install <installation-name> <bundle-container-image> <other-flags>

This is very specific for the Docker driver, meaning that for other drivers, this might not have a correspondent.
This is why it would probably make more sense to pass, instead of the container image, the bundle itself.

Starting with #73 we have support for bundle repos - so in theory, we could reference a bundle very much like we do Helm charts: <repo/bundle> - having the install command as follows:

$ duffle install <installation-name> <repo/bundle> <other-flags>

In addition, I think it would make things easier if we also supported installing bundles that are not in a repo, by passing a bundle file - local, or remote - the analogy for this would be kubectl create -f <file>:

$ duffle install <installation-name> -f <bundle-file> <other-flags>

Do these two methods seem a good starting point to reference the bundle to install?

The idea would be to do something like this:

• Have the driver generate a cloud-init file and then run az vm create
• The Cloud-Init should...
  • Use runcmd to load env vars, then run /cnab/app/run
  • Use power_state to shutdown the VM
• az vm create should (at minimum) set the resource group, and set the VM image to whatever the bundle.json specifies

I think we have decided that manifest.json is a bad name, and bundle.json is a better name.

We'd like to support Packer as a build target. What is the equivalent workflow to Docker's docker build && docker push? What do we need to do to configure for Packer?

run will be replaced with install, uninstall, upgrade, delete, and status.

The bundle.json (formerly manifest.json) contains replacement rules that look like this:

"images": [​
    { ​
      "name": "frontend",​
      "uri": "gabrtv.azurecr.io/gabrtv/vote-frontend:a5ff67...",​
      "refs": [​
        {​
          "path": "./charts/azure-voting-app/values.yaml",​
          "field": "AzureVoteFront.deployment.image"​
        }​
      ]​
    }
]

We need a library implementation that reads the file at path, determines whether it is JSON, YAML, XML or (stretch goal) TOML, and then replaces the value at field with the given rewritten image name.

Per @fibonacci1729 's suggestion, we should be able to add duffle exec TARGET, where TARGET is a user-specified name. This would execute an action in the invocation image whose name is the TARGET.

$ duffle install my_thing example/foo:1.2.3
$ duffle exec foobar my_thing example/foo:1.2.3  # Executes action=foobar in claim my_thing 

My gut feel is that we would merely execute that action and then pipe the results to STDOUT, but not change the claim.

Other options:

• could generate a new revision (this might be a good idea) of the claim

I don't know what we do if the exec'ed option fails.

Add Go support for an Arm parameters.json file format (or similar).

Depends on #62

In this part, we need to write a library that takes a credentialset definition and resolves the source data and places them into a destination. SO this involves doing logic to read the credentialset.CRED.source and reading files, reading env vars, reading static values, and executing commands, then taking the results and putting them into a destination datastructure.

The initial input of a duffle build is a duffle.toml file; its output is a manifest.json file, which can then be used as input for duffle install, together with a file containing values for parameters.

This issue is to track the initial generation of the manifest from duffle.toml.

This function is a crutch on Windows and should be removed in Go 1.11 with os.Symlink gaining the ability to symlink with user-level permissions.

https://github.com/deis/duffle/blob/4da2e653dec511f9a8898b1f3e73e5d4645f2d7c/pkg/osutil/osutil.go#L22-L38

The duffle uninstall command should run the uninstall action on a Claim.

$ duffle uninstall foo

The above should find the claim named foo and run the CNAB for that bundle, with the action set to uninstall

Most of the logic is present in the duffle install command.

It is an open question whether we need to allow parameter values to be passed to Duffle for an uninstall.

Depends on #47

We need a pkg/credentialset that can read credentialset definitions.

The credentialset spec is here: https://github.com/deis/duffle/blob/master/docs/201-credentialset.md

In this first step, we really just need the unmashalling and the support for $HOME/.duffle/credentials.

I'll create a separate issue for resolving credentialsets into a list of credentials.

We need a nice pretty CRUD interface for storing claims. For now, the default implementation can be local (on disk) storage. But we want to make it work for object storage, a database, or some other backend.

Create a Cobra-based CLI with the base command and help text

Based on the current implementation, the builder writes the build logs under ~/.duffle/logs.

We should implement duffle logs, which reads the log files - the implementation should be fairly similar to Kubed/Draft.

We need a proposal for how this would work.

Depends on #48

The duffle list command should print one line for every Claim in the claim storage. It should provide a tabular list of info about the installed CNAB apps that the claim storage knows about.

The current suggestion is that we use Arm's parameters.json format. Where is that documented?

The duffle upgrade command takes a claim name and a new bundle, and it replaces the claim's old bundle with the new bundle, executing the upgrade action on the new bundle.

This one may be rather complex, so don't feel like we need to solve all of the problems.

$ duffle upgrade my_app example/my_bundle:1.2.3
# my_app is created
$ duffle upgrade my_app example/my_bundle:2.3.4

Most of the base logic can be found in duffle install in #47

The duffle status command loads an image from an existing Claim and runs the status action on that container.

$ duffle install my_app example/new_bundle:1.2.3
$ duffle status my_app    # runs the `status` action in example/new_bundle:1.2.3

Note that the intention is that the CNAB bundle itself is what generates the status data. So all we are doing is piping the output of that call to STDOUT.

Most of the base logic for this exists in duffle install #47

We need a way to build the containers required in a CNAB bundle. There's a WIP over at #44 which implements building the containers individually by pulling most of the code from previous work (kubed).

There's still a few things we need to do though:

• change it to build only the Dockerfiles present in duffle.toml rather than searching for every Dockerfile from the root directory
• come up with a "manifest" format that is constructed during either duffle build or duffle push

We need the manifest so that we have the "bill of materials" for running a CNAB bundle via duffle pull/duffle run. @radu-matei has done some research/experimentation on how Docker creates and manages these manifests, and there's one proposal in #23 (comment) that mimics the Docker multi-arch manifest artifact. Right now the question stands: can we upload a custom manifest via duffle pull, or will the registry refute an invalid schema? We're running under the assumption that it doesn't. As long as we implement the schemaVersion and mediaType present in our custom manifest, we hope to be able to upload a custom manifest to a registry...

We have Hope. Rebellions are built on Hope. ™️

We should determine the basic underlying architecture/layout of a CNAB manifest and document it. The architecture should be able to handle all the existing example use cases:

#11
#12
#13
#14
#15
#16
#21

relatedly, it should also present a proposal for how a CNAB manifest is addressed in storage via #5 and #6.

The duffle downgrade command rolls back to a previous version of a release by executing the downgrade action on the older bundle

This one may be overly complex to do well, and may be better handled by having users just run upgrade on older images. So don't do this one w/o giving the team a chance go offer feedback.

Duffle exposes a driver layer where custom drivers can be used to execute the bundle's invocation image. This can be done via shell script, Go program, python, node.js... whatever....

What would be nice is to be able to demo this with Nomad.

Depends on #47

Figure out what it would look like to use GitHub for storing CNAB bundle manifests.