ksonnet / ksonnet

A CLI-supported framework that streamlines writing and deployment of Kubernetes configurations to multiple clusters.

ksonnet

The team behind ksonnet is stepping back from the project. As a result, work on ksonnet will end and the GitHub repositories will be archived. You can read more about our decision here.

Official Site

ksonnet is a framework for writing, sharing, and deploying Kubernetes application manifests. With its CLI, you can generate a complete application from scratch in only a few commands, or manage a complex system at scale.

Specifically, ksonnet allows you to:

• Reuse common manifest patterns (within your app or from external libraries)
• Customize manifests directly with powerful object concatenation syntax
• Deploy app manifests to multiple environments
• Diff across environments to compare two running versions of your app
• Track the entire state of your app configuration in version controllable files

All of this results in a more iterative process for developing manifests, one that can be supplemented by continuous integration (CI).

Install

The ksonnet CLI, ks, can be installed in three different ways. Choose the method that best matches your setup:

Homebrew on macOS

If you are using Homebrew on macOS, you can install ks with the following command:

brew install ksonnet/tap/ks

Download a prebuilt binary for your OS

See the releases page to download the latest released binary.

Build a docker image

A docker image can be built and used similarly to our manual build as seen here

Manually build and install

You can download and manually build from source by following these instructions.

Run through an example

Here we provide some commands that show some basic ksonnet features in action. You can run these commands to deploy and update a basic web app UI, via a Kubernetes Service and Deployment. This app is shown below:

Note that we will not be implementing the entire app in this example, so the buttons will not work!

Minimal explanation is provided here, and only basic ksonnet features are shown---this is intended to be a quick demonstration. If you are interested in learning more, see Additional Documentation.

Prerequisites

• You should have access to an up-and-running Kubernetes cluster — supported versions are 1.7 (default) and 1.8 (beta).

  If you do not have a cluster, choose a setup solution from the official Kubernetes docs.

• You should have kubectl installed. If not, follow the instructions for installing via Homebrew (MacOS) or building the binary (Linux).

• Your $KUBECONFIG should specify a valid kubeconfig file, which points at the cluster you want to use for this demonstration.

Commands

Start by creating your app directory. If you are running Kubernetes 1.8.x, you'll need to add --api-spec=version:v1.8.0 to the end of the following command:

# The ks-example app directory is created at the current path, and the
# app itself references your current cluster using $KUBECONFIG
ks init ks-example

You can copy and paste the commands below to deploy the web app UI:

# 'ks' commands should be run within a ksonnet app directory
cd ks-example

# Autogenerate a basic manifest
ks generate deployed-service guestbook-ui \
  --image gcr.io/heptio-images/ks-guestbook-demo:0.1 \
  --type ClusterIP

# Deploy your manifest to your cluster
ks apply default

Now there should be a Deployment and Service running on your cluster! Try accessing the guestbook-ui service in your browser. (How you do this may depend on your cluster setup).

If you are unsure what to do, we suggest using kubectl proxy.

# Set up an API proxy so that you can access the 'guestbook-ui' service locally
kubectl proxy > /dev/null &
PROXY_PID=$!
QUICKSTART_NAMESPACE=$(kubectl get svc guestbook-ui -o jsonpath="{.metadata.namespace}")
GUESTBOOK_SERVICE_URL=http://localhost:8001/api/v1/proxy/namespaces/$QUICKSTART_NAMESPACE/services/guestbook-ui
open $GUESTBOOK_SERVICE_URL

(Remember, the buttons won't work in this example.)

Now let's try upgrading the container image to a new version:

# Bump the container image to a different version
ks param set guestbook-ui image gcr.io/heptio-images/ks-guestbook-demo:0.2

# View updated param values
ks param list

# Update your cluster with your latest changes
ks apply default

Check out the webpage again in your browser (force-refresh to update the javascript). Notice that it looks different! Clean up:

# Teardown
ks delete default

# There should be no guestbook service left running
kubectl get svc guestbook-ui

(If you ended up copying and pasting the kubectl proxy code above, make sure to clean up that process with kill -9 $PROXY_PID).

Now, even though you've made modifications to the Guestbook app and removed it from your cluster, ksonnet still tracks all your manifests locally:

# View all expanded manifests (YAML)
ks show default

If you're still wondering how ksonnet differs from existing tools, the tutorial shows you how to use other ksonnet features to implement the rest of the Guestbook app (and yes, the buttons will work!).

Additional documentation

ksonnet is a feature-rich framework. To learn more about how to integrate it into your workflow, check out the resources below:

• Tutorial - What can I build with ksonnet and why? This finishes the Guestbook app from the example above.

• Interactive tour of ksonnet - How do ks commands work under the hood?

• CLI Reference - What ksonnet commands are available, and how do I use them?

• Concept Reference - What do all these special ksonnet terms mean (e.g. prototypes) ?

• Design Docs - What are the detailed design specs, and what's next on the feature roadmap?

Troubleshooting

If you encounter any problems that the documentation does not address, file an issue.

Contributing

Thanks for taking the time to join our community and start contributing!

Before you start

• Please familiarize yourself with the Code of Conduct before contributing.
• Read the contribution guidelines in CONTRIBUTING.md.
• There is a mailing list or the #ksonnet channel on Slack if you want to interact with other members of the community.

Pull requests

• We welcome pull requests. Feel free to dig through the issues and jump in.

Changelog

See the list of releases to find out about feature changes.

ksonnet's Issues

Pretty print output of `ks dep list`

I know Alex has a plan for this, just don't want it to slip through.

It should:

• indicate which libraries have been downloaded to local
• have something that can be easily copy/pasted into ks dep get

e.g. something along the lines of:

Registry      library@latest release 
---------- -----------------------
*incubator     [email protected]     
incubator   [email protected]    
incubator     [email protected]     
*incubator     [email protected] 

Add command `ks component rm <component-name>`

With the increasing amount of metadata surrounding components, ex: params, we need a better way to remove components than rm components/foo.jsonnet. This new command should handle cleaning up of metadata related to the component.

Improve overall handling of complex jsonnet param schemas

• Support top of file imports
• Edit: Decided not to do for milestone Consider more RHS support. (We might decide, e.g., to allow simple +, or not.)
• Edit: Decided not to do for milestone Support more than primitive param types.
• Support multi-line string literals.

Consider returning the URI on `apply` for services

From @jessicayuen on October 4, 2017 21:3

It would make for a better demo story if we were able to do the following:

$ ks generate deployment-exposed-with-service frontend \
> --name frontend \
> --image gcr.io/google-samples/gb-frontend:v4
$ ks apply default
INFO: <FRONTEND_URL>

Consider whether it's better to return the URI by default, or enable it through a flag.

Copied from original issue: vmware-archive/kubecfg#168

Make sure dep management supports YAML metadata files as well as JSON

Marshaling in https://github.com/hausdorff/ksonnet/blob/395491ab0aeb6d853e235aa1a4025ee7a14b8c76/metadata/app/schema.go should support YAML.

Move core logic of new commands (`dep` and `registry`) to pkg kubecfg

We took a shortcut to hit the review deadline, but we need to come back and fix this.

initialize ksonnet migration

• PR #2 rename commands to ks
• PR #2 rename packages to ksonnet
• migrate ksonnet.next issues
• migrate ksonnet.next PRs
• setup CI

Hook up Jsonnet prototype changes to `use`

From @hausdorff on September 11, 2017 17:59

@jessicayuen did the core work last week, so we just need to:

• Allow the use API to use this API.
• Add a couple system prototypes that exercise this code path.
• Add a couple tests to the prototype tests that exercise this code path.

This is slotted in the feature complete milestone, since we're not reading from files at this point.

Copied from original issue: vmware-archive/kubecfg#125

Support `diff` between two environments

From @jessicayuen on September 27, 2017 20:11

From spec:

ksonnet diff <env-one> [env-two]

Diff should model git’s diff between two commits / branches. In the case of environments, diff will output the changes between the components deployed at the API server located at <env-one> and the API server located at <env-two>. <env-#> is the name of an environment. Environments must be created using ksonnet env add before use.

Examples:
Output the differences in components deployed at remote and local “us-west/dev”.
ksonnet diff us-west/dev

Output the differences in components deployed at remote “us-west/dev” and remote “us-west/prod” environments.
ksonnet diff remote:us-west/dev remote:us-west/prod

Output the differences in components deployed at local “us-west/dev” and remote “us-west/prod” environments.
ksonnet diff local:us-west/dev remote:us-west/prod

Copied from original issue: vmware-archive/kubecfg#163

Support hyphenated component names

The Jsonnet language does not support object identifiers with hyphens.

Ex: If a component is called guestbook-ui, we internally generate "component imports" in the form:

guestbook-ui: import "/Users/jyuen/go/src/github.com/ksonnet/ksonnet/guestbook/components/guestbook-ui.jsonnet",

Because the hyphen causes the Jsonnet to not compile, we will get the following error on apply:

➜  guestbook git:(ks-generate) ../ks apply default
ERROR Error reading /Users/jyuen/go/src/github.com/ksonnet/ksonnet/guestbook/environments/default/default.jsonnet: <extvar:__ksonnet/components>:2:12-13 Expected one of :, ::, :::, +:, +::, +:::, got: -

  guestbook-ui: import "/Users/jyuen/go/src/github.com/ksonnet/ksonnet/guestbook/components/guestbook-ui.jsonnet",

Add dependencies to -J path

@hausdorff @jessicayuen

• Vendor path
  ks apply and ks default don't work for components generated from the incubator prototypes--this is because vendor/ is not included in the default jsonnet lib paths. I'm not sure if root.go needs to be modified to include it or somewhere else; either way, I don't think it'll be a difficult fix.
  • Adding vendor/ to the -J paths in ksonnet/ksonnet.
  • Flatten out the vendor/ directory.
  • Changing all prototypes in ksonnet/parts from import "incubator/whatever/whatever.libsonnet" to use import "whatever/whatever.libsonnet" instead.

Environment URI normalization

From @jessicayuen on September 19, 2017 19:36

Go doesn't seem to have a standard way / default library to normalize URIs. As part of the environments feature, we want to be able to deploy to a specific cluster given the environment's URI. We cross-check against kubecfg for this URI's location. For fail-safe comparison, we probably want to normalize these URIs.

Copied from original issue: vmware-archive/kubecfg#144

Layer of abstraction for commands that take an 'env' arg.

From @jessicayuen on September 25, 2017 23:13

There is a large subset of functionalities existing in cmd/root.go that are used only by commands that take an env argument, ex: ksonnet apply dev, ksonnet diff dev.

Each have a similiar workflow of having to modify the --cluster flag to point to the server the environment is located at, and what not.

All this common functionality is currently located in cmd/root.go and too tightly coupled with commands that have no notion of environments. There needs to be a layer of abstraction added for environment-using commands.

Copied from original issue: vmware-archive/kubecfg#156

Implement package management

Roughly, this involves:

• Implement the app.yaml schema, where we store registry and dependency information for an app
• Implement ks registry
  • Implement registry.yaml schema for general registries to expose what libraries they keep track of
  • Implement the registry protocol for github.
  • Implement ks registry describe
  • Implement ks registry add
  • Implement ks registry remove [won't ship for feature complete]
  • Implement ks registry list
• Implement ks pkg
  • Implement the parts.yaml schema, where we store registry and dependency information for a library
  • Implement dependency cache disk data structure
  • Implement ks pkg describe
  • Implement ks pkg search
  • Implement ks pkg install
  • Implement ks pkg list
  • Implement ks pkg refresh

A single `SpecificationSchema` should be able to have multiple "types" of template

From @hausdorff on September 7, 2017 23:25

In other words, it doesn't make much sense to have multiple templates that are essentially the same, but written in Jsonnet, YAML, and JSON. A better idea is to have a single prototype be able to (possibly) take multiple forms, and maybe default to Jsonnet.

Copied from original issue: vmware-archive/kubecfg#117

Resolve a library prototype's `k.libsonnet` based on the release specified in parts.yaml

NOT necessary for feature complete. Just wanted to put this here so we don't forget if/when it becomes an issue.

Intergrate with or borrow from kubecrt

There's a significant ongoing effort to develop robust Helm Charts for composable infrastructure. It would be nice to be able to integrate with Helm Charts in a way kubecrt already does it - by transpiling a charts config into a manifest, which then can be used in ksonnet.

Right now, I'm doing this via a build process, but maybe in addition to supporting YAML, JSON, and Jsonnet, a kubecrt charts' config would be a worthy add-on.

--help flag gets dropped in nested commands (ks prototype preview)

Tried to run

ks prototype preview --help

But got

ERROR No prototype names matched '--help'

Not super urgent but needs to be fixed.

update README

README doesn't showcase any of the ksonnet.next features added, specifically environments and prototypes.

We need to update this prior to release.

Consolidate type checking between params and prototypes

Currently there exists two areas of parameter type checking in cmd/prototype.go and metadata/params/params.go. Since these are operating on the same parameters, we should consolidate this discrepancy.

Generate the default environment from the kubeconfig file

From @jessicayuen on September 18, 2017 18:57

• When we run init, output what we're generating default from. Right now, I believe it's nothing unless you explicitly specify the URI. (e.g., we might output something like Generating 'default' environment from current context in kubeconfig file at '~/whatever'.)
• I think if we want to get the URI from the current context, it's probably fine to just print out where we got it from.
• If there is no kubeconfig file, don't generate a default environment, and explain why. (e.g., Could not generate 'default' environment, kubeconfig file not found)

Probably also need to consider in this case whether we want users to use ksonnet apply default, or implement a special case ksonnet apply that will just look for the default environment if is not specified.

Copied from original issue: vmware-archive/kubecfg#134

Formalize unique cluster ID on env creation

From @jessicayuen on September 18, 2017 18:59

We should get formal about how the env commands produce a unique ID for a cluster. I'm thinking a hash of URI + server cert? This is a temporary solution of course, we'll need to fix it up stream.

Copied from original issue: vmware-archive/kubecfg#135

Implement "complex" environments

From @jessicayuen on September 28, 2017 18:12

"complex" environments, are based off this design.

• • PR #164 Add intrinsic for ksonnet.importComponents()
• • PR #167 Update directory structure

    environments/
      base.libsonnet
      us-east/
         dev/
           dev.jsonnet
           spec.json
           .metadata/
              k8s.libsonnet
              k.libsonnet
    

• • PR #167 Generate base.libsonnet

    local components = ksonnet.importComponents();
    components + {
      // Insert user-specified overrides here.
    }
    

• • PR #167 Generate .libsonnet file on env add

  local base = import "base.libsonnet";
  local k = import "k.libsonnet";
  local dep = k.apps.v1beta1.deployment;
  
  base + {
     // Insert user-specified overrides here
  }
  

• • PR #167 Commands that take env need to apply <env>.libsonnet file

Copied from original issue: vmware-archive/kubecfg#165

Consider adding client-go flags back to `diff`

client-go exposes a bunch of interesting and useful flags for talking to the server (e.g., --insecure-skip-tls-verify).

These were removed from diff because it's not clear how the flags should act when given two remote clusters.

There seem to be a couple of options for dealing with this.

1. Make diff more like git diff. git diff only works on local copies of a remote, so something like git diff origin/foo..bar/baz works on local copies. To obtain those copies, users might use something like git fetch. This actually fits with what we want to do for environments anyway, since we want to be able to tell users when something deleted locally is still present on the cluster.
2. Somehow add them back (this has a high potential of being confusing, I think)

Autogenerate .gitignore file (e.g. for libs or hidden files)

Upon ks init.

Consolidate `ks diff` and `ks param diff`

Instead of

ks param diff dev prod

change the syntax to

ks diff param:dev param:prod

(to be more in line with existing ks diff syntax).

Renaming env needs to also rename the nested jsonnet file

In the demo, doing

ks env set default --name=dev

renames environments/default to environments/dev but fails to rename environments/default/default.jsonnet to environments/dev/dev.jsonnet

`ks prototype use` giving wrong usage message

Note the second line says preview:

➜  ksonnet git:(master) ./ks prototype use
ERROR Command 'prototype preview' requires a prototype name

Usage:
  ks prototype use <prototype-name> <componentName> [type] [parameter-flags] [flags]

...

Handle environment namespace defaults

From @jessicayuen on October 6, 2017 21:2

Sorry, I forget what we decided here. When a user runs something like env set foo --namespace="", we have the following options:

Unset the namespace.
Do nothing.
Set namespace to "default".
I'm fine pushing this to an issue and punting for now; I just want to make sure we make this decision purposefully. So why don't we do that?

That said, my inclination is to set it to "default". We also might also consider having an env unset command.

Also, consider removing len(namespace) != 0 when overriding namespace, depending on the outcome of the decision above.

Copied from original issue: vmware-archive/kubecfg#172

Update `prototype use` to include examples redirecting to `components/`

From @hausdorff on September 5, 2017 20:57

This makes it clearer how to use it.

Copied from original issue: vmware-archive/kubecfg#110

Implement environment params

• PR #40 Generate components/params.libsonnet file on ks init
• PR #40 Generate environments/<env>/params.libsonnet file on env add ...
• PR #40 Expand the environment params.libsonnet file and make accessible as extVar.
• PR #40 Components generated from ks prototype use ... need to populate components/params.libsonnet
• PR #40 Components generated from ks prototype use ... need to reference the params from the component file.
• Operations needed (around manipulating jsonnet) to support the below commands:
  • PR #40 Write component parameters
  • PR #40 Read component parameters
  • PR #40 Write environment parameters
  • PR #40 Read environment parameters
• Commands:
  • PR #40 ks param diff <env> <env> [--component]
  • PR #40 ks param set <component> <param> <value> [--env]
  • PR #40 ks param list <component> [--env]

Disallowed `ioutil.*` functions used

Currently we use afero to mock all FS calls. We're calling some functions we shouldn't, like ioutil.ReadFile in various places. We should replace these with afero calls.

Update documentation

Migrated from here

This involves:

• Updating the ksonnet README.
• Adding ks to the ksonnet site (possibly move it to ksonnet.io).
• Adding a "case studies" section. Using the tutorial for this, for now.

Add command `ks lint --params`

Allow users to lint on their environment param files, outputting errors where there is a param for a component that doesn't exist, etc.

• Lint on params not associated to real components
• Other invalid configurations, ex: If a component is deleted, and there is still references to it, it can be reported to the user
• Requires a design doc

`ks apply <env>` isn't being deployed to the correct context

It works if the --context flag is supplied, but otherwise does not deploy to the context / namespace specified by the environment.

Investigate --namespace flag overrides

From @jessicayuen on September 29, 2017 23:21

As we discussed offline, there's a few things that need to be investigated:

1. Whether the client-go flags are necessary in the env commands?
2. What are the side effects of overriding client-go's --namespace flag with a custom flag of the same name? Does it behave as expected?

Related: #161

Copied from original issue: vmware-archive/kubecfg#166

Check UX for namespace scenarios

example: What happens when an environment has namespace test but the test namespace doesn't exist on the cluster?

(and other such flows)

Make sure that error messaging is appropriate.

Add links to http://ksonnet.heptio.com/

From @jhermann on July 3, 2017 9:6

Link your org's and project's "Website" settings to http://ksonnet.heptio.com/ to make things more discoverable.

Copied from original issue: vmware-archive/kubecfg#49

Generate ksonnet-lib from multiple Kubernetes versions.

Migrated from here.

Currently, we support only generating ksonnet-lib from Kubernetes build v1.7.0. We should expand support for generating ksonnet-lib for more than one Kubernetes build.

[polish] Make sure CLI commands have the right verbs

There are some potential CLI changes we need to make, based on the demo. Feel free to contest any one of these:

• Rename (or at least alias) ks prototype use to ks generate
• Figure out how we want to deal with the other ks prototype * commands. Do we keep prototype in the command, and if so, is that inconsistent with ks generate?

There might be more questions as we finish up ksonnet params and environments, but want to make sure this is on our radar for the demo next week.

Do not use kubeconfig current namespace when using environments

From @hausdorff on September 21, 2017 5:3

Currently if you run a command like kubecfg apply default, it will automatically use the namespace of the current-context. This is obviously nonsensical, since the current context might have nothing to do with the cluster we're deploying to (i.e., it might refer to a namespace on a totally different cluster.

Copied from original issue: vmware-archive/kubecfg#148

Init env metadata from provided context

Automatically init namespace and URI from some context. ks env add us-west/staging --context your-context

Better error messaging when env is not supplied for `kubecfg show -f <FILE>`

From @abiogenesis-now on September 11, 2017 21:15

If an env isn't supplied with the kubecfg show -f <FILE> command, kubecfg can't load the appropriate ksonnet lib files from /environments with the -J flag.

We need better error messaging so the user knows what to do in this scenario.

Copied from original issue: vmware-archive/kubecfg#127

Building via Make

What is the proper way to build via Make? At this point, I get an error due to kubecfg missing, but, then, which version of kubecfg should I fetch?

Everything installs properly via go get, but that does not allow me to install a specific version of ksonnet, which is my requirement.

I apologize, but I'm not experienced much with the Go toolchain.

Better pretty-printing for `dep`, `registry`, and `prototype` commands

`prototype use -h` should output help docs

prototype use -h should output help docs when a single argument --help or -h is provided.

Currently receives the error:

➜  foobar git:(master) ✗ ../ks prototype use -h
ERROR No prototype names matched '-h'

Expect the same behavior outside of a ksonnet project dir, however currently receives:

➜  ksonnet git:(master) ✗ ./ks prototype use -h 
ERROR 'prototype use' can only be run in a ksonnet application directory:

No ksonnet application found

`ks prototype preview` doesn't seem to work for JSON and YAML

(I'm wondering if this is because of the extVar?)

ks prototype preview works fine when the type is specified as jsonnet. However, it will error out if either json or yaml is used.

e.g. for the following command (YAML)

$ ks prototype preview deployment-exposed-with-service yaml \
> --name guestbookui \
> --image alpinejay/dns-single-redis-guestbook:1.1 \
> --type LoadBalancer

the output is:

ERROR preview:1:11-12 Did not expect: ":"

For JSON, the output is:

ERROR preview:8:23-24 Expected token OPERATOR but got ("}", "}")

Add `ks registry describe` command

Intended usage:

$ ks registry describe incubator
https://www.github.com/ksonnet/parts/incubator

Change 'uri' references to 'server'

To remain consistent with client-go / kubectl, update references of uri to server. This includes the command line flags as well as environment's spec.json

Consider un-setting "SilenceUsage" at the root command

Having this flag set at the root command means that child commands aren't able to un-set this flag. This is unhelpful when we actually want a command to display usage on usage-related errors, without explicitly calling cmd.Usage() each time.