replicatedhq / kotsadm

Kotsadm has been merged into the KOTS repo

THIS REPO HAS BEEN MIGRATED TO https://github.com/replicatedhq/kots

Kotsadm

Kotsadm is an installable admin console for managing Kubernetes Off-The-Shelf (KOTS) software. Kotsadm provides a nextgen admin experience for any KOTS application, designed to meet the needs of a wide spectrum of enterprise IT admins, from a “click-to-deploy” model to “automated operations”.

Once deployed, Kotsadm gives administrators the ability to get an application configured, installed and updated using step-through configuration, and automated preflight checks.

For advanced cluster operators, we recommend setting up the GitOps & internal registry integrations to move away from click-to-deploy and instead leverage an Kotsadm's automated operations.

Distributing a KOTS application

Software vendors can package their Kubernetes applications or Helm charts as a KOTS application in order to distribute the application to cluster operators. Kotsadm can serve as the whitelabeled (or multi-app), in-cluster admin console for any KOTS application.

Kots

Kots CLI is a kubectl plugin to help manage Kubernetes Off-The-Shelf software. See the documentation for more information on Kots CLI.

Installing

For more information on installing Kotsadm locally see https://kots.io/kotsadm/installing/installing-a-kots-app/.

Contributing

For docs on setting up a dev environment to build and run kotsadm locally, read the CONTRIBUTING guide.

Community

For questions about using KOTS & Kotsadm, there's a Replicated Community forum, and a #kots channel in Kubernetes Slack.

kotsadm's Issues

Memory pressure and CPU pressure messages should be switched

Showing the memory pressure under the cpu count and the cpu pressure under the memory count feels backwards.

The gitops and dashboard tabs should not be visible when there are no licenses installed

While installing the first license on a new install of kotsadm, the gitops and dashboard tabs are visible. We should hide these, until there is an application installed. If we are installing the 2nd, or later application, the top nav buttons should be visible.

The cluster management tab should always be visible, when it's a kurl-enabled cluster.

We should include all `kotsKinds` for the active version in a support bundle

There are currently too many client-side Graphql queries made on load

Currently, when loading the application dashboard, the front end make a lot of different graphql queries. Taking a look through these, it looks to be pretty inefficient and ad-hoc.

For example, switching tabs calls getApp and getLicenseType and getMetadata. Why do we need three calls? Shouldn't GraphQL be letting the client construct queries that make this a easier?

We currently hard code the queries in the client so that the react components don't construct the query they need, which results in a lot of duplication of queries and inefficiencies. In short, this is making the GraphQL implementation in this project function more like REST, but without status codes.

I think we need to document a strategy here to get this under control before we run into concurrent socket limits and/or put too much load on the kotsadm-api requesting duplicate data.

Logo change in the Admin Console is applied when the update is available, not installed

When a logo or title change is included in an application yaml, the change should not be applied until the version of the application is installed/deployed

kotsadm operator should remove objects that have been removed from the application between versions

Prevent kots.tar.gz from adding to the kotsadm api image size

Currently we download and leave the kots.tar.gz file in the tmp directory of the api. This is 15 MB and doesn't help make our image size as small as possible!

Feature: Add the ability to transition from click-to-deploy to GitOps

Having proper, first-class support for gitops is required for operationalizing an application. But we don't want to make it a requirement to get started. The workflow we are targeting here is "click to deploy" a trial or POC, and then have a path to operationalizing the installation without starting over.

Design and implementation doc for discussion: https://docs.google.com/document/d/1DU8fMN1UZhc5fvE_fJRmUPmTzfDeP6m7PJqg5tNbI7I

Rollback button does not rollback

Hello -- The Rollback button in KotsAdm does not seem to perform any action and doesn't actually rollback to the previous release.

Steps to repro:

• Make a deployment with replicas: 1.
• Update the deployment with replicas: 5.
• After update click the Rollback button, still replicas: 5.

Expectation:

• Rollback to replicas: 1

Move gitops config to K8s configmap and secrets

Currently, kotsadm stores gitops data in it's own postgres database. The private key is encrypted using the admin console's global key, and the rest of the data is stored in a few postgres tables.

As part of the larger effort to mature the gitops implementation, we should move the data out of postgres and store it as native kubernetes objects.

Why? This helps with moving away from pg in the product (long term plans), but it also allows an easier deployment strategy for anyone who wants to deploy configuration as code. Instead of learning our private GraphQL API, kotsadm installation users will be able to create a config map or secret and deploy it to the cluster.

Global Config

We currently support only 1 provider (in the 1.5.0 designs). But this isn't fixed forever, it's a v1.5.0 limitation only.

We should create a secret, in the kotsadm namespace, names kotsadm-gitops. It should have the following structure (the values should be properly base64 encoded)

apiVersion: v1
kind: Secret
metadata: 
  name: kotsadm-gitops
data:
  provider.0.type: "github enteprise"
  provider.0.hostname: "github.somebigbank.com"
  provider.0.publickey: "BEGIN..."
  provider.0.privatekey: "---"

If the secret is not present or does not contain any keys, then gitops is unconfigured globally in the admin console.
Currently it can only have provider.0.* fields, when configured.

Repo Config

Each repo can build off of this config by enabling gitops, and specifying a repo, branch, path, and format (-k or -f). This data is not sensitive, and can be stored in a config map, as YAML formatted data. All repo config will be in the same config map.

apiVersion: v1
kind: ConfigMap
metadata:
  name: kotsadm-gitops
data:
  app-slug-1:
    repo: somebigbank/gitops-deploy
    branch: master
    path: /app-1
    format: "single"

The kots installer will have to be modified to alter the RBAC policy so that the kotsadm-api process can create, edit, delete, and read this configmap and this secret. https://github.com/replicatedhq/kots/blob/e61e655c1f182ef5d4b25aca36176d99e3a9a91c/pkg/kotsadm/api_objects.go#L15

Additionally, the kurl add-on will need this rbac change also: https://github.com/replicatedhq/kurl/blob/be9b576bded39cb7337f6d6be454dac84c79da00/addons/kotsadm/1.3.0/api.yaml#L5 (only in the 1.5.0 version of the addon)

Proposal: Move this repo into the kots repo

As the KOTS project starts to mature and get some use, it's somewhat difficult to build and test replicatedhq/kots and replicatedhq/kotsadm independently. They are loosely coupled right now, we don't make any guarantees or publish any documents about which version of the kots cli supports which versions of the kotsadm project.

Building, testing and releasing this project would be easier if these repos were combined.

This isn't a technical design at this point, I'm hoping to get some feedback on the idea, and if it makes sense, start thinking through how we could perform this migration without breaking current work, prs, etc.

GitOps: Upstream updates don't respect the path when creating a commit

React app crashes when viewing certain file in support bundle

When I visit kots/admin_console/kotsadm-api-<hash>.txt in the admin console and attempt to scroll sideways (two finger scroll on mac), the react app crashes with this error: https://reactjs.org/docs/error-decoder.html/?invariant=185

When deploying an application to a cluster that already has kotsadm, the cluster rolebinding is not updated

The first application works great, but the second one does not add the newly created service account to the existting cluster role. This means that the preflight and support bundle cannot succeed in the new (second) namespace.

Steps to reproduce:

1. kots install
2. confirm that preflights work
3. kots install (to namespace 2)
4. see that preflights don't work
5. exec into the kotsadm-operator pod running in namespace 2 and run
6. kubectl preflight https://troubleshoot.replicated.com
7. uncompress the support-bundle file
8. cat cluster-resources/nodes-errors.json to see the error

After editing config, the version label changes to "Unknown"

Step to reproduce:

Install a replicated app that has a config spec.
Full out data on the config screen. There's a version label created.
Edit the config, click save.
There's a new version created, but the label is now "Unknown".

multiapp the HeaderLogo doesn't change on focus to new app

kotsadm/web/src/components/shared/NavBar.jsx

Line 141 in 7da30cd

<div className="HeaderLogo">

headerlogo seems to stay with the initial app that was installed (though the community license flag only shows up on the correct app)

If a gitops path is specified without the leading /, kotsadm strips whatever the first character is off the path

There is no way to determine the namespace an app is being deployed to

We should add a repl{{ Namespace }} template function or similar in order to allow apps to determine what namespace they are being deployed to. This is used as part of fully qualified service domain names (<service>.<namespace>.svc.cluster.local), among other things.

Resources are not removed with a namespace specified in the yaml when upgrading

Steps to reproduce:

1. create a release, specify a namespace other than the target namespace in the yaml for a deployment
2. install that release
3. create a new release and remove that deployment
4. install the new release, the deployment is not removed

additionalNamespaces does not resolve the issue

diffAndRemovePreviousManifests does not call docsByNamespace
https://github.com/replicatedhq/kotsadm/blob/master/operator/pkg/client/deploy.go#L105

Unable to run local development environment

I ran the following:

make kotsadm
skaffold dev

The builds go through fine, but I get a big pile of errors for github.com/schemahero/schemahero CRs

 - unable to recognize "STDIN": no matches for kind "Database" in version "databases.schemahero.io/v1alpha2"
 - unable to recognize "STDIN": no matches for kind "Table" in version "schemas.schemahero.io/v1alpha2"
 - unable to recognize "STDIN": no matches for kind "Table" in version "schemas.schemahero.io/v1alpha2"
 - unable to recognize "STDIN": no matches for kind "Table" in version "schemas.schemahero.io/v1alpha2"
 - unable to recognize "STDIN": no matches for kind "Table" in version "schemas.schemahero.io/v1alpha2"

What do I need to do to get this working?

upload command requires a namespace

kotsadm/web/src/components/tree/KotsApplicationTree.jsx

Line 237 in 7da30cd

{`kubectl kots upload --slug ${this.props.match.params.slug} ~/${this.props.match.params.slug}`}

i think that for this upload to work the cluster operator will need to provide the -n $namespace

kotsadm deploys the first version of a replicated app before the config is saved

When deploying a new replicated app using the UI-based kotsadm workflow, we should not deploy the first version until after the initial config is saved, if there is a config screen.

Multi-app, GitOps versions behind is off

kotsadm/web/src/components/watches/WatchSidebarItem/KotsSidebarItem.jsx

Line 29 in 235c1f4

versionsBehindText = `${isBehind} ${isBehind >= 2 || typeof isBehind === 'string' ? "versions" : "version"} behind`

it seems that after an update or two, the sidebar thinks that the app is perpetually 2 versions behind.

Method to determine whether kotsadm-operator was able to get elevated permissions to deploy CRDs

As an application packager, I'd like to configure my application to deploy CRDs along with the application itself. This only works, however, if kotsadm-operator was able to get elevated privileges at the cluster scope. I'd like to be able to query whether this happened before deploying my app, because if we don't have permission to add CRDs, then I will need kotsadm to run a preflight check to check those CRDs exist.

Not sure if this is a template function or annotation or what. The acceptance criteria for this story is the ability to dynamically decide whether to either

1. include a preflight check for CRDs and skip CRD deploy
2. deploy CRDs at runtime and skip preflight check

and to execute that. I'm not sure if preflight supports conditional evaluation yet but if not then that should part of the scope. We also need to make sure CRDs can be dynamically excluded.

Example (aspirational)

In preflight.yaml

analyzers:
  - customResourceDefinition:
       apiVersion: some.group.com
       kind: myobject
       when: repl{{ not (KotsadmOperatorClusterAdmin) }}

In crds.yaml

kind: CustomResourceDefinition
metadata:
  name: myobject
  annotations:
    kots.io/when: repl{{ KotsadmOperatorClusterAdmin }}

Development Environment: Does not match production kots installs

I noticed that the development environment for kotsadm deploys the github.com/schemahero/schemahero operator and CRDs, and that skaffold creates and terminates the namespace on every skaffold dev invocation/exit. This causes two issues:

1. cannot ctrl+c skaffold dev and then run it again, need to wait some time for schemahero namespace to terminate, or else you get errors like

Error from server (Forbidden): error when creating "migrations/kustomize/base/schemahero.yaml": secrets "schemahero-webhook-server-secret" is forbidden: unable to create new content in namespace schemahero-system because it is being terminated

2. This does not match what the kots CLI does in production, which is to run a single Job that invokes the schemahero CLI (using schemahero Operatorless Mode)

Would it be reasonable to convert the local dev environment to closer match what happens in production, using a Job to do applies?

Job pods not being delete when job is deleted via hook delete policy

I'm not sure what the intent is but I was expecting for the job pods to be deleted when the job is deleted.
It looks like this PropagationPolicy is missing here: https://github.com/replicatedhq/kotsadm/blob/master/operator/pkg/client/hooks.go#L71

If the intent is not to delete the pods then please close this issue.