elastisys / compliantkubernetes

Documentation for the Compliant Kubernetes project

Compliant Kubernetes Documentation

This is the main repository for documentation about the Compliant Kubernetes project. For Compliant Kubernetes code, please refer to:

• compliantkubernetes-kubespray for setting up a vanilla Kubernetes cluster on top of a compliant cloud provider;
• compliantkubernetes-apps for augmenting a vanilla Kubernetes cluster with security and observability.

Prerequisites

Python 3. You can check that it is already present on your Linux/macOS as follows:

python3 --version

For generating figures, please install:

sudo apt-get install graphviz make

For generating docs/stylesheets/style.css, please install:

npm install -g sass

Usage

Note

For Mac users, you might have to install cairo: brew install cairo

To view locally:

python3 -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txt

mkdocs serve

Caution

The command mike serve also works to preview a page, but it does not seem to support live preview. This means that you need to restart mike serve after every file change, which is not really productive

• To view locally: mike deploy compliantkubernetes ck8s -t 'main' and then mike serve.
• To re-generate figures: make -C docs/img. For simplicity, please commit generated figures. Prefer PNG (width == 1200px), to facilitate embedded logos.
• For continuous preview of figures: make -C docs/img preview.
• To generate docs/stylesheets/style.css, please use sass extra_sass/style.css.scss > docs/stylesheets/style.css.

Tech Stack

• mkdocs
• mkdocs-material
• GitHub Pages
• Graphviz
• mike
• sass

Deployment

GitHub Actions will deploy the main branch automatically.

compliantkubernetes's Issues

Document Disaster Recovery

Please fill docs/operator-manual/disaster-recovery.md with information on how to backup and recover a Compliant Kubernetes cluster. Specifically:

• Why are backups required for compliance? (https://www.isms.online/iso-27001/annex-a-12-operations-security/, A.12.3.1 Information Backup)
• What should be backed up?
• What is backed up in Compliant Kubernetes? When? Whereto?
• How to test backups of Compliant Kubernetes?
• How to restore Compliant Kubernetes?
• Further reading: Velero.

Acceptance Criteria

• $FILE documents items above.
• $FILE conforms to Contributor Guide in README.md.
• If any user actions are required, please create an issue for missing documentation in the "User Guide" part of the documentation.

[1] Document OpenDistro/Elasticsearch once OpenID works

The dashboard documentation does not contain screenshot with OpenDistro/Elasticsearch, since OpenID support does not currently work in the demo environment. Once OpenID support is enabled in the demo environment, please update docs/user-guide/dashboard.md.

Acceptance criteria:

• $FILE contains relevant text.
• $FILE contains relevant screenshots.
• $FILE conforms to the Contributor Guide in README.md.

[1] Document Prometheus

Please fill docs/user-guide/prometheus.md. Specifically:

• What is the compliance need?
  • Capacity management (see https://www.isms.online/iso-27001/annex-a-12-operations-security/, A.12.1.3)
  • Incident management (see https://www.isms.online/iso-27001/annex-a-16-information-security-incident-management/)
• Why Prometheus and Grafana? (go-to solution; CNCF graduate)
• What does Prometheus offer out-of-the-box in Compliant Kubernetes?
  • Node health (with screenshot)
  • Pod health (with screenshot)
• Links to further reading: Prometheus, Grafana

Acceptance criteria:

• $FILE contains above content.
• $FILE follows Contributor Guide in README.md.

Update Operator Manual openstack about terraform init 0.15

https://compliantkubernetes.io/operator-manual/openstack/#initialize-and-apply-terraform
https://compliantkubernetes.io/operator-manual/openstack/#cleanup

MODULE_PATH="$(pwd)/kubespray/contrib/terraform/openstack"
pushd "${MODULE_PATH}"
for CLUSTER in "${SERVICE_CLUSTER}" "${WORKLOAD_CLUSTERS[@]}"; do
  terraform init 
  terraform apply -var-file="${CK8S_CONFIG_PATH}/${CLUSTER}-config/cluster.tfvars" -state="${CK8S_CONFIG_PATH}/${CLUSTER}-config/terraform.tfstate"
done
popd

review and update cluster setup documentation on exoscale

[2] Document how to use Ambassador as ingress controller

Support for Ambassador (Ingress controller - https://www.getambassador.io/docs/latest/) as an option to nginx has been requested.

Acceptance criteria:

• Can set up compliant kubernetes in a way that Ambassador is used instead of nginx as ingress controller in the workload cluster.
• Show how to configure key Ambassador settings for workload cluster.

Implementation hint:

• Kubespray seem to support Ambassador already: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ambassador.md
• Installing as cluster user (via Helm etc.): https://www.getambassador.io/docs/latest/tutorials/getting-started/

Document Operator Manual - Getting Started

Please fill docs/operator-manual/getting-started.md. Content is up to someone working in operations.

Acceptance criteria:

• $FILE helps someone get started, i.e., have the right tools installed.
• $FILE conform with the Contributor Guide in README.md.

Operator manual for vSphere

Now that kubespray supports vSphere thanks to @jakubkrzywda and kubernetes-sigs/kubespray#7306, we ought to have an Operator manual that shows how to install Compliant Kubernetes using it.

Deferral notice (2021-03-01)

Please defer this issue until we have determined whether we can have a unified Operator manual format or not.

Figure out how we should use Terraform v0.15

Terraform v0.15 introduces some breaking CLI changes (hence confirming the suspicion that it is a 0.x version product 😄).

In particular, terraform init and apply no longer take a "module" argument, a behavior on which we heavily rely on in our documentation, e.g., here.

Let's figure out a new way to achieve the same effect.

Desirable properties of the new way:

• I have a config repo, which submodules compliantkubernetes-kubespray which submodules kubespray.
• I can check in the TF configs in the config repo.
• terraform init and terraform apply leave the compliantkubernetes-kubespray and kubespray untouched, i.e., without dirty files.

Network Policy example: permit only certain Pods in a certain namespace to connect to a database

As per the Product Management meeting on 2021-06-14, we should have easy-to-understand examples online that shows the utility of policy as code. These should be showcased on the https://compliantkubernetes.io/ciso-guide/policy-as-code/ page, with some minor text accompanying them.

This issue requests exemplifying the power of Network Policies by restricting such that only Pods in a certain Deployment and Namespace can connect to a database, running in a StatefulSet in the same Namespace.

[1] Document Dashboard

Please fill docs/user-guide/dashboard.md with the relevant content, about CK8s dashboard:

• What is the compliance need? (https://www.isms.online/iso-27001/annex-a-18-compliance/, A.18.2.3 Technical Compliance Review)
• Page-by-page description of the dashboard with one snapshot per page.

Acceptance criteria:

• $FILE is filled with relevant content.
• $FILE follows the Contributor Guide in README.md.

[1] User-facing release notes

The changelog in compliantkubernetes-apps and compliantkubernetes-kubespray is mostly written for developers and operators, not for users. It would be nice to also have some sort of changelog or releasenotes for the users. It could be placed in the respective git repos, but I think it could be a good fit to put it here instead.

I suggest that this documentation is updated when we create a new release for one of the projects, not when a change is merged to one of the main branches. I also think that we should link to the full changelogs as well.

Policy as code example: No deleting namespaces that are marked as “protected” via a label

As per the Product Management meeting on 2021-06-14, we should have easy-to-understand examples online that shows the utility of policy as code. These should be showcased on the https://compliantkubernetes.io/ciso-guide/policy-as-code/ page, with some minor text accompanying them.

This issue requests a policy that prevents deletion of namespaces that are marked with a label that says protected=yes.

You can find the example code here, actually, so perhaps re-host it and of course link to the original author's work.

[2] Document how to handle cloud credentials

Ideally, I would like the process to be "create cloud-config manually, restart the cloud-controller".

Note: SafeSpring currently only supports Cinder v2, which requires the in-tree Kubernetes cloud provider, so yes, we need to restart the cloud-controller, as opposed to deploying a Helm chart with an out-of-tree cloud provider.

Ensure docs properly explain how to create and configure Dex in azure

Apparently there's some issues with the docs in https://dexidp.io/docs/connectors/microsoft/

So it would be nice to just double check exactly what is needed and then update the upstream documentation or add some document in here.

[1] harbor - add new users

Add instructions/guidelines on how to add new uses to Harbor (with and without OIDC).

[2] Unify operator-manual apps configuration

Throughout the operator-manual of the different cloud providers and kubernets installers we do differ in the documentation for the configuration changes to compliantkubernetes-apps. This makes it more difficult to make sure that the documentation is up-to-date.
It would be nice to unify the common settings using the same format.

Policy as code example: Only allow members of group X to delete a namespace

As per the Product Management meeting on 2021-06-14, we should have easy-to-understand examples online that shows the utility of policy as code. These should be showcased on the https://compliantkubernetes.io/ciso-guide/policy-as-code/ page, with some minor text accompanying them.

This issue requests a policy that prevents deletion of namespaces unless the user who issues the call is part of a particular group.

The point of this is to show how one could protect, e.g., kube-system from non-administrators, but to allow namespace creation and deletion for other namespaces.

Add Upgrade instructions to Administrator Manual

Users that want to self-manage Compliant Kubernetes need documentation that tells them how to upgrade to new versions. This information is probably available, but in spite of the section of the documentation mentions upgrades in the title, I do not actually see instructions for it for any cloud.

Please add upgrade instructions, even if these are as simple as "pull from this git repo, re-run kubespray, then follow release notes in the apps project" or whatever.

Duplicated clean up instructions

The instructions on how to clean up the cluster are duplicated:

• docs/operator-manual/common.md
• docs/operator-manual/clean-up.md

Currently, both files are in use and referenced from other parts of the documentation.

Document Operator Manual - AWS

Please create a tutorial on how to set up Compliant Kubernetes on top of AWS.

https://github.com/elastisys/compliantkubernetes/blob/main/docs/operator-manual/getting-started.md

Acceptance criteria:

• Please provide a Markdown document which interlaces prose and copy-paste-able code on how to set up the latest version of Compliant Kubernetes on top of AWS.
• The document should be accessible for someone familiar with common tools, such as Terraform, Ansible, Kubernetes, Helm and Helmfile, but unfamiliar with Compliant Kubernetes.

[1] Document Kubernetes API

Please fill the file docs/user-guide/kubernetes-api.md with relevant content:

• How does the kubeconfig of a Compliant Kubernetes cluster typically look? How does a user install/merge it with their existing kubeconfig?
• What Kubernetes roles does a user have by default? What is allowed and what is disallowed? Why?
• Please show step-by-step how to use kubectl, including auth flow via Dex and Terminal screenshots.
• Further reading: Dex, Kubernetes Tutorial, other relevant documentation

Acceptance criteria:

• $FILE is filled with above content.
• $FILE follows Contributor Guide in README.md.

Replace documentation referring to old dashboard

https://compliantkubernetes.io/user-guide/dashboard/ refers to the deprecated ck8s-dash project and should be replaced with up-to-date documentation about current available dashboards (Grafana, etc.).

[1] Document backup via Velero

Please create a new file docs/user-guide/backup.md with user-facing documentation on how Compliant Kubernetes addresses the backup need. This text may serve as inspiration. Specifically:

• Why backup? (see A.12.3.1 "Information Backup" in https://www.isms.online/iso-27001/annex-a-12-operations-security/)
• What does the user need to back up?
  • Container Images => no, Harbor is auto-magically backed up in Compliant Kubernetes
  • Kubernetes Resources => yes, but consider using GitOps
  • PVCs => yes, use Velero (see below)
• Why Velero?
• What is backed up? How to configure backup? (Please give copy-paste-able code examples.)

Acceptance criteria:

• $FILE contains content above.
• $FILE follow Contributor Guide in README.md.
• mkdocs.yml's nav section is updated to point to $FILE.

[1] Document Harbor

Please put content in the docs/user-guide/harbor.md file. Specifically:

• What is a container registry and why it is needed? (with GraphViz/dot diagram of what is around the container registry, i.e., CI/CD pipeline and Kubernetes cluster)
• What is Harbor?
• Why is Harbor used in Compliant Kubernetes?
• How to achieve compliance, in particular technical vulnerability management, with Harbor?
  • Image scanning (with screenshot of Harbor)
  • Restricting pull access of vulnerable images (with screenshot of Harbor and failed kubectl run / kubectl describe deployment)
  • Mention that by default Compliant Kubernetes only allows pulling images from Harbor or official Docker images (with screenshot of failed and successful pull)
• Further reading, linking to Harbor, Trivy and OpenPolicyAgent.

Acceptance criterial:

• docs/user-guide/harbor.md contains above content.
• docs/user-guide/harbor.md conforms with Contributor Guide in README.md.

[1] Update documentation after updating kubespray wrapper

Acceptance criteria:

• Review and test all documents in https://compliantkubernetes.io/ > Operator Manual > Create Cluster
• Ensure steps are still up-to-date

Add GCP logo to supported clouds

Since we support GCP, it would be nice to have that logo there in the logo cloud on the "Compliance basics" page as well.

[2] Update operator-manual apps configuration

We've made changes to the apps repository that has not been updated in the operator-manual docs, meaning the apps configuration docs are out-of-date.

Someone with good knowledge on the apps repo should go through the docs and update them so that they are more likely to work.

Also it might be smart to checkout a specific apps version in the docs.
Then it makes it easier to know when these documentations are out-of-date.

Lacking cloud integration for aws

There is no mention of how to use cloud integration when running on aws.
https://compliantkubernetes.io/operator-manual/aws/

[2] Add Contributor Guide

There seems to be some "tribal knowledge" on how to develop most efficiently with CK8s. This is particularly problematic with quick issues. Let's write a Contributor Guide to lift this tribal knowledge and give best practices on how to develop quickly. This guide could serve as inspiration.

[1] Document how to access Falco and OPA logs

Unsure where to track this.

Currently, it is unclear how to access Falco and OPA logs. It would be great to clarify that and update the documentation accordingly.

[1] Document Compliant Kubernetes Logging (Elasticsearch)

Please fill the docs/user-guide/elasticsearch.md document with relevant content:

• Why logging? (see A.12.4.1 in https://www.isms.online/iso-27001/annex-a-12-operations-security/)
• Why Elasticsearch?
• What logs can be found in Elasticsearch? Describe each of the following:
  • Application logs
  • Kubernetes API logs
  • Falco warnings
  • OPA warnings
• Give a few example queries:
  • How to find logs of a specific Pod?
  • How to find all Kubernetes API actions of a specific user?
  • How to find Falco alerts?
• Link to further reading: Elasticsearch, Kubernetes API logs, Fluentd, OPA

Acceptance criteria:

• $FILE contains content above.
• $FILE follow Contributor Guide in README.md.

Document Compliant Kubernetes Configuration

Please fill docs/operator-manual/configuring.md. An operators should know best what this document should contain and how to structure it.

Acceptance criteria:

• $FILE documents how to configure / re-configure a Compliant Kubernetes cluster.
• $FILE conforms with Contributor Guide in README.md.

Document checklist

Please document what "else" is needed to ensure Compliant Kubernetes is a compliant and secure platform. Specifically, think of "non-code" issues like:

• Operator and user training
• Setting up individual SSH keys
• Daily/weekly checklist
• Upgrades/QA etc.

[2] Tutorial: document backup restore of compliant kubernetes

Document how to use user-alertmanager

We get questions on how to set up alerts for application metrics. Let's document this.

Acceptance criteria:

• compliantkubernetes.io / User Guide / Alerting contains information on how to create an alert for an application deployed by the user.

Document Compliant Kubernetes on Baremetal

Lennart knows what to write here. :)

Ensure AWS documentation is up to date

@OlleLarsson recently set up Compliant Kubernetes on AWS, but found that the documentation is not entirely up to date. This ought to be fixed. Olle, if you are reading this, can you share some more details to help whoever picks this task up?

[2] Document usage of NetworkPolicies

Please document some suggested use of NetworkPolicies.
Preferably add https://github.com/ahmetb/kubernetes-network-policy-recipes as a reference for further reading.

[3] Document Compliant Kubernetes on OpenStack

We noticed that we are lacking documentation for OpenStack, which is on-part with e.g. Exoscale. Let's fix this. An initial sketch is here (#90) and the recent unification (#84) should make this tasks a bit quicker.

Please be mindful about:

• Ensuring proper sizing: For example, the defaults in the Exoscale Terraform provider are "just right" for CK8s. Docs for Managed OVH Kubernetes include "sizing hints". Docs for AWS and EKS-D have commands (sed) to set nodes to t3.large.
• Make sure that snippets are separated into configure snippets -- need human thinking and intervention -- and apply snippets -- generally don't need human intervention and are idempotent.
• Include verification steps after apply steps. More is better. How would you convince yourself that the apply step succeeded? The verification steps would also serve for troubleshooting. Prefer non-destructive verification steps.

Ensure AWS docs properly explain how to create S3 buckets and configure Dex

How to create S3 buckets is already covered here, but does not seem to be used in the AWS documentation.

How to configure Dex is briefly covered here, but should be factored out in common.md.

[5] Document DR procedures

We currently have some DR documentation but they are far from complete. Let us document how to thoroughly test DR and perform DR. The document should be written with the following expectation:

1. The on-call person will pre-read the docs in non-panic mode.
2. The on-call person will execute the docs in panic mode.

ADR on external health checks

We currently have a few "external" health checks, where we use the blackbox exporter to check public endpoints of some services.
This is done to measure uptime and health of the services from an end users perspective (e.g. it must be considered downtime if the ingress is down but the application behind it is still functioning).
Unfortunately, we don't have a decision on how and what should or should not be exposed in these health checks.
Some of them currently expose the "full status" of the application publicly, while other hide it and consider a 401 or 403 status code as "healthy". See the current configuration.

To unify the health checks and make sure we don't expose information that we do not want to, we need an ADR to guide the continued development.

Definition of done: An ADR is merged to this repository with a clear decision on what can or cannot be exposed in public health checks. This ADR should consider the need to measure uptime and health of the services that are part of Compliant Kubernetes.

Improve documentation on creating a service account for CI/CD

We get questions on how to create a service account for CI/CD. Please improve the documentation here, based on feedback received from our customers: https://compliantkubernetes.io/user-guide/ci-cd/

Continue unifying the operator-manual apps configuration

We started unifying the documentation for AWS, Azure and Exoscale (see #84 and #89), but ran out of time. Let's continue with GCP and OVH.

Refactor documentation to highlight differences and similarities between various cloud environments that Compliant Kubernetes supports

Instead of having 1 documentation per supported cloud provider, it would be great to focus on commonalities and differences. We should use Content tabs.

Document Creating/Destroying/Upgrading a Cluster

Please fill docs/operator-manual/lifecycle.md. Operators should be in the best position to decide how to structure this document and what information it should contain.

Acceptance criteria:

• $FILE describes how to create/destroy/upgrade a Compliant Kubernetes cluster.
• $FILE conforms with Contributor Guide in README.md.
• $FILE uses only tools describe in docs/operators-manual/getting-started.md.

Search

I find myself using the search on the GitHub to search the docs through the repository quite often.
Perhaps it would be a good addition to introduce a search on the website as well?

It appears to be some kind of possible according to https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/.

Document break glass

Please fill docs/operator-manual/break-glass.md with information on how to deal with "break glass" scenarios in Compliant Kubernetes. Break glass scenarios are those in which the system has gone in such an unrecoverable situation, that normal access control methods need to be overridden. For example, cluster-admin token or root SSH key might need to be taken out.

Acceptance criteria:

• $FILE is filled with relevant information.
• $FILE conforms with "Contributor Guide" in README.md.

Document cluster installation on Google Cloud

Description

1. Install and configure Compliant Kubernetes on Google Cloud. Make sure to make relevant setting such that dynamic storage provisioning and load balancing works with the cloud provider's services.
2. Document the procedure in a fashion similar to this Operator's Guide for Exoscale.

Acceptance criteria

• Documentation complete and well-written, matching the overall style of the documentation already on compliantkubernetes.io
• Another engineer can copy-paste commands from the documentation and get a fully-working Compliant Kubernetes setup on Google Cloud. The review process must capture this aspect.