k3s-io / docs

k3s Documentation

K3s Website and Docs

This repo contains the content of the K3s documentation website found at https://docs.k3s.io. Please open an issue if you have suggestions for new content or edits. We also gladly accept community PRs.

The website is built using Docusaurus 2, a modern static website generator.

Installation

$ yarn

Local Development

$ yarn start

This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.

You can also use Docker to launch the website without needing to install and configure yarn:

docker run --rm -it -v $PWD:$PWD -w $PWD -p 3000:3000 node yarn start -h 0.0.0.0

Build

$ yarn build

This command generates static content into the build directory and can be served using any static contents hosting service.

Deployment

An automated GH action will deploy the website to GitHub Pages once a PR has been merged to main.

docs's Issues

Reorganize networking content in the docs

Networking information is divided into two different sections:

• https://docs.k3s.io/installation/network-options
• https://docs.k3s.io/networking

I think that separation is strange and confusing. As a consequence we have related content spread across these two. We should think of the reasons why we need two different sections and have a clear separation for each of them. Right now it is not clear what goes where

The kubernetes-dashboard cannot be installed through the document

https://docs.k3s.io/installation/kube-dashboard#deploying-the-kubernetes-dashboard
The dashboard of k8s has released the latest version v3.0.0-alpha0, and the installation method has been updated. Using the current document method to install will prompt a 404 error.Maybe you should limit the installation to a specific version instead of getting it from the github release page.

Packaged components list is out of date

On https://docs.k3s.io/installation/packaged-components it says:

K3s comes with a number of packaged components that are deployed as AddOns via the manifests directory: coredns, traefik, local-storage, and metrics-server.

And later:

Care should also be taken not to conflict with names in use by the default K3s packaged components, even if those components are disabled.

This requires that the list of packaged components is complete, but it is missing several names, like ccm, rolebindings, runtimes.

No reference to `k3s-ansible` repo in documentation

There is no reference to the k3s-ansible repository in the documentation.

This could be added in the Quick-Start page. This would allow easy bootstrap of multi-node cluster. Since the k3s-ansible repo is already part of the k3s-io system, i don't see how it would be a problem.

automatic upgrade, not tolerating my default and still on deprectaded k3s version

https://docs.k3s.io/upgrades/automated

I wanted to suggest adding the well known toleration for control plane by default
node-role.kubernetes.io/control-plane
https://kubernetes.io/docs/reference/labels-annotations-taints/#node-role-kubernetes-io-control-plane-taint
and update the examples to provide a more recent kubernetes version..

i can make a PR if this decision was not on purpose, i mean, i guess by default k3s doesn't have this toleration
but having it so causes no harm?

Failed to determine GroupVersionResource for manifest

Ran kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml --insecure-skip-tls-verify

Output:

kubectl get deploy -n system-upgrade  --insecure-skip-tls-verify
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
system-upgrade-controller   1/1     1            1           21m

Now ran my plans.tf file:

resource "kubernetes_manifest" "server_plan" {
  manifest = {
    apiVersion = "upgrade.cattle.io/v1"
    kind       = "Plan"
    metadata = {
      name      = "server-plan"
      namespace = "system-upgrade"
    }
    spec = {
      concurrency = 1
      cordon      = true
      nodeSelector = {
        matchExpressions = [
          {
            key      = "node-role.kubernetes.io/control-plane"
            operator = "Exists"
          }
        ]
      }
      tolerations = [
        {
          operator = "Exists"
          effect   = "NoSchedule"
        }
      ]
      serviceAccountName = "system-upgrade"
      upgrade = {
        image = "rancher/k3s-upgrade"
      }
      channel = "https://update.k3s.io/v1-release/channels/stable"
    }
  }
}

resource "kubernetes_manifest" "agent_plan" {
  manifest = {
    apiVersion = "upgrade.cattle.io/v1"
    kind       = "Plan"
    metadata = {
      name      = "agent-plan"
      namespace = "system-upgrade"
    }
    spec = {
      concurrency = 1
      cordon      = true
      nodeSelector = {
        matchExpressions = [
          {
            key      = "node-role.kubernetes.io/control-plane"
            operator = "DoesNotExist"
          }
        ]
      }
      tolerations = [
        {
          operator = "Exists"
          effect   = "NoSchedule"
        }
      ]
      prepare = {
        args  = ["prepare", "server-plan"]
        image = "rancher/k3s-upgrade"
      }
      serviceAccountName = "system-upgrade"
      upgrade = {
        image = "rancher/k3s-upgrade"
      }
      channel = "https://update.k3s.io/v1-release/channels/stable"
    }
  }
}

On apply I get the error:

Error: Failed to determine GroupVersionResource for manifest
│ 
│   with kubernetes_manifest.server_plan,
│   on plans.tf line 1, in resource "kubernetes_manifest" "server_plan":
│    1: resource "kubernetes_manifest" "server_plan" {
│ 
│ no matches for kind "Plan" in group "upgrade.cattle.io"
╵
╷
│ Error: Failed to determine GroupVersionResource for manifest
│ 
│   with kubernetes_manifest.agent_plan,
│   on plans.tf line 35, in resource "kubernetes_manifest" "agent_plan":
│   35: resource "kubernetes_manifest" "agent_plan" {
│ 
│ no matches for kind "Plan" in group "upgrade.cattle.io"

"Stopping K3s" page is misleading

Discussed in k3s-io/k3s#8382

Hello,

Following the discussion linked above, I think this whole documentation page should be moved and redone : https://docs.k3s.io/upgrades/killall

There is no sense having this page in the upgrades section (as it is not needed for upgrades), maybe it could be a new docs section, or a paragraph in the FAQ ?

This page could also be expanded to give general instructions about stopping and starting the k3s service (without killing the pods) with a paragraph dedicated to the killall script as a special case.

k3s - update on the air-gap installation instruction

Original issue: https://github.com/rancher/docs/issues/2351

Copying over relevant bits from original issue since K3s docs have been moved off the old rancher/docs repo.

---

During the validation for this issue: rancher/rancher#25495 (comment)

I found there are some improvements we can do for the documentation for k3s air-gap installation: https://rancher.com/docs/k3s/latest/en/installation/airgap/

• for the section Prepare Images Directory, add some explanation for why we still need this when we have a private registry
• for the section Create Registry YAML, add some explanation for when and how the private registry will be used by k3s, why the private registry is not used for k3s' own images
• for the section Install K3s, add a sample command when installing k3s with an external database

NATS datastore

There is no info on how to use NATS datastore with k3s in https://docs.k3s.io/datastore

Ubuntu 22.04 has nftables enabled by default

I’ve just set up a new machine with the latest, stable Ubuntu 22.04. This release has nftables enabled by default, so requires the same fix as Raspberry Pi OS to revert to legacy iptables. I would suggest updating the requirements page to reflect this and perhaps updating the known issues section.

Note that this issue surfaced when trying to define Ingress resources to access a simple HTTP service (whoami) on a default k3s installation (i.e. as per the quick start instructions). Matching requests to the public traefik ports (80 & 443) were failing with a 504 timeout error because it couldn’t route the network traffic, even though the service was accessible via the private k3s network address associated with the service. Enabling traefik debug didn’t shed any light on the problem, and I spent lots of hours trying to track down the issue.

I’ve attached a file containing deployment, service and ingress definitions to reproduce, assuming whoami is added to /etc/hosts as an alternative hostname for the machine.
whoami.txt

docs: cli documentation nor faq describes what a server or agent is.

Environmental Info:
K3s Version:

Node(s) CPU architecture, OS, and Version:

Cluster Configuration:

Describe the bug:

Browsing the documentation at
https://docs.k3s.io/faq
https://docs.k3s.io/cli/server
https://docs.k3s.io/cli/agent

Does not link to, or define what a server or agent is.

However, this is described in https://docs.k3s.io/architecture

The faq and cli documentation should link to the architecture page for words like agent and server, as the CLI and FAQ are more prominent entry places for a user than the architecture overview.

Steps To Reproduce:

• Installed K3s:

Expected behavior:

Actual behavior:

Additional context / logs:

Update Sizing Guide based on stress test data

Similar to rancher/rke2-docs#111, we need to update the https://docs.k3s.io/installation/requirements#large-clusters with newer information. Additionally, the page should be reworked to consolidate the "large cluster" info with the "single node" information in the upper portion of the page.

[Translation] K3s Docs Into Arabic.

Hi,

I would like to start translating the docs of K3s into Arabic, to allow Arabic speakers to understand better this project.

Best regards.

[K3s] add --audit-policy-file parameter in hardening guide

Request Summary:

In current hardening guide example, we don't have any --audit-policy-file parameter defined in the k3s server command example.

Details:

If we don't provide any --audit-policy-file, the audit logging is disabled (default behaviour of k8s).

Having some kind of "default" audit.k8s.io/v1:Policy file and the parameter in the command example could be a better description of "how to enable it", especially when we enable every other parameters about audit-log.

Thank you

BTW, if you provide me some information about what we should do on this one, I would be able to do a PR.

NOTE: Creation requested by @jtravee in this comment, as a clone of https://github.com/rancher/docs/issues/3656

Incorrect file name listing k3s images

In "docs/installation/private-registry.md" in the section "Adding Images to the Private Registry" it says to:
"obtain the k3s-images.txt file from GitHub", but that file does not seem to exist.

However, a file "scripts/airgap/image-list.txt" does exist.

Perhaps that is the intended file?

improve config.yaml documentation, provide example with all available keys

https://docs.k3s.io/installation/configuration#configuration-file
i was looking at the ways we can configure a k3s node and i would like to give a try to the config.yaml method

however it's not exactly clear what can i include on this file,
there is a good example about the merging of configuration of different sources
but i am having some doubts about if every flag or setting is available:

for example:

can i specify k3s version? (i believe not but that would be cool)
can i put flannel related settings, and specify ip addresses?
can i specify the node is a server or an agent or is it assumed? [server: true; agent:true]?
does the file allows to configure server only flags? (i guess its possible due to have the cluster init example)
does it support envs or subshelling from other sources? (like $(ip address | grep....))

my actual question is, can i have every k3s setting in this file?

I wanted to suggest to add to the table the corresponding config keys in the following page,
or since they all have the same names, if these are available as a config.yaml setting
https://docs.k3s.io/reference/env-variables

English Errors in Rancher products K3s page

Page: https://www.rancher.com/products/k3s

There are 2 errors on this page:

1. In the text block under "Why K3s and Rancher?->A CNCF Sandbox Project" the description for this is missing the word "to", it should read:
   "K3s was originally a Rancher project. In June 2020 Rancher donated K3s to the CNCF as a testament to the Rancher's commitment to the open source community"

2. In the text block under "Why K3s and Rancher?->Perfect for Edge" the word "available" is misspelled "avalible".

IPv6-only instructions are missing

There is a section for "Single-stack IPv6 installation", however there's no instructions on how to actually do it - the section is empty except for warnings.

Docs for RHEL / CentOS should also mention they apply for Fedora

This note here on firewalld should also mention that this apply to Fedora as well. It will be easier find-able if one use search and put in Fedora.

Installation of CRD for system-upgrade-controller is missing

Instructions for installation of CRD for system-upgrade-controller is missing.

Following the instructions will lead to a message like this when applying the plans:

resource mapping not found for name: "server-plan" namespace: "system-upgrade" from "system-upgrade-plan.yaml": no matches for kind "Plan" in version "upgrade.cattle.io/v1"
ensure CRDs are installed first
resource mapping not found for name: "agent-plan" namespace: "system-upgrade" from "system-upgrade-plan.yaml": no matches for kind "Plan" in version "upgrade.cattle.io/v1"
ensure CRDs are installed first

related: rancher/system-upgrade-controller#298

Document use of base template for containerd config

Need to document the improved templating functionality from

• k3s-io/k3s#7991

With TLS and Without TLS examples might be misleading

As this is about Private Registry Configuration, the https://github.com/k3s-io/docs/blob/main/docs/installation/private-registry.md#with-tls and https://github.com/k3s-io/docs/blob/main/docs/installation/private-registry.md#without-tls examples might be misleading. In particular,

mirrors:
  docker.io:

should be

mirrors:
  mycustomreg.com:

A reader may skip over the first mirrors example and use an example from either TLS section. The result is an error "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \" that the image cannot be found.

Slack link in docs is weird

The slack link on this page: https://docs.k3s.io/faq#im-having-an-issue-where-can-i-get-help, which is https://rancher-users.slack.com/archives/CGGQEHPPW takes me to a page where I needed to have suse or rancher email address to sign in.

The slack link on the main rancher.com page works. It sends me to: https://slack.rancher.io/

Docs might need updating to reflect PSP depreciation

At present the docs do a good job of explaining how to harden k3s by using PSPs.

However, it appears that PSPs are deprecated as of Kubernetes 1.21

This likely means that the docs should be updated accordingly.

Unfortunate I'm far from a subject-matter expert in this area, so I can't viably propose a PR myself, hence I am just flagging this here.

Document pitfalls when attempting a downgrade

Downgrades are NOT SUPPORTED and are not something that we recommend anyone to do. We have made some updates when using system-upgrade-controller now that will fail earlier in the process if a user attempts a downgrade on the cluster, but should document the potential pitfalls when having cordon: true set in the upgrade plans.

Document the default(?) path used by local-path-provisioner for --default-local-storage-path

My understanding is that the default for --default-local-storage-path is /var/lib/rancher/k3s/storage and so if you want to change where local-path-provisioner saves the PVs you need to pass that to server instances, is that correct? If so, can that be added to the docs (I'm happy to make a PR if that helps), and perhaps the option of changing the storage path should be given on https://docs.k3s.io/storage#setting-up-the-local-storage-provider, given its a k3s-specific option?

Certificate Rotation in Advanced Options ?

Is there any reason why Certificate Rotation is hidden away under Advanced Options ?

Surely its a fairly critical bit of sysadmin and not an advanced option ? Maybe it could go under "Installation" instead ?

Also, perhaps a bit of clarity on what breaks if the certificates go unrotated would not go amiss ? Equally I'm unclear why k3s needs a hard reboot and not a reload (SIGHUP) ?

air-gap installation instruction to include steps for installing k3s-selinux packages offline

For air-gap installation, provide instructions on how to download and install container-selinux and k3-selinux package manually/offline.

[INFO]  Skipping k3s download and verify
[INFO]  Skipping installation of SELinux RPM
[ERROR]  Failed to find the k3s-selinux policy, please install:
    dnf install -y container-selinux
    dnf install -y https://rpm.rancher.io/k3s/stable/common/centos/8/noarch/

Add "why the name kiss 'k3s'" to FAQ?

I was chatting with someone last night about k3s, and I said, "well I'm pretty sure k3s is a shortening of KISS (keep it simple,)" and they said, "wouldn't that be k2s?" This made me wonder, why is it called "k3s", and I couldn't find any information about it on the website or documentation. Would you consider adding it to the FAQ?

Thanks!

Be clear about flag locations

data-dir and other flags need to be given "after" all subcommands for them to used properly. We need to warn the user about this in the docs.

i.e. k3s --data-dir=/tmp/k3s etcd-snapshot save is invalid. You need k3s etcd-snapshot save --data-dir=/tmp/k3s

Helmchart fields are out of date

As mentioned in some k3s issues, https://docs.k3s.io/helm#helmchart-field-definitions is out of date and needs to be updated with all the latest fields. We need to cover everything in https://github.com/k3s-io/helm-controller/blob/master/pkg/apis/helm.cattle.io/v1/types.go#L19 and https://github.com/k3s-io/helm-controller/blob/master/pkg/apis/helm.cattle.io/v1/types.go#L49.

Need clear documentation for upgrading hardened setup v1.24 to v1.25

Due to the drop of PSPs in v1.25, there needs to be clear documentation around how to perform an upgrade to this minor.

Documentation still shows --secrets-encryption as experimental

Per k3s-io/k3s#6581 the --secrets-encryption flag is no longer experimental, and the docs for managing secrets encryption doesn't mention it as experimental. But the reference for the Server Configuration still lists it under Experimental Options https://docs.k3s.io/reference/server-config#experimental-options

Unclear private registry documentation

The below is copied from https://github.com/rancher/docs/issues/3995.

Request Summary:

I found the page explaining private registry configuration confusing. It would help to change some of the examples and perhaps add some explanation so that people don't get confused.

Details:

In the With TLS section we have the following example:

mirrors:
  docker.io:
    endpoint:
      - "https://mycustomreg.com:5000"
configs:
  "mycustomreg:5000":
    auth:
      username: xxxxxx # this is the registry username
      password: xxxxxx # this is the registry password
    tls:
      cert_file: # path to the cert file used in the registry
      key_file:  # path to the key file used in the registry
      ca_file:   # path to the ca file used in the registry

From this, I understand that a mirror, docker.io, is being configured. Image names that reference this mirror use the base URL https://mycustomreg.com:5000 to find and download the images. However, it is not clear to me how the configs section applies to the mirror. Presumably this section applies to the docker.io mirror, but I don't see anything that references docker.io in this section. All we have is "mycustomreg:5000", which doesn't occur anywhere in the mirrors section. Was "mycustomreg:5000" supposed to be "mycustomreg.com:5000"? Or was it supposed to be docker.io? Or does it somehow relate to the mirrors section in a way I'm not seeing? Or not at all? This is the first point of confusion for me.

The second point of confusion is about the use of the name docker.io in the examples. As far as I know, docker.io is the default registry. So by defining a docker.io mirror in the private registries, are we changing the default registry? If so, this should be described in the docs. If not, perhaps a less confusing name than docker.io could be used in the examples.

These comments also apply to the Without TLS section.

`curl` command for downloading k3s airgap images not working

I was trying out the curl command to download the airgap images for k3s from the documentation, and it threw this error:

$ sudo curl -L -O /var/lib/rancher/k3s/agent/images/k3s-airgap-images-amd64.tar.zst https://github.com/k3s-io/k3s/releases/download/v1.29.1-rc2%2Bk3s1/k3s-airgap-images-amd64.tar.zst
curl: (3) URL using bad/illegal format or missing URL
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

I have fixed this in a PR #243 . I'd appreciate if the awesome team at k3s would take a look at it.
Thank you

Add the documentation for make traefik auto redirect to https

Aswome job for make kubernetes easier to all
I want to be able to find easy on k3s documentation how to auto redirect to https with traefik without modifing manifest

Like this way
kubectl apply -f httpstraefik.yaml

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      websecure:
        tls:
          enabled: true
      web:
        redirectTo: websecure

Its a common issue why people don't use the bundled traefik because they can't find how to make http to https redirection

Like we see in some forums
https://stackoverflow.com/questions/68575472/k3s-redirect-http-to-https/71989847#71989847

Move OS prereqs out of "advanced configuration"

[Moved from https://github.com/k3s-io/k3s/issues/8805, which I am now closing]

Thank you for the great work, and in advance for your consideration!

As the documentation says, nm-cloud-setup must be disabled for installation to proceed. I didn't see this, and therefore struggled to understand why the service would not start. I inspected the startup commands, and found the reference to nm-cloud-setup, and that led me to the conclusion it should be deleted.

"Well, problem solved, RTFM, right?" Indeed. And yet. I would like to suggest that installation prerequisites should not go into a section called Advanced Options / Configuration. I was just trying to do some quick and dirty testing, with nothing "advanced" on my mind! It would therefore not occur to me to read anything in a section with Advanced in the title. There is nothing to suggest I'd need to do this in the Quick Start Guide, nor in the Requirements section.

I would like to suggest the following:

• A blurb that links to Requirements at the top of the Quick Start Guide. No matter how quick you want to be, you need to meet the requirements, right?
• A clear callout in said Requirements to common OS-specific requirements such as this. Perhaps is entire section should be moved out of Advanced Options / Configuration and into Requirements.
• I'd consider copying the nm-cloud-setup check in the systemctl script into the get.k3s.io install script as well, perhaps along with checks for ufw and firewalld.

EDIT: Oh, one more add: I discovered the SELinux enforcement with the default k3s-selinux ruleset is not compatible with local-path-provisoner; that should probably be called out as well. See here: rancher/local-path-provisioner#362

Thank you again!

Confusing documentation on secrets encryption.

I can't quite figure out this page:

https://docs.k3s.io/cli/secrets-encrypt

I'm trying to figure out how to enable secrets encryption on my cluster. I didn't do anything special to turn it on, so I presume it is off.

Under 'Encryption key rotation', it says "Starting K3s without encryption and enabling it at a later time is currently not supported." Does this mean that it is impossible to enable secrets encryption on a cluster that didn't have it on from the start? Or does it mean that you have to restart the k3s server in order to enable secrets encryption?

There is also a section called "Secrets Encryption Disable/Enable" - but this seems to presuppose that the --secrets-encrypt flag is already set, I've disabled secrets encryption and I now want to re-enable it.

Example Traefik chart config is out of date

When creating the traefik-config.yaml in /var/lib/rancher/k3s/server/manifests/traefik-config.yaml

kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    image:
      name: traefik
      tag: v2.8.5
    forwardedHeaders:
      enabled: true
      trustedIPs:
        - 10.0.0.0/8
    ssl:
      enabled: true
      permanentRedirect: false

official guide needs to be updated as I get an image pull error and back off.

There must be an overlooked step that is missing from that guide.

Originally posted by @gavan1 in k3s-io/k3s#1313 (comment)

Document flag deprecation policy

We wrote up a deprecation policy (defined here: https://github.com/k3s-io/k3s/blob/master/docs/adrs/deprecating-and-removing-flags.md) that's been discussed and agreed upon within the team -- let's write up a page on the docs and make sure it's publicly stated.

Release notes are 2 months out of date: PR bot broken?

The release notes on the docs page are about 2 months out of date vs actual releases

For example it's missing 1.30 entirely: https://github.com/k3s-io/docs/tree/main/docs/release-notes

https://docs.k3s.io/release-notes/v1.29.X

A bot used to make PRs, maybe it's broken?

See last PR: #249

[Docs] Update architecture requirement docs to reflect that 4k is no longer a requirement

This should be noted that it is as of May 2023 patch releases per k3s-io/k3s#7335.

Deploying the Kubernetes Dashboard fails on v3 due to a breaking change

Environmental Info:
K3s Version:
v1.27.4+k3s1

Node(s) CPU architecture, OS, and Version:
N/A

Cluster Configuration:
N/A

Describe the bug:
As of kubernetes-dashboard v3.0.0-alpha0 there is a breaking change that causes the instructions on Kubernetes Dashboard to fail with the following error:

error: resource mapping not found for name: "selfsigned" namespace: "kubernetes-dashboard" from "https://raw.githubusercontent.com/kubernetes/dashboard/v3.0.0-alpha0/charts/kubernetes-dashboard.yaml": no matches for kind "Issuer" in version "cert-manager.io/v1"
ensure CRDs are installed first

See v3.0.0-alpha0 quoted below:

Starting from the release v3 of the Kubernetes Dashboard, the underlying architecture has changed, and it requires a clean installation. Please remove the previous installation first.
Kubernetes Dashboard now uses cert-manager and nginx-ingress-controller by default to work properly. Please make sure you have them installed in your cluster if you want to use a manifest-based installation path. The helm-based approach can install all required dependencies automatically for you if needed.

Expected Results
Update the documentation to add the required setups and configuration to deploy the latest version of the kubernetes-dashboard.

Workaround:
Deploy v2.7.0 of the kubernetes-dashboard.

Documentation doesn't give details on external database setup and permissions

When I read the existing documentation at https://docs.k3s.io/datastore about setting up a High Availability External DB there isn't any detail about how the database should be created and what permissions the user provided to k3s needs to have. I would suggest the Drupal database setup page as an excellent example of the details that should go into this documentation for both MySQL and PostgreSQL: https://www.drupal.org/docs/getting-started/installing-drupal/create-a-database#create-a-database-using-mysqlmariadb-commands

The requirements for k3s might actually be the exact same as for Drupal but I don't know that. Details like collation order and character set should be provided and an example database creation command are very helpful. Similarly providing a list of the specific permissions required for the database userid are also needed so the user doesn't end up assigning all permissions which has security implications.

Using the Drupal docs as a start an making any necessary modifications would be a great improvement to the k3s docs on the topic.