paxa / kube-backup

Kubernetes resource yamls backup to git

Kube-backup

Kubernetes resource state backup to git

Git structure

_global_ - global resources such as Node, ClusterRole, StorageClass
_grafana_ - grafana configs (when grafana enabled)
<namespace> - such as kube-system, default, etc...
  <ResourceType> - folder for each resource type
    <resource-name.yaml> - file for each resource

Screenshots

Deployment

Yaml manifests are in deploy folder.

Create Deployment Key

Github and gitlab support adding key only for one repository

• Create repo
• Generate ssh key ssh-keygen -f ./new_key
• Add new ssh key to repo with write access
• Save key to 2_config_map.yaml (see comments in file)

Testing Deployment

I recommend to run it periodically with kubernetes' CronJob resource, if you want to test how it works without waiting then can change running schedule or create pod with same parameters

Commands

• kube_backup backup - pull remote git repository, save kubernetes state, make git commit in local repository
• kube_backup push - push changes to remote repository
• kube_backup help - shows help

Docker image by default runs kube_backup backup && kube_backup push

Config

• GIT_REPO_URL - remote git URL like [email protected]:kuberhost/kube-backup.git (required)
• BACKUP_VERBOSE use 1 to enable verbose logging
• TARGET_PATH - local git repository folder, default ./kube_state
• SKIP_NAMESPACES - namespaces to exclude, separated by coma (,)
• ONLY_NAMESPACES - whitelist namespaces
• GLOBAL_RESOURCES - override global resources list, default is node, apiservice, clusterrole, clusterrolebinding, podsecuritypolicy, storageclass, persistentvolume, customresourcedefinition, mutatingwebhookconfiguration, validatingwebhookconfiguration, priorityclass
• EXTRA_GLOBAL_RESOURCES - use it to add resources to GLOBAL_RESOURCES list
• SKIP_GLOBAL_RESOURCES - blacklist global resources
• RESOURCES - default list of namespaces resources, see KubeBackup::TYPES
• EXTRA_RESOURCES - use it to add resources to RESOURCES list
• SKIP_RESOURCES - exclude resources
• SKIP_OBJECTS - use it to skip individual objects, such as kube-backup/ConfigMap/kube-backup-ssh-config (separated by coma, spaces around coma ignored)
• GIT_USER - default is kube-backup
• GIT_EMAIL - default is kube-backup@$(HOSTNAME)
• GIT_BRANCH - Git branch, default is master
• GIT_PREFIX - Path to the subdirectory in your repository
• GRAFANA_URL - grafana api URL, e.g. https://grafana.my-cluster.com
• GRAFANA_TOKEN - grafana API token, create at https://your-grafana/org/apikeys
• TZ - timezone of commit times. e.g. :Europe/Berlin

Security

To avoid man in a middle attack it's recommended to provide known_hosts file. Default known_hosts contain keys for github.com, gitlab.com and bitbucket.org

Custom Resources

Let's say we have a cluster with prometheus and certmanager, they register custom resources and we want to add them in backup.

Get list of custom resource definitions:

$ kubectl get crd

NAME                                    CREATED AT
alertmanagers.monitoring.coreos.com     2018-06-27T10:33:00Z
certificates.certmanager.k8s.io         2018-06-27T09:39:43Z
clusterissuers.certmanager.k8s.io       2018-06-27T09:39:43Z
issuers.certmanager.k8s.io              2018-06-27T09:39:44Z
prometheuses.monitoring.coreos.com      2018-06-27T10:33:00Z
prometheusrules.monitoring.coreos.com   2018-06-27T10:33:00Z
servicemonitors.monitoring.coreos.com   2018-06-27T10:33:00Z

Or get more useful output:

$ kubectl get crd -o json | jq -r '.items | (.[] | [.spec.names.singular, .spec.group, .spec.scope]) | @tsv'
alertmanager    monitoring.coreos.com  Namespaced
certificate     certmanager.k8s.io     Namespaced
clusterissuer   certmanager.k8s.io     Cluster
issuer          certmanager.k8s.io     Namespaced
prometheus      monitoring.coreos.com  Namespaced
prometheusrule  monitoring.coreos.com  Namespaced
servicemonitor  monitoring.coreos.com  Namespaced

Set env variables in container spec:

env:
  - name: EXTRA_GLOBAL_RESOURCES
    value: clusterissuer
  - name: EXTRA_RESOURCES
    value: alertmanager, prometheus, prometheusrule, servicemonitor, certificate, issuer

---

Special thanks to Pieter Lange for original idea

kube-backup's Issues

Writer.get_changes Nests GIT_PREFIX Twice

When running kube-backup push with GIT_PREFIX set, Writer.get_changes attempts to find changes under ./kube_state/$GIT_PREFIX/$GIT_PREFIX and thus finds nothing.

Not working when git status has double quotes

Hello, everyone.

This is a case with some whitespace in k8s object name:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: "a b c rolebiding"
  namespace: ddd
...

It’s ok at first(and commit message is fine).

git status --porcelain "." --untracked-files=all
?? ddd/RoleBinding/a b c rolebinding.yaml

However when update or delete this object, added double quotes to lines.

M "ddd/RoleBinding/a b c rolebinding.yaml"
or
D "ddd/RoleBinding/a b c rolebinding.yaml"

Error occurred in parsing. The problem is "ddd.

05:56:31.549: RUN git commit -m "Updated PersistentVolume, RoleBinding, Deployment, HorizontalPodAutoscaler, ConfigMap, Ingress, NetworkPolicy, Pod, in namespaces aaa, bbb, ccc, "ddd, eee. 206 items"

error: sh: syntax error: unterminated quoted string

Of course, this is not a common case, but when I found this case at work 😱

In my opinion, add some parsing logic to here:

# KubeBackup::push_changes!
changes_lines.each do |line|
  info = line.strip.match(/^(?<prefix>.+?)\s+(?<file>.+)$/)
  info["file"].sub!(prefix, '') if prefix
  file_parts = info["file"].sub(/\.yaml$/, '').split("/")
  if file_parts[0] != "_global_"
    namespaces << file_parts[0]
  end
  resources << file_parts[1]
end

Thanks!

process 10 exit with code 1

# kubectl -n kube-backup logs kube-system-backup-1563803760-fr68x 
14:01:52.411: Args: {target_path: "./kube_state", repo_url: "ssh://[email protected]:32222/mbr/k8-repo-test.git", extra_resources: "certificate,issuer,alertmanager,prometheus,prometheusrule,servicemonitor", verbose: true}
14:01:52.411: Types: [:serviceaccount, :secret, :deployment, :daemonset, :statefulset, :configmap, :cronjob, :ingress, :networkpolicy, :persistentvolumeclaim, :role, :rolebinding, :service, :pod, :endpoints, :resourcequota, :horizontalpodautoscaler, :certificate, :issuer, :alertmanager, :prometheus, :prometheusrule, :servicemonitor]
14:01:52.411: RUN ssh-keyscan -H git.example.org >> /root/.ssh/known_hosts
error: . Use --trace to view backtrace
14:01:57.480: Process 10 exit with code 1

14:01:57.480: 

1. What's the problem here?
2. Would it be possible to provide a better output (what is process 10) here?

Thanks.

# kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.8", GitCommit:"0c6d31a99f81476dfc9871ba3cf3f597bec29b58", GitTreeState:"clean", BuildDate:"2019-07-08T08:46:01Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.8", GitCommit:"0c6d31a99f81476dfc9871ba3cf3f597bec29b58", GitTreeState:"clean", BuildDate:"2019-07-08T08:38:54Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Error Creating Service Account

I get an error when applying 1_service_account.yaml on k8s 1.24

$ kubectl apply -f 1_service_account.yaml 
serviceaccount/kube-backup-user created
clusterrole.rbac.authorization.k8s.io/kube-backup-view-all created
rolebinding.rbac.authorization.k8s.io/psp:unprivileged created
error: resource mapping not found for name: "kube-backup-user" namespace: "" from "1_service_account.yaml": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
ensure CRDs are installed first

Writer.remove_repo_content Inconsistency with GIT_PREFIX

The Writer.remove_repo_content method uses GIT_PREFIX to search for files to remove, but does not add GIT_PREFIX to the path when building the list of files to remove. The job fails with the error: error: No such file or directory @ apply2files - ./kube_state/_global_. Use --trace to view backtrace

Trust SSH host on first use enabled by default

Its fairly easy to execute a MiTM attack on deployments of this project because by default there's no preseeded SSH known_hosts file. This is bad.

Instead, the known_hosts file gets seeded with whatever the public key of the host is at runtime. Because of the ephemeral nature of containers, this basically means this happens on every run.
https://github.com/kuberhost/kube-backup/blob/d27f3956bfe38c037d8714f4a8d052f3e64bc9f8/lib/kube_backup/writter.rb#L162

If you want to improve UX, you could preseed some SSH keys from EG github.com, bitbucket etc. but as-is this configuration is pretty unsafe.

Could you explain the difference with the other repository?

I would like to know what is the difference with the other repository. Perhaps you could add a paragraph to the readme?

https://github.com/pieterlange/kube-backup

Screenshots

Use subdir within repo

It would be great if there were an ENV variable like GIT_SUBDIR I could set to the clustername and all files would be created under this directory. This would allow me to backup multiple clusters to one git repo

Not working when Grafana is not available

Backup not working when Grafana plugin is enable and Grafana is not available,
e.g. 503 Service Temporarily Unavailable

In my opinion, add some skip grafana plugin logic when grafana is not available in run method of grafana.rb.

# grafana.rb
# TODO: add check grafana status
if !@grafana_url || @grafana_url == ''
  KubeBackup.logger.info "Skip Grafana plugin"
  return
end