[Release-1.28] - Snapshot retention does not work with etcd-s3-folder about k3s

[aganesh-suse]

Validated on release-1.28 branch with version v1.28.11-rc3+k3s1

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent

Config.yaml:

token: xxxx
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server

etcd-snapshot-retention: 2
etcd-snapshot-schedule-cron: "* * * * *"
etcd-s3: true
etcd-s3-access-key: <access-key>
etcd-s3-secret-key: <secret-key>
etcd-s3-bucket: <s3-bucket>
etcd-s3-folder: <s3-folder>
etcd-s3-region: <region>

debug: true

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.28.11-rc3+k3s1' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

4. Perform:

$ sudo k3s etcd-snapshot prune
$ sudo k3s etcd-snapshot list 

pruned snapshot should not be listed after being deleted.

Replication Results:

• k3s version used for replication:

$ k3s -v
k3s version v1.28.10+k3s1 (a4c5612e)
go version go1.21.9

 $ sudo /usr/local/bin/k3s etcd-snapshot prune 
time="2024-06-21T16:57:05Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2024-06-21T16:57:05Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2024-06-21T16:57:05Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2024-06-21T16:57:05Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2024-06-21T16:57:05Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2024-06-21T16:57:05Z" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation."
time="2024-06-21T16:57:06Z" level=info msg="Snapshot on-demand-ip-172-31-30-182-1718988908 deleted."
time="2024-06-21T16:57:06Z" level=info msg="Snapshot <s3-folder>/on-demand-ip-172-31-30-182-1718988908 deleted."

 $ sudo /usr/local/bin/k3s etcd-snapshot list 2>/dev/null On SERVER1: 18.188.91.198
Name                                      Location                                                                                         Size    Created
etcd-snapshot-ip-172-31-30-182-1718988421 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718988421 1224736 2024-06-21T16:47:01Z
etcd-snapshot-ip-172-31-30-182-1718988481 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718988481 3276832 2024-06-21T16:48:01Z
etcd-snapshot-ip-172-31-30-182-1718988543 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718988543 3772448 2024-06-21T16:49:03Z
etcd-snapshot-ip-172-31-30-182-1718988601 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718988601 4104224 2024-06-21T16:50:01Z
etcd-snapshot-ip-172-31-30-182-1718988664 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718988664 4272160 2024-06-21T16:51:04Z
etcd-snapshot-ip-172-31-30-182-1718988724 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718988724 4505632 2024-06-21T16:52:04Z
etcd-snapshot-ip-172-31-30-182-1718988783 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718988783 5267488 2024-06-21T16:53:03Z
etcd-snapshot-ip-172-31-30-182-1718988841 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718988841 5443616 2024-06-21T16:54:01Z
etcd-snapshot-ip-172-31-30-182-1718988903 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718988903 5623840 2024-06-21T16:55:03Z
on-demand-ip-172-31-30-182-1718988908     s3://<s3-bucket>/<s3-folder>/on-demand-ip-172-31-30-182-1718988908     5652512 2024-06-21T16:55:08Z
on-demand-ip-172-31-30-182-1718988916     s3://<s3-bucket>/<s3-folder>/on-demand-ip-172-31-30-182-1718988916     5705760 2024-06-21T16:55:16Z
on-demand-ip-172-31-30-182-1718988916     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-172-31-30-182-1718988916            5705760 2024-06-21T16:55:16Z
on-demand-ip-172-31-30-182-1718988925     s3://<s3-bucket>/<s3-folder>/on-demand-ip-172-31-30-182-1718988925     5738528 2024-06-21T16:55:25Z
on-demand-ip-172-31-30-182-1718988925     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-172-31-30-182-1718988925            5738528 2024-06-21T16:55:25Z
etcd-snapshot-ip-172-31-30-182-1718988961 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718988961 5836832 2024-06-21T16:56:01Z
etcd-snapshot-ip-172-31-30-182-1718988961 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-172-31-30-182-1718988961        5836832 2024-06-21T16:56:01Z
etcd-snapshot-ip-172-31-30-182-1718989021 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718989021 6656032 2024-06-21T16:57:01Z
etcd-snapshot-ip-172-31-30-182-1718989021 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-172-31-30-182-1718989021        6656032 2024-06-21T16:57:01Z

on-demand-ip-172-31-30-182-1718988908 is still listed above - after it was supposed to be deleted.

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.28.11-rc3+k3s1 (617b0e84)
go version go1.21.11

 $ sudo /usr/local/bin/k3s etcd-snapshot prune 
time="2024-06-21T16:57:05Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2024-06-21T16:57:05Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2024-06-21T16:57:05Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2024-06-21T16:57:05Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2024-06-21T16:57:05Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2024-06-21T16:57:05Z" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation."
time="2024-06-21T16:57:05Z" level=info msg="Snapshot on-demand-ip-172-31-21-205-1718988904 deleted."
time="2024-06-21T16:57:05Z" level=info msg="Snapshot on-demand-ip-172-31-21-205-1718988904 deleted."

$ sudo /usr/local/bin/k3s etcd-snapshot list
Name                                      Location                                                                                            Size    Created
on-demand-ip-172-31-21-205-1718988915     s3://<s3-bucket>/<s3-folder>/on-demand-ip-172-31-21-205-1718988915     5759008 2024-06-21T16:55:15Z
on-demand-ip-172-31-21-205-1718988915     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-172-31-21-205-1718988915               5759008 2024-06-21T16:55:15Z
on-demand-ip-172-31-21-205-1718988925     s3://<s3-bucket>/<s3-folder>/on-demand-ip-172-31-21-205-1718988925     5808160 2024-06-21T16:55:25Z
on-demand-ip-172-31-21-205-1718988925     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-172-31-21-205-1718988925               5808160 2024-06-21T16:55:25Z
etcd-snapshot-ip-172-31-21-205-1718988964 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-21-205-1718988964 6852640 2024-06-21T16:56:04Z
etcd-snapshot-ip-172-31-21-205-1718988964 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-172-31-21-205-1718988964           6852640 2024-06-21T16:56:04Z
etcd-snapshot-ip-172-31-21-205-1718989020 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-21-205-1718989020 6852640 2024-06-21T16:57:00Z
etcd-snapshot-ip-172-31-21-205-1718989020 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-172-31-21-205-1718989020           6852640 2024-06-21T16:57:00Z

on-demand-ip-172-31-21-205-1718988904 is not on the list post delete.