Snapshot retention does not work with etcd-s3-folder about k3s

[reefland]

Was just going to open an issue on this...

$ sudo k3s etcd-snapshot ls | grep -c "s3://"
463

Even if I force prune, doesn't help:

$ sudo k3s etcd-snapshot prune --etcd-snapshot-retention 10

$ sudo k3s etcd-snapshot ls | grep -c "s3://"
463

$ sudo k3s --version
k3s version v1.29.5+k3s1 (4e53a323)
go version go1.21.9

[brandond]

@reefland on-demand and scheduled snapshots have different base names, and pruning on-demand snapshots does not affect the scheduled ones. You'll need to specify the correct snapshot name to prune scheduled snapshots:
sudo k3s etcd-snapshot prune --etcd-snapshot-retention 10 --etcd-snapshot-name etcd-snapshot

If you've set a different default name for scheduled snapshots via the server's etcd-snapshot-name cli flag or config file key, use that instead of etcd-snapshot. The default name for on-demand snapshots is on-demand, and these are the only ones that are pruned by default via the CLI.

[reefland]

I'm using the default snapshot names:

Name                           Location                                                                       Size      Created
etcd-snapshot-k3s01-1714233603 s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1714233603                         87822368  2024-04-27T12:00:03-04:00
etcd-snapshot-k3s02-1714255201 s3://k3s/etcd-snapshots/etcd-snapshot-k3s02-1714255201                         84918304  2024-04-27T18:00:01-04:00
etcd-snapshot-k3s01-1714255202 s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1714255202                         88690720  2024-04-27T18:00:02-04:00
...

$ sudo k3s etcd-snapshot ls | grep -c "s3://k3s/etcd-snapshots/etcd-snapshot-"
466

Tried to use your suggestion with --etcd-snapshot-name but it didn't like it:

$ sudo k3s etcd-snapshot prune --etcd-snapshot-retention 10 --etcd-snapshot-name etcd-snapshot

Incorrect Usage: flag provided but not defined: -etcd-snapshot-name
...
FATA[0000] flag provided but not defined: -etcd-snapshot-name

But using --name etcd-snapshot instead seems to work, long list scrolled by:

INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1717538403 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s03-1717538402 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1717516805 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s02-1717516804 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s03-1717516802 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1717495205 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s02-1717495204 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s03-1717495201 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1717473604 deleted.
....
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1714276802 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s03-1714255204 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1714255202 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s02-1714255201 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1714233603 deleted.

However, the count did not drop:

$ sudo k3s etcd-snapshot ls | grep -c "s3://k3s/etcd-snapshots/etcd-snapshot-"

466

If I search for the 1st one it said was deleted, still shows up:

$ sudo k3s etcd-snapshot ls |grep "k3s01-1717538403"

etcd-snapshot-k3s01-1717538403 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-k3s01-1717538403 96759840  2024-06-04T18:00:03-04:00
etcd-snapshot-k3s01-1717538403 s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1717538403                         96759840  2024-06-04T18:00:03-04:00

If I run the prune command again, gives me the same long list. And the k3s logs don't show any errors...

Jun  5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Checking if S3 bucket k3s exists"
Jun  5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="S3 bucket k3s exists"
Jun  5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Applying snapshot retention=10 to local snapshots with prefix etcd-snapshot in /var/lib/rancher/k3s/server/db/snapshots"
Jun  5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Applying snapshot retention=10 to snapshots stored in s3://k3s/etcd-snapshots/etcd-snapshot"
Jun  5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1717538403"
Jun  5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s03-1717538402"
Jun  5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1717516805"
...
Jun  5 12:48:51 k3s01 k3s[128858]: time="2024-06-05T12:48:51-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s03-1714255204"
Jun  5 12:48:51 k3s01 k3s[128858]: time="2024-06-05T12:48:51-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1714255202"
Jun  5 12:48:51 k3s01 k3s[128858]: time="2024-06-05T12:48:51-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s02-1714255201"
Jun  5 12:48:51 k3s01 k3s[128858]: time="2024-06-05T12:48:51-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1714233603"
Jun  5 12:48:51 k3s01 k3s[128858]: time="2024-06-05T12:48:51-04:00" level=info msg="Reconciling ETCDSnapshotFile resources"
Jun  5 12:48:52 k3s01 k3s[128858]: time="2024-06-05T12:48:52-04:00" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

Suggestion?

[brandond]

Incorrect Usage: flag provided but not defined: -etcd-snapshot-name

Hmm, that's a bug. The other commands have aliases, I'll have to see why we missed it for that flag

   --name value                                                 (db) Set the base name of the etcd on-demand snapshot (appended with UNIX timestamp). (default: "on-demand")
   --snapshot-compress, --etcd-snapshot-compress                (db) Compress etcd snapshot
   --snapshot-retention value, --etcd-snapshot-retention value  (db) Number of snapshots to retain. (default: 5)
   --s3, --etcd-s3                                              (db) Enable backup to S3

What do you have in the config.yaml? I am assuming that the rest of the etcd-s3 config options are in there; does that include etcd-s3-folder?

[reefland]

etcd-s3: true
snapshot-compress: true
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 10
etcd-s3-bucket: "k3s"
etcd-s3-access-key: "<redacted>"
etcd-s3-secret-key: "<redacted>"
etcd-s3-endpoint: "truenas.<redacted>:9000"
etcd-s3-region: "minio"
etcd-s3-folder: "etcd-snapshots"

BTW - The docs https://docs.k3s.io/cli/etcd-snapshot reference --etcd-snapshot-schedule-cron but the k3s command gives me a warning:

WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping

Not sure why it has a problem with the entry above.

[brandond]

WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping

Yeah, that flag is only valid on the server command-line to configure the snapshot schedule. It is not valid to pass that on the etcd-snapshot command line because it is inherently a one-shot option, not scheduled via cron. The warning on the CLI is not great but is an artifact of how the config parser converts the config file to CLI args.

Look at this a little more closely, I guess even on-demand prune is broken :/ The best you can probably do at the moment is to manually run etcd-snapshot delete with a list of snapshots to delete from S3. That should do the correct thing.