[Backport release-1.26] etcd snapshot cleanup fails if node name changes

[rancherbot]

This is a backport issue for #3714, automatically created via rancherbot by @vitorsavian

Original issue description:

Environmental Info:

RKE2 Version:
rke2 version v1.21.14+rke2r1 (514ae51)
go version go1.16.14b7

Node(s) CPU architecture, OS, and Version:

Linux testmachine 5.15.70-flatcar rancher/rancher#1 SMP Thu Oct 27 12:53:14 -00 2022 x86_64 Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz GenuineIntel GNU/Linux

Cluster Configuration:
We have multiple rke2 clusters, but all of them have at least 3 control plane nodes and multiple workers

Describe the bug:
We have multiple rke2 clusters and all of them have automatic etcd snapshots enabled (taken every 5 hours). We also configured s3 uploading of those snapshots. Recently, we found that no s3 snapshots are uploaded anymore. We investigated the issue and found the following rke2-server output:

Dec 14 05:27:01 testmachine[1665]: time="2022-12-14T05:27:01Z" level=error msg="failed to save local snapshot data to configmap: ConfigMap \"rke2-etcd-snapshots\" is invalid: []: Too long: must have at most 1048576 bytes"

I checked the code and found that rke2 is leveraging the etcd snapshot capabilities from k3s for this. A function is executed periodically on all control plane nodes. The function takes local snapshots, uploads them to s3 (if configured) and also reconciles a configmap which contains all snapshots and metadata about them. Looking at the code it seems that the reconcilation of that "sync" configmap is based on the name of the node which executes the etcd snapshot. Same goes for the s3 retention functions (only old objects which contain the node name will be cleaned up). As we are replacing all our nodes in the clusters whenever there is a new flatcar version, the node names change quite often. This leads to orphaned entries in the config map and also orphaned objects in the s3 buckets (although this could be worked around with a lifecycle policy).

Are there any ideas what could be done to fix this?

I found this bug report which describes the too large configmap in the rancher repo.

Steps To Reproduce:

Enable etcd snapshots and s3 uploading. After replacing the control plane nodes with new machines (new names), there will be orphaned entries in the 'rke2-etcd-snapshots' configmap. Whenever the configmap grew too large, no new snapshots will be uploaded to s3 anymore.

Expected behavior:
The sync configmap only contains the snapshots of the current nodes of the clusters and removes all other ones.

[aganesh-suse]

Validated on release-1.26 branch with commit e181a2d

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

cat /etc/os-release | grep PRETTY
PRETTY_NAME="Ubuntu 22.04.2 LTS"

Cluster Configuration:

Server config: 3 etcd, control planes servers/1 agent config

Config.yaml:

Main ETCD SERVER (+CONTROL PLANE) CONFIG:

token: blah
node-name: "server1"
etcd-snapshot-retention: 2
etcd-snapshot-schedule-cron: "* * * * *"
etcd-s3: true
etcd-s3-access-key: xxx
etcd-s3-secret-key: xxx
etcd-s3-bucket: s3-bucket-name
etcd-s3-folder: rke2snap/commit-setup
etcd-s3-region: us-east-2
write-kubeconfig-mode: "0644"

Sample Secondary Etcd, control plane config.yaml:

token: blah
server: https://x.x.x.x:9345
node-name: "server3"
write-kubeconfig-mode: "0644"

AGENT CONFIG:

token: blah
server: https://x.x.x.x:9345
node-name: "agent1"

Testing Steps

1. Create config dir and place the config.yaml file in server/agent nodes:

$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2

Note: First round node-names:
<version|commit>-server1
server2
server3
agent1
2. Install RKE2:
Validate issue Using Commit:

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_COMMIT='e181a2dc8b691dc337f06b79de8745945061eaa6' INSTALL_RKE2_TYPE='server' INSTALL_RKE2_METHOD=tar sh -

Reproduce issue Using Version:

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION='v1.26.7+rke2r1' INSTALL_RKE2_TYPE='server' INSTALL_RKE2_METHOD=tar sh -

3. Wait for 2 minutes.
   Note: The snapshot gets created every 1 minute (etcd-snapshot-schedule-cron: "* * * * *") . Retention is for 2 snapshots (etcd-snapshot-retention: 2).
   Reference for cron job format: https://cloud.google.com/scheduler/docs/configuring/cron-job-schedules
   After 2 minutes: 2 snapshots are created with name etcd-snapshot-server1-2-xxxx if node-name: server1-2 in config.yaml),
4. Check outputs of:

sudo ls -lrt /var/lib/rancher/rke2/server/db/snapshots

sudo rke2 etcd-snapshots list

4a. Also check the s3 bucket/folder in aws to see the snapshots listed.
5. Update the node-name in the config.yaml:
node-names:
<version|commit>-server1-<|suffix1>
server2-<|suffix1>
server3-<|suffix1>
agent1-<|suffix1>
6. restart the rke2 service for all nodes.

sudo systemctl restart rke2-server

7. Wait for 2 more minutes and check the snapshot list:

sudo ls -lrt /var/lib/rancher/rke2/server/db/snapshots

sudo rke2 etcd-snapshots list

7a. Also check the s3 bucket/folder in aws to see the snapshots listed.

8. Repeat steps 5 through 7 once more.
   node names:
   <version|commit>-server1-<|suffix2>-<|suffix1>
   server2-<|suffix2>-<|suffix1>
   server3-<|suffix2>-<|suffix1>
   agent1-<|suffix2>-<|suffix1>

Replication Results:

• rke2 version used for replication:

$ rke2 -v 
rke2 version v1.26.7+rke2r1 (374bdaf23d963e7d16bcaa820e95f792b1d3c661)
go version go1.20.6 X:boringcrypto

Node names in order of update for the main etcd server:

version-setup-server1
version-setup-server1-27169            
version-setup-server1-17-27169   

Final list of snapshots - after multiple node name changes:

$ sudo ls -lrt /var/lib/rancher/rke2/server/db/snapshots 
total 78664
-rw------- 1 root root  9838624 Aug 11 23:01 etcd-snapshot-version-setup-server1-1691794860
-rw------- 1 root root 10076192 Aug 11 23:02 etcd-snapshot-version-setup-server1-1691794922
-rw------- 1 root root 12566560 Aug 11 23:05 etcd-snapshot-version-setup-server1-27169-1691795104
-rw------- 1 root root 13660192 Aug 11 23:06 etcd-snapshot-version-setup-server1-27169-1691795161
-rw------- 1 root root 16470048 Aug 11 23:08 etcd-snapshot-version-setup-server1-17-27169-1691795284
-rw------- 1 root root 17915936 Aug 11 23:09 etcd-snapshot-version-setup-server1-17-27169-1691795340
Execute: 
$ sudo rke2 etcd-snapshot list 
time="2023-08-11T23:09:37Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-08-11T23:09:37Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-08-11T23:09:37Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-08-11T23:09:37Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-08-11T23:09:37Z" level=info msg="Checking if S3 bucket xxx exists"
time="2023-08-11T23:09:38Z" level=info msg="S3 bucket xxx exists"
Name                                                    Size     Created
etcd-snapshot-version-setup-server1-1691794922          10076192 2023-08-11T23:02:03Z
etcd-snapshot-version-setup-server1-17-27169-1691795284 16470048 2023-08-11T23:08:06Z
etcd-snapshot-version-setup-server1-17-27169-1691795340 17915936 2023-08-11T23:09:01Z
etcd-snapshot-version-setup-server1-27169-1691795104    12566560 2023-08-11T23:05:06Z
etcd-snapshot-version-setup-server1-27169-1691795161    13660192 2023-08-11T23:06:03Z
etcd-snapshot-version-setup-server1-1691794860          9838624  2023-08-11T23:01:01Z

As we can see above, previous snapshots with different node-names are still listed and not cleaned up.

Validation Results:

• rke2 version used for validation:

$ rke2 -v
rke2 version v1.26.7+dev.e181a2dc (e181a2dc8b691dc337f06b79de8745945061eaa6)
go version go1.20.6 X:boringcrypto

Node Names in the order of update for the main etcd server:
commit-setup-server1              
commit-setup-server1-1158         
commit-setup-server1-25917-1158 

After updating node-names 2 times, the snapshots listed are:

$ sudo ls -lrt /var/lib/rancher/rke2/server/db/snapshots 
total 29092
-rw------- 1 root root 14282784 Aug 16 23:18 etcd-snapshot-commit-setup-server1-25917-1158-1692227882
-rw------- 1 root root 15499296 Aug 16 23:19 etcd-snapshot-commit-setup-server1-25917-1158-1692227944

$ sudo rke2 etcd-snapshot list 
time="2023-08-16T23:19:31Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-08-16T23:19:31Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-08-16T23:19:31Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-08-16T23:19:31Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-08-16T23:19:31Z" level=warning msg="Unknown flag --cni found in config.yaml, skipping\n"
time="2023-08-16T23:19:31Z" level=info msg="Checking if S3 bucket xxx exists"
time="2023-08-16T23:19:31Z" level=info msg="S3 bucket xxx exists"
Name                                                     Size     Created
etcd-snapshot-commit-setup-server1-25917-1158-1692227882 14282784 2023-08-16T23:18:04Z
etcd-snapshot-commit-setup-server1-25917-1158-1692227944 15499296 2023-08-16T23:19:06Z

As we can see, the previous snapshots with old node-names are no longer retained and get cleaned up.

Additional context / logs: