s3 snapshots do not show in the rancher 2.6 UI

[davidnuzik]

This is duplicated from

• s3 snapshots do not show in the UI dashboard#3693

Rancher Server Setup

• Rancher version: v2.6-head 246fbd5fa 8/5/21 (single-node docker)
• Installation option (Docker install/Helm Chart): Docker 19.03

Downstream cluster: rke2 provisioned by rancher, 3 etcd, 2 cp, 3 worker
Downstream k8s version: v1.21.3-rc3+rke2r2

Describe the bug

1. Create a cluster. Submit valid s3 information.
2. Take a snapshot. This appears to be successful.

2021/08/05 22:35:57 [INFO] rkecluster fleet-default/dave-rke2-s3test: refreshing etcd create state
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: starting etcd snapshot
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: refreshing etcd create state
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: provisioning bootstrap node(s) dave-rke2-s3test-e-59bc958d8-bvkrg
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: provisioning bootstrap node(s) dave-rke2-s3test-e-59bc958d8-bvkrg
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: provisioning bootstrap node(s) dave-rke2-s3test-e-59bc958d8-bvkrg: waiting for plan to be applied
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: provisioning bootstrap node(s) dave-rke2-s3test-e-59bc958d8-bvkrg: waiting for plan to be applied
2021/08/05 22:36:22 [INFO] rkecluster fleet-default/dave-rke2-s3test: provisioning bootstrap node(s) dave-rke2-s3test-e-59bc958d8-bvkrg: waiting for plan to be applied

3. Inspect the s3 bucket - everything is there it looks like the snapshot was successful and I can retake snapshots and they will work and upload to my bucket.
   

4. Inspect the snapshots tab - none of the snapshots show.
   

Note: Refreshing the UI does not fix the issue. Cleaning browser cache and launching the UI again also does not fix.

Result
The UI does not show any of the (apparently) successfully-taken s3 snapshots.

Expected Result

The UI should show the s3 snapshots, just like it does for local snapshots.

Additional Info
Blocks testing/closure of rancher/rancher#32391

In the rancher/rancher issue, Donnie mentioned the following:
Rancher looks at the configmap rke2-etcd-snapshots in the downstream RKE2 cluster to get the information for the snapshots. In this case, the configmap exists, but it is empty.

> kubectl -n kube-system get configmap rke2-etcd-snapshots -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: "2021-08-05T22:30:30Z"
  name: rke2-etcd-snapshots
  namespace: kube-system
  resourceVersion: "378"
  uid: 118aca17-9b4d-428b-af38-32043e65e8ac

[davidnuzik]

I owe providing rke2 server logs - will work with dev to figure out how to get ssh access to rancher-provisioned rke2 nodes. This is not possible in the Rancher 2.6 UI for some reason (you can do it in rke1 but not rke2).

[davidnuzik]

Worked with Brian on a call and we observed rke2 logs - Brian should have the information he needs now to get going on this (e.g. I will not be posting rke2 logs here in GitHub as should not be needed - if anyone wants them posted here just let me know).

[davidnuzik]

rancher/rancher#34088 is changing how s3 works so waiting on that now before I test in Rancher.

[davidnuzik]

So, Rancher 2.6.0 does not support 1.22 yet - so I tested in standalone RKE2 only.
Once we have a 2.6.1 or 2.6.2 I imagine we'll likely add support for 1.22 Kubernetes, at which point, QA should check this anyway with normal test plan checks.

My checks PASSED

Reproduction Steps:

Not required as I fully understand the issue.

Validation Steps:

Host OS: Ubuntu 20.04 amd64
RKE2 version: from commitId f377e3eedca2e9037b173981af6cccb5821b072b (latest I found today 8/18/21 on master branch)
Tested with a 1-node rke2 cluster.

Validation steps:

1. Create /etc/rancher/rke2/config.yaml with the following content. Take note we specify an s3 folder (which is what we are focusing our testing on here as per the code change).

"write-kubeconfig-mode": "0644"
"etcd-s3": true
"etcd-s3-access-key": "REDACTED_ACCESS_KEY"
"etcd-s3-bucket": "dave-secret-bucket"
"etcd-s3-endpoint": "s3.us-east-2.amazonaws.com"
"etcd-s3-folder": "dave-rke2v122"
"etcd-s3-region": "us-east-2"
"etcd-s3-secret-key": "REDACTED_SECRET_KEY"
"etcd-s3-skip-ssl-verify": true
"etcd-snapshot-retention": 2
"etcd-snapshot-schedule-cron": "8 * * * *"

2. Install RKE2 via the install.sh script:
   curl -sfL https://get.rke2.io | INSTALL_RKE2_COMMITID=f377e3eedca2e9037b173981af6cccb5821b072b sh -
3. Ensure the config.yaml took properly by checking the rke2.io/node-args annotation on the node with kubectl.
4. A snapshot is automatically taken as per my cron schedule.
5. An on-demand snapshot is successful:
   rke2 etcd-snapshot --s3 --s3-skip-ssl-verify --s3-region "us-east-2" --s3-bucket "dave-secret-bucket" --s3-folder "dave-rke2v120" --s3-access-key "REDACTED_ACCESS_KEY" --s3-secret-key "REDACTED_SECRET_KEY"

Result:
My scheduled snapshots and my on-demand snapshot are successfully uploaded to the bucket in the correct folder