Move etcd snapshot management CLI to request/response

[brandond]

Proposed Changes

Moves etcd-snapshot commands over to a request/response process, similar to how secrets-encrypt and certificate rotate-ca work. Benefits of this include:

• No longer having to recreate the whole server control config in the CLI just to take a snapshot.
  We were having issues with config passed as server args not being visible to the etcd-snapshot cli, for example cluster/service cidr, node name, and so on.
• Ensures that the process for on-demand snapshots is consistent with that of scheduled snapshots.
• Ensures that multiple snapshots are not taken concurrently.

It does slightly reduce the feedback available via the CLI when taking a snapshot, as the actual operation is now completed server-side, the client is just sent back a list of snapshots or an error.

• etcd-snapshot save/prune/delete now only output either a short error message, or the name of the created/deleted snapshots
• The json and yaml format output of etcd-snapshot list has changed
• The text format output of etcd-snapshot list is identical

Rancher CAPR only uses the text format output of etcd-snapshot save so this should be fine:

• https://github.com/rancher/rancher/blob/9ccc45e6775024fbf4db7e61cc0efb6be4fb44a3/pkg/capr/planner/instructions.go#L125-L153

Example CLI output:

root@k3s-server-1:~# k3s etcd-snapshot save
INFO[0000] Snapshot on-demand-k3s-server-1-1712336224 saved.

root@k3s-server-1:~# k3s etcd-snapshot save --etcd-s3
FATA[0000] see server log for details: s3 bucket name was not set

root@k3s-server-1:~# k3s etcd-snapshot save --etcd-s3 --etcd-s3-bucket foo
FATA[0001] see server log for details: failed to test for existence of bucket foo: 401 Unauthorized

root@k3s-server-1:~# k3s etcd-snapshot save --etcd-snapshot-dir /foo
FATA[0000] see server log for details: Internal error occurred: etcd-snapshot error ID 18952

root@k3s-server-1:~# k3s etcd-snapshot list
Name                                  Location                                                                              Size   Created
etcd-snapshot-k3s-server-1-1712336221 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-k3s-server-1-1712336221 634912 2024-04-05T16:57:01Z
on-demand-k3s-server-1-1712336224     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-k3s-server-1-1712336224     643104 2024-04-05T16:57:04Z

Example server logs when a generic error with ID is sent:

ERRO[0193] etcd-snapshot error ID 18952: failed to get etcd-snapshot-dir: stat /foo: no such file or directory
ERRO[0193] Sending HTTP 500 response to 127.0.0.1:36124: etcd-snapshot error ID 18952

Types of Changes

bugfix/enhancement

Verification

See linked issue

Testing

Linked Issues

• k3s etcd-snapshot save fails on host with IPv6 only #9214

User-Facing Change

The `k3s etcd-snapshot` command has been reworked for improved consistency. All snapshots operations are now performed by the server process, with the CLI acting as a client to initiate and report results. As a side effect, the CLI is now less noisy when managing snapshots.

Further Comments

[codecov]

Codecov Report

Attention: Patch coverage is 24.47917% with 290 lines in your changes are missing coverage. Please review.

Project coverage is 42.94%. Comparing base (7f65975) to head (3056aa2).
Report is 4 commits behind head on master.

Files	Patch %	Lines
pkg/etcd/snapshot_handler.go	0.78%	125 Missing and 1 partial ⚠️
pkg/etcd/snapshot.go	0.00%	79 Missing ⚠️
pkg/cli/etcdsnapshot/etcd_snapshot.go	61.32%	24 Missing and 17 partials ⚠️
pkg/clientaccess/token.go	53.84%	9 Missing and 9 partials ⚠️
pkg/etcd/s3.go	0.00%	10 Missing ⚠️
pkg/server/secrets-encrypt.go	0.00%	7 Missing ⚠️
pkg/util/apierrors.go	0.00%	4 Missing ⚠️
pkg/etcd/etcd.go	40.00%	3 Missing ⚠️
pkg/server/cert.go	0.00%	1 Missing ⚠️
pkg/server/token.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9816      +/-   ##
==========================================
- Coverage   52.67%   42.94%   -9.73%     
==========================================
  Files         157      158       +1     
  Lines       13822    13991     +169     
==========================================
- Hits         7281     6009    -1272     
- Misses       5155     6842    +1687     
+ Partials     1386     1140     -246     

Flag	Coverage Δ
e2etests	35.99% <18.75%> (-13.27%)	⬇️
inttests	19.72% <20.05%> (-2.41%)	⬇️
unittests	16.57% <4.52%> (-0.26%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dereknola]

Generally LGTM, some random questions on my end.

[dereknola]

I assume we are sending the full error back to the client on failure, we don't try and hide it like secrets-encryption?

[brandond]

Many errors are probably just returned as-is at the moment. Let me take a look at whether or not that exposes anything.

[dereknola]

It should be fine, but if you want to obfuscate you may find https://github.com/k3s-io/k3s/blob/master/pkg/server/secrets-encrypt.go#L471 helpful as it makes grepping logs alot easier for the user.

[brandond]

I refactored this a bit. Most that indicate problems with the input from the user are still passed through with minimal information, while others just send a more generic message with error ID. For example:

root@k3s-server-1:~# k3s etcd-snapshot save
INFO[0000] Snapshot on-demand-k3s-server-1-1712288593 saved.

root@k3s-server-1:~# k3s etcd-snapshot save --etcd-s3
FATA[0000] see server log for details: s3 bucket name was not set

root@k3s-server-1:~# k3s etcd-snapshot save --etcd-s3 --etcd-s3-bucket foo
FATA[0001] see server log for details: failed to test for existence of bucket foo: 401 Unauthorized

root@k3s-server-1:~# k3s etcd-snapshot save --etcd-snapshot-dir /foo
FATA[0000] see server log for details: Internal error occurred: etcd-snapshot error ID 18952

ERRO[0193] etcd-snapshot error ID 18952: failed to get etcd-snapshot-dir: stat /foo: no such file or directory
ERRO[0193] Sending HTTP 500 response to 127.0.0.1:36124: etcd-snapshot error ID 18952

[brandond]

Confirmation that this works on an ipv6-only node:

root@systemd-node-1:/# ip addr show eth0
1880: eth0@if1881: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:08 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fd7c:53a5:aef5::242:ac11:8/64 scope global nodad
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:8/64 scope link
       valid_lft forever preferred_lft forever

root@systemd-node-1:/# kubectl get node -o wide
NAME             STATUS   ROLES                       AGE    VERSION                INTERNAL-IP                  EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
systemd-node-1   Ready    control-plane,etcd,master   109m   v1.29.3+k3s-1a9615a7   fd7c:53a5:aef5::242:ac11:8   <none>        openSUSE Leap 15.4   6.6.0-1001-aws   containerd://1.7.11-k3s2

root@systemd-node-1:/# k3s etcd-snapshot save
WARN[0000] Unknown flag --node-ip found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-cidr found in config.yaml, skipping
WARN[0000] Unknown flag --service-cidr found in config.yaml, skipping
WARN[0000] Unknown flag --system-default-registry found in config.yaml, skipping
WARN[0000] Unknown flag --disable found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
INFO[0000] Snapshot on-demand-systemd-node-1-1712700964 saved.