[release-1.24] Multiple issues with temporary bootstrap etcd config

[brandond]

• Backport Multiple issues with temporary bootstrap etcd config #7368

[mdrahman-suse]

Validated on branch release-1.24 with commit f575bd1

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

PRETTY_NAME="Ubuntu 22.04.2 LTS"

Cluster Configuration:

1 server

Config.yaml:

write-kubeconfig-mode: 644
token: summer
debug: true
cluster-init: true

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s
3. Stop k3s server
4. Start k3s server with additional etcd params

$ sudo k3s server --etcd-arg='--quota-backend-bytes=7000000000'

5. Check logs that the change is applied

Replication Results:

• k3s version used for replication:

k3s version v1.24.13+k3s1 (3f79b289)
go version go1.19.8

• Observed the logs that change is not applied and default is used

$ sudo k3s server --etcd-arg='--quota-backend-bytes=7000000000'
...
{
    "level": "info",
    "ts": "2023-05-18T20:32:22.523Z",
    "caller": "embed/etcd.go:308",
    "msg": "starting an etcd server",
    "etcd-version": "3.5.3",
    "git-sha": "Not provided (use ./build instead of go build)",
    "go-version": "go1.19.8",
    "go-os": "linux",
    "go-arch": "amd64",
    "max-cpu-set": 2,
    "max-cpu-available": 2,
    "member-initialized": true,
    "name": "ip-172-31-3-94-b89f23a2",
    "data-dir": "/var/lib/rancher/k3s/server/db/etcd-tmp",
    "wal-dir": "",
    "wal-dir-dedicated": "",
    "member-dir": "/var/lib/rancher/k3s/server/db/etcd-tmp/member",
    "force-new-cluster": true,
    "heartbeat-interval": "500ms",
    "election-timeout": "5s",
    "initial-election-tick-advance": true,
    "snapshot-count": 10000,
    "snapshot-catchup-entries": 5000,
    "initial-advertise-peer-urls": [
        "http://127.0.0.1:2400"
    ],
    "listen-peer-urls": [
        "http://127.0.0.1:2400"
    ],
    "advertise-client-urls": [
        "http://127.0.0.1:2399"
    ],
    "listen-client-urls": [
        "http://127.0.0.1:2399"
    ],
    "listen-metrics-urls": [],
    "cors": [
        "*"
    ],
    "host-whitelist": [
        "*"
    ],
    "initial-cluster": "",
    "initial-cluster-state": "new",
    "initial-cluster-token": "",
    "quota-size-bytes": 2147483648,
    "pre-vote": true,
    "initial-corrupt-check": true,
    "corrupt-check-time-interval": "0s",
    "auto-compaction-mode": "",
    "auto-compaction-retention": "0s",
    "auto-compaction-interval": "0s",
    "discovery-url": "",
    "discovery-proxy": "",
    "downgrade-check-interval": "5s"
}
...
{"level":"info","ts":"2023-05-18T20:32:22.594Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
...

INFO[0013] Defragmenting etcd database
{"level":"info","ts":"2023-05-18T20:52:24.510Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-05-18T20:52:24.513Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/k3s/server/db/etcd/member/snap/db","current-db-size-bytes":5120000,"current-db-size":"5.1 MB","current-db-size-in-use-bytes":2277376,"current-db-size-in-use":"2.3 MB"}
{"level":"info","ts":"2023-05-18T20:52:24.559Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/k3s/server/db/etcd/member/snap/db","current-db-size-bytes-diff":-2998272,"current-db-size-bytes":2121728,"current-db-size":"2.1 MB","current-db-size-in-use-bytes-diff":-163840,"current-db-size-in-use-bytes":2113536,"current-db-size-in-use":"2.1 MB","took":"48.935609ms"}
{"level":"info","ts":"2023-05-18T20:52:24.559Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}

Validation Results:

• k3s version used for validation:

k3s version v1.24.13+k3s-f575bd1c (f575bd1c)
go version go1.19.8

• Observed the logs that change is applied

$ sudo k3s server --etcd-arg='--quota-backend-bytes=7000000000'
{
    "level": "info",
    "ts": "2023-05-18T20:58:23.038Z",
    "caller": "embed/etcd.go:308",
    "msg": "starting an etcd server",
    "etcd-version": "3.5.3",
    "git-sha": "Not provided (use ./build instead of go build)",
    "go-version": "go1.19.8",
    "go-os": "linux",
    "go-arch": "amd64",
    "max-cpu-set": 2,
    "max-cpu-available": 2,
    "member-initialized": true,
    "name": "ip-172-31-3-94-634f89e2",
    "data-dir": "/var/lib/rancher/k3s/server/db/etcd-tmp",
    "wal-dir": "",
    "wal-dir-dedicated": "",
    "member-dir": "/var/lib/rancher/k3s/server/db/etcd-tmp/member",
    "force-new-cluster": true,
    "heartbeat-interval": "500ms",
    "election-timeout": "5s",
    "initial-election-tick-advance": true,
    "snapshot-count": 10000,
    "snapshot-catchup-entries": 5000,
    "initial-advertise-peer-urls": [
        "http://127.0.0.1:2400"
    ],
    "listen-peer-urls": [
        "http://127.0.0.1:2400"
    ],
    "advertise-client-urls": [
        "http://127.0.0.1:2399"
    ],
    "listen-client-urls": [
        "http://127.0.0.1:2399"
    ],
    "listen-metrics-urls": [],
    "cors": [
        "*"
    ],
    "host-whitelist": [
        "*"
    ],
    "initial-cluster": "",
    "initial-cluster-state": "new",
    "initial-cluster-token": "",
    "quota-size-bytes": 7000000000,
    "pre-vote": true,
    "initial-corrupt-check": true,
    "corrupt-check-time-interval": "0s",
    "auto-compaction-mode": "",
    "auto-compaction-retention": "0s",
    "auto-compaction-interval": "0s",
    "discovery-url": "",
    "discovery-proxy": "",
    "downgrade-check-interval": "5s"
}
{
    "level": "info",
    "ts": "2023-05-18T20:58:23.040Z",
    "caller": "etcdserver/backend.go:81",
    "msg": "opened backend db",
    "path": "/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/db",
    "took": "1.329412ms"
}
...
{"level":"info","ts":"2023-05-18T20:58:23.087Z","caller":"etcdserver/quota.go:117","msg":"enabled backend quota","quota-name":"v3-applier","quota-size-bytes":7000000000,"quota-size":"7.0 GB"}
...
INFO[0007] Defragmenting etcd database
{"level":"info","ts":"2023-05-18T20:58:30.580Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-05-18T20:58:30.584Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":2281472,"current-db-size":"2.3 MB","current-db-size-in-use-bytes":2273280,"current-db-size-in-use":"2.3 MB"}
{"level":"info","ts":"2023-05-18T20:58:30.624Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-77824,"current-db-size-bytes":2203648,"current-db-size":"2.2 MB","current-db-size-in-use-bytes-diff":-77824,"current-db-size-in-use-bytes":2195456,"current-db-size-in-use":"2.2 MB","took":"44.040852ms"}
{"level":"info","ts":"2023-05-18T20:58:30.624Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
INFO[0007] etcd temporary data store connection OK