Etcd cluster keeps getting corrupted on Talos since version 1.8.x upgrade ( etcd 3.5.16)

[rkoosaar]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Hi folks, I have a really odd issue that I'm troubleshooting. I have a 3 node Talos (1.8.3) cluster at home where etcd (3.5.16) keeps getting corrupted after a while. Initially I thought it could be a disk related issue. So I bought brand new disks and swapped them around. I installed a new cluster last night (around 8pm) and when I woke up this morning (8am) cluster was not working and etcd was reporting cluster corrupted.
Looking at the logs, it seems something happened around 6am, but I'm unable to work out what the cause is.
So far I have redeployed the cluster in the past week 4 times and every time etcd has ended up corrupted.
Any help/guidance to troubleshoot this would be much appreciated.

What did you expect to happen?

Cluster not the get corrupted

How can we reproduce it (as minimally and precisely as possible)?

I'm not 100% sure how this can be reproduced in your env as I don't fully understand why this happens

Anything else we need to know?

I have actually saved a log bundles from all 3 cluster nodes using talosctl -n node_ip support
I'm just not sure which log files would be helpful. If you could advise which logs are needed I can provide them:
the log bundle has folders:
kubernetes-logs
service-logs (etcd.log file here, I pasted it in the relevant log section)
and separately log files:
controller-runtime.log
dmesg.log

Etcd version (please run commands below)

here is the output of EtcdConfigs.etcd.talos.dev file from node1

$ etcd --version
# paste output here
metadata:
    namespace: etcd
    type: EtcdConfigs.etcd.talos.dev
    id: etcd
    version: 1
    owner: etcd.ConfigController
    phase: running
    created: 2024-11-20T20:09:34Z
    updated: 2024-11-20T20:09:34Z
spec:
    advertiseValidSubnets:
        - 10.1.1.0/24
    advertiseExcludeSubnets:
        - 10.1.1.30
    listenValidSubnets:
        - 10.1.1.0/24
    listenExcludeSubnets: []
    image: gcr.io/etcd-development/etcd:v3.5.16
    extraArgs:
        listen-metrics-urls: http://0.0.0.0:2381

$ etcdctl version
# paste output here

Etcd configuration (command line flags or environment variables)

paste your configuration here

metadata:
namespace: etcd
type: EtcdConfigs.etcd.talos.dev
id: etcd
version: 1
owner: etcd.ConfigController
phase: running
created: 2024-11-20T20:09:34Z
updated: 2024-11-20T20:09:34Z
spec:
advertiseValidSubnets:
- 10.1.1.0/24
advertiseExcludeSubnets:
- 10.1.1.30
listenValidSubnets:
- 10.1.1.0/24
listenExcludeSubnets: []
image: gcr.io/etcd-development/etcd:v3.5.16
extraArgs:
listen-metrics-urls: http://0.0.0.0:2381

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

I'm not 100% sure how I can run the below commands on talos

$ etcdctl member list -w table
# paste output here
talosctl -n 10.1.1.32 etcd members
NODE        ID                 HOSTNAME    PEER URLS                CLIENT URLS              LEARNER
10.1.1.32   05f2a341182982bc   nectarine   https://10.1.1.33:2380   https://10.1.1.33:2379   false
10.1.1.32   4ce3ddbc6cc0469b   melon       https://10.1.1.32:2380   https://10.1.1.32:2379   false
10.1.1.32   d9983876898d0723   mango       https://10.1.1.31:2380   https://10.1.1.31:2379   false
$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

id:0723934b321f01b0>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.708327Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.25µs","request":"header:<ID:5087822155244138456 > lease_revoke:<id:02bc934b32ef139b>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.708355Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef139b","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.708372Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321f01b0","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709189Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.598µs","request":"header:<ID:5087822155244138455 > lease_revoke:<id:0723934b321e383c>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709267Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.711µs","request":"header:<ID:5087822155244138457 > lease_revoke:<id:02bc934b32ef5746>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709318Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef5746","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709336Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321e383c","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.840207Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.109µs","request":"header:<ID:9420566471738703565 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/network/ingress-nginx-internal-leader\" mod_revision:482768 > success:<request_put:<key:\"/registry/leases/network/ingress-nginx-internal-leader\" value_size:439 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.846801Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.293µs","request":"header:<ID:514416733623534472 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/spegel-leader-election\" mod_revision:482773 > success:<request_put:<key:\"/registry/leases/kube-system/spegel-leader-election\" value_size:450 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.922691Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.287µs","request":"header:<ID:5087822155244138460 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/services/endpoints/openebs-system/openebs.io-local\" mod_revision:482782 > success:<request_put:<key:\"/registry/services/endpoints/openebs-system/openebs.io-local\" value_size:542 >> failure:<request_range:<key:\"/registry/services/endpoints/openebs-system/openebs.io-local\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.969917Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.089µs","request":"header:<ID:514416733623534475 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/spegel-leader-election\" mod_revision:482773 > success:<request_put:<key:\"/registry/leases/kube-system/spegel-leader-election\" value_size:450 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.988979Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.809µs","request":"header:<ID:514416733623534478 username:\"kube-apiserver\" auth_revision:1 > lease_grant:<ttl:15-second id:0723934b3225c38d>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.039835Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.076µs","request":"header:<ID:5087822155244138461 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-ea1044b5-360f-4ad6-a190-a6c1a4190ab6-e-0\" mod_revision:474638 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-ea1044b5-360f-4ad6-a190-a6c1a4190ab6-e-0\" value_size:8401 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-ea1044b5-360f-4ad6-a190-a6c1a4190ab6-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207388Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.406µs","request":"header:<ID:5087822155244138465 > lease_revoke:<id:0723934b321bc809>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207492Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.201µs","request":"header:<ID:5087822155244138466 > lease_revoke:<id:02bc934b32ef3ecb>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207531Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef3ecb","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207598Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321bc809","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208243Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.962µs","request":"header:<ID:5087822155244138468 > lease_revoke:<id:0723934b321e790a>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208313Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.534µs","request":"header:<ID:5087822155244138467 > lease_revoke:<id:02bc934b32ef3db5>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208366Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"20.283µs","request":"header:<ID:5087822155244138469 > lease_revoke:<id:0723934b321c69c9>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208351Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321e790a","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208410Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.463µs","request":"header:<ID:5087822155244138470 > lease_revoke:<id:02bc934b32ef09ab>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208424Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef3db5","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208448Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321c69c9","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208448Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef09ab","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.218575Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.363µs","request":"header:<ID:514416733623534481 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:482780 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:412 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.246152Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.879µs","request":"header:<ID:9420566471738703566 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.254819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.528µs","request":"header:<ID:9420566471738703567 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.263370Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.398µs","request":"header:<ID:9420566471738703568 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.275016Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.819µs","request":"header:<ID:9420566471738703569 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.283479Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.384µs","request":"header:<ID:9420566471738703570 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.296399Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.681µs","request":"header:<ID:9420566471738703572 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.304905Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.013µs","request":"header:<ID:9420566471738703573 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.314273Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.867µs","request":"header:<ID:9420566471738703574 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.322761Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.973µs","request":"header:<ID:9420566471738703575 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.332116Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.185µs","request":"header:<ID:9420566471738703576 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.472883Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.726µs","request":"header:<ID:9420566471738703578 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/database/db9c8771.cnpg.io\" mod_revision:482784 > success:<request_put:<key:\"/registry/leases/database/db9c8771.cnpg.io\" value_size:428 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.478827Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.75µs","request":"header:<ID:5087822155244138473 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-0f2da8c6-25ff-4fdb-83f1-de2df988b575-e-0\" mod_revision:456502 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-0f2da8c6-25ff-4fdb-83f1-de2df988b575-e-0\" value_size:7837 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-0f2da8c6-25ff-4fdb-83f1-de2df988b575-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.498891Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.983µs","request":"header:<ID:514416733623534485 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/helm-controller-64cd59b777-l22bw\" mod_revision:2901 > success:<request_put:<key:\"/registry/pods/flux-system/helm-controller-64cd59b777-l22bw\" value_size:5391 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.520308Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.01µs","request":"header:<ID:514416733623534487 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/openebs-system/openebs-localpv-provisioner-56d8447b9-kbxm7\" mod_revision:4743 > success:<request_put:<key:\"/registry/pods/openebs-system/openebs-localpv-provisioner-56d8447b9-kbxm7\" value_size:4789 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.544561Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.851µs","request":"header:<ID:514416733623534491 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-controller-manager\" mod_revision:482767 > success:<request_put:<key:\"/registry/leases/kube-system/kube-controller-manager\" value_size:432 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.549218Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.825µs","request":"header:<ID:514416733623534492 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/kustomize-controller-54f65db458-fg22s\" mod_revision:3124 > success:<request_put:<key:\"/registry/pods/flux-system/kustomize-controller-54f65db458-fg22s\" value_size:5343 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.582441Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.94µs","request":"header:<ID:5087822155244138474 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-9750c179-d6c3-40d0-b89f-6dc1e6fd45c4-e-0\" mod_revision:474686 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-9750c179-d6c3-40d0-b89f-6dc1e6fd45c4-e-0\" value_size:8387 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-9750c179-d6c3-40d0-b89f-6dc1e6fd45c4-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.708052Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.388µs","request":"header:<ID:5087822155244138475 > lease_revoke:<id:02bc934b32ef1d03>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.708127Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef1d03","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709060Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.758µs","request":"header:<ID:5087822155244138476 > lease_revoke:<id:0723934b321deb0c>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709134Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.642µs","request":"header:<ID:5087822155244138478 > lease_revoke:<id:02bc934b32ef3bcf>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709165Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.181µs","request":"header:<ID:5087822155244138477 > lease_revoke:<id:469b934b33aa80b1>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709191Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.744µs","request":"header:<ID:5087822155244138479 > lease_revoke:<id:02bc934b32ef1fa0>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709214Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.752µs","request":"header:<ID:5087822155244138480 > lease_revoke:<id:0723934b321d2570>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709222Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321deb0c","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709286Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef3bcf","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709313Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef1fa0","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709293Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321d2570","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709333Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"469b934b33aa80b1","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.852623Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.431µs","request":"header:<ID:514416733623534495 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:482780 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:410 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.993053Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.448µs","request":"header:<ID:9420566471738703580 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/external-resizer-driver-longhorn-io\" mod_revision:482772 > success:<request_put:<key:\"/registry/leases/storage/external-resizer-driver-longhorn-io\" value_size:410 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.028156Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.859µs","request":"header:<ID:9420566471738703582 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/driver-longhorn-io\" mod_revision:482778 > success:<request_put:<key:\"/registry/leases/storage/driver-longhorn-io\" value_size:404 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.028822Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.164µs","request":"header:<ID:5087822155244138481 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-a37c368b-f784-476d-a0e4-be4db6645b7e-e-0\" mod_revision:361515 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-a37c368b-f784-476d-a0e4-be4db6645b7e-e-0\" value_size:7850 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-a37c368b-f784-476d-a0e4-be4db6645b7e-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.044577Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.882µs","request":"header:<ID:5087822155244138482 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-0beb7017-cc7c-42c5-9509-4c1bb1fa4cc1-e-0\" mod_revision:475880 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-0beb7017-cc7c-42c5-9509-4c1bb1fa4cc1-e-0\" value_size:8399 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-0beb7017-cc7c-42c5-9509-4c1bb1fa4cc1-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.044630Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.259µs","request":"header:<ID:5087822155244138483 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-05d7cd83-dcd5-4385-913e-0ec1d3d8c06e-e-0\" mod_revision:481730 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-05d7cd83-dcd5-4385-913e-0ec1d3d8c06e-e-0\" value_size:8400 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-05d7cd83-dcd5-4385-913e-0ec1d3d8c06e-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.045521Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.235µs","request":"header:<ID:5087822155244138484 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-85502885-8e7a-4d4a-a58e-0dd33bd54cc5-e-0\" mod_revision:482731 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-85502885-8e7a-4d4a-a58e-0dd33bd54cc5-e-0\" value_size:8401 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-85502885-8e7a-4d4a-a58e-0dd33bd54cc5-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.056059Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.953µs","request":"header:<ID:9420566471738703585 username:\"kube-apiserver\" auth_revision:1 > lease_grant:<ttl:15-second id:02bc934b32f0bee0>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.139893Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.315µs","request":"header:<ID:514416733623534498 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kubelet-csr-approver\" mod_revision:482781 > success:<request_put:<key:\"/registry/leases/kube-system/kubelet-csr-approver\" value_size:451 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.187110Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.253µs","request":"header:<ID:514416733623534500 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/flux-system/source-controller-leader-election\" mod_revision:482775 > success:<request_put:<key:\"/registry/leases/flux-system/source-controller-leader-election\" value_size:458 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.195634Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.015µs","request":"header:<ID:514416733623534502 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/flux-system/kustomize-controller-leader-election\" mod_revision:482740 > success:<request_put:<key:\"/registry/leases/flux-system/kustomize-controller-leader-election\" value_size:468 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.208502Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.898µs","request":"header:<ID:5087822155244138486 > lease_revoke:<id:0723934b321bfe07>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.208593Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321bfe07","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209357Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.243µs","request":"header:<ID:5087822155244138487 > lease_revoke:<id:0723934b321eb240>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209427Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.749µs","request":"header:<ID:5087822155244138488 > lease_revoke:<id:02bc934b32ef52fd>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209463Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.023µs","request":"header:<ID:5087822155244138489 > lease_revoke:<id:0723934b321ec88f>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209488Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.756µs","request":"header:<ID:5087822155244138490 > lease_revoke:<id:0723934b321dfb59>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209513Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.696µs","request":"header:<ID:5087822155244138492 > lease_revoke:<id:02bc934b32ef453e>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209517Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321eb240","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209544Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.952µs","request":"header:<ID:5087822155244138491 > lease_revoke:<id:02bc934b32ef2ba3>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209565Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef453e","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209592Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321dfb59","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209588Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321ec88f","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209628Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef2ba3","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209631Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef52fd","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.477676Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.433µs","request":"header:<ID:5087822155244138493 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-d34b0c22-6951-4be9-b742-37bff4cd2efc-e-0\" mod_revision:482626 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-d34b0c22-6951-4be9-b742-37bff4cd2efc-e-0\" value_size:8390 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-d34b0c22-6951-4be9-b742-37bff4cd2efc-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.494209Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.532µs","request":"header:<ID:514416733623534506 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/flux-system/image-reflector-controller-leader-election\" mod_revision:482755 > success:<request_put:<key:\"/registry/leases/flux-system/image-reflector-controller-leader-election\" value_size:488 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710090Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.661µs","request":"header:<ID:5087822155244138494 > lease_revoke:<id:469b934b33aa516c>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710191Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"469b934b33aa516c","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710658Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.44µs","request":"header:<ID:5087822155244138495 > lease_revoke:<id:02bc934b32ef4e33>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710740Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.006µs","request":"header:<ID:5087822155244138497 > lease_revoke:<id:02bc934b32ef5148>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710799Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef4e33","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710820Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.853µs","request":"header:<ID:5087822155244138496 > lease_revoke:<id:0723934b321c30b9>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710847Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef5148","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710896Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.145µs","request":"header:<ID:5087822155244138498 > lease_revoke:<id:469b934b33aab5f6>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710938Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.95µs","request":"header:<ID:5087822155244138499 > lease_revoke:<id:0723934b321d9310>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710978Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.44µs","request":"header:<ID:5087822155244138500 > lease_revoke:<id:0723934b321d6665>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710963Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321c30b9","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710978Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"469b934b33aab5f6","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.711009Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321d6665","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.711018Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321d9310","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.796843Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.747µs","request":"header:<ID:514416733623534509 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/cilium-operator-resource-lock\" mod_revision:482771 > success:<request_put:<key:\"/registry/leases/kube-system/cilium-operator-resource-lock\" value_size:408 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.837363Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.5µs","request":"header:<ID:514416733623534511 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/external-snapshotter-leader-driver-longhorn-io\" mod_revision:482745 > success:<request_put:<key:\"/registry/leases/storage/external-snapshotter-leader-driver-longhorn-io\" value_size:428 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.897227Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.009µs","request":"header:<ID:514416733623534513 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-controller-manager\" mod_revision:482767 > success:<request_put:<key:\"/registry/leases/kube-system/kube-controller-manager\" value_size:428 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.912839Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.143µs","request":"header:<ID:514416733623534515 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/storage/csi-attacher-787d99954-j6tt7\" mod_revision:12987 > success:<request_put:<key:\"/registry/pods/storage/csi-attacher-787d99954-j6tt7\" value_size:3913 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.934714Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.311µs","request":"header:<ID:514416733623534517 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/cert-manager/cert-manager-cainjector-c879b86d6-w7tg9\" mod_revision:130011 > success:<request_put:<key:\"/registry/pods/cert-manager/cert-manager-cainjector-c879b86d6-w7tg9\" value_size:4165 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.957608Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.483µs","request":"header:<ID:514416733623534519 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/security/authentik-worker-6c9b654886-bj8vj\" mod_revision:478106 > success:<request_put:<key:\"/registry/pods/security/authentik-worker-6c9b654886-bj8vj\" value_size:4627 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.982359Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.64µs","request":"header:<ID:514416733623534521 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/notification-controller-556b8867f8-79nbc\" mod_revision:2849 > success:<request_put:<key:\"/registry/pods/flux-system/notification-controller-556b8867f8-79nbc\" value_size:5424 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.000088Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.055µs","request":"header:<ID:514416733623534523 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:475 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.009692Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.335µs","request":"header:<ID:514416733623534524 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/storage/volsync-8466895bb6-mmlrf\" mod_revision:12652 > success:<request_put:<key:\"/registry/pods/storage/volsync-8466895bb6-mmlrf\" value_size:6091 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.011609Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.855µs","request":"header:<ID:514416733623534525 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:473 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.019020Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.332µs","request":"header:<ID:5087822155244138501 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-38a643b6-abcd-457b-94e0-b75d357d9864-e-0\" mod_revision:480448 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-38a643b6-abcd-457b-94e0-b75d357d9864-e-0\" value_size:8399 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-38a643b6-abcd-457b-94e0-b75d357d9864-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.020536Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.689µs","request":"header:<ID:514416733623534526 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.029988Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"11.455µs","request":"header:<ID:514416733623534528 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.038232Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.985µs","request":"header:<ID:514416733623534529 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/database/cloudnative-pg-586475f9d5-drx9k\" mod_revision:29569 > success:<request_put:<key:\"/registry/pods/database/cloudnative-pg-586475f9d5-drx9k\" value_size:5602 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.040470Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.149µs","request":"header:<ID:514416733623534530 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.043611Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"14.402µs","request":"header:<ID:5087822155244138502 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-7c5c589f-0e6d-4e70-81ce-12c0d4c349c4-e-0\" mod_revision:482081 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-7c5c589f-0e6d-4e70-81ce-12c0d4c349c4-e-0\" value_size:7851 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-7c5c589f-0e6d-4e70-81ce-12c0d4c349c4-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.053946Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.008µs","request":"header:<ID:514416733623534533 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.059050Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.624µs","request":"header:<ID:514416733623534534 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/cert-manager/cert-manager-659dbd9c7f-cmm4j\" mod_revision:130005 > success:<request_put:<key:\"/registry/pods/cert-manager/cert-manager-659dbd9c7f-cmm4j\" value_size:4795 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.064973Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.353µs","request":"header:<ID:514416733623534535 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.067134Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.292µs","request":"header:<ID:9420566471738703587 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/driver-longhorn-io\" mod_revision:482778 > success:<request_put:<key:\"/registry/leases/storage/driver-longhorn-io\" value_size:404 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.073474Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.807µs","request":"header:<ID:514416733623534538 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.076837Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.029µs","request":"header:<ID:514416733623534539 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:482780 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:408 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.083302Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.122µs","request":"header:<ID:514416733623534540 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/image-reflector-controller-7cdc88c5c7-vjdng\" mod_revision:2937 > success:<request_put:<key:\"/registry/pods/flux-system/image-reflector-controller-7cdc88c5c7-vjdng\" value_size:5448 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.083940Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.07µs","request":"header:<ID:514416733623534541 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.091384Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6µs","request":"header:<ID:514416733623534542 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.210758Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.282µs","request":"header:<ID:5087822155244138506 > lease_revoke:<id:0723934b321bb66b>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.210836Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321bb66b","error":"etcdserver: corrupt cluster"}

[serathius]

Logs you provided are just from request failures due to past corruption alarm. Can you provide the logs from time the corruption happen?

[rkoosaar]

This is what the log bundle had from the time I generated it. Not sure what happened then or why it starts from that time. Maybe it got overwritten since its constantly saying cluster corrupted. do you think any other log file from the bundle might help? or me uploading the whole bundle?

[ahrtr]

This is what the log bundle had from the time I generated it.

Could you provide the complete log of all etcd instances?

or me uploading the whole bundle?

Yes, please. Is it possible to upload all the db files (under the ${data_dir}/member/snap/db) If it doesn't have any sensitive data?

[rkoosaar]

Sorry no I only have log bundles from the nodes (nodes themselves have been reset). I'll see if there's any sensitive data in the log bundles and take that out and then provide those.
I have a fresh cluster running again, same 1.8.3 with same etcd version.

[rkoosaar]

here are the support log bundles i got from nodes running - talosctl -n node_ip support -
support-31-33.zip

[ahrtr]

here are the support log bundles i got from nodes running - talosctl -n node_ip support - support-31-33.zip

Unfortunately, the log in this bundle isn't complete either. The first log entry is the same (etcdserver: corrupt cluster) as you already posted in this ticket.

I also do not see the data files as mentioned in #18933 (comment). If you can provide that files, I can analyze the db file directly.

[rkoosaar]

I'm not sure how I could get the etcd db of talos nodes as there is no direct shell access. just commands via talosctl

[ahrtr]

I'm not sure how I could get the etcd db of talos nodes as there is no direct shell access. just commands via talosctl

You might want to raise the question in talos community?

[samfili]

Hi folks, I have a really odd issue that I'm troubleshooting. I have a 3 node Talos (1.8.3) cluster at home where etcd (3.5.16) keeps getting corrupted after a while. Initially I thought it could be a disk related issue. So I bought brand new disks and swapped them around. I installed a new cluster last night (around 8pm) and when I woke up this morning (8am) cluster was not working and etcd was reporting cluster corrupted. Looking at the logs, it seems something happened around 6am, but I'm unable to work out what the cause is. So far I have redeployed the cluster in the past week 4 times and every time etcd has ended up corrupted. Any help/guidance to troubleshoot this would be much appreciated.

I have the same problem, in addition, I have it happening on the VMware software platform: cluster - 3 control plane, 3 working nodes, everything works fine after a clean installation for about 4-6 hours etcd then goes into the error "etcdserver: no leader", this is the last thing I could diagnose, build an image of Talos in Factory-installed with extensions :

customization:
systemExtensions:
officialExtensions:
- siderolabs/drbd
- siderolabs/iscsi-tools
- siderolabs/util-linux-tools
- siderolabs/vmtoolsd-guest-agent
- siderolabs/zfs

Interestingly, if you completely turn off the virtual machines with Talos, then wait a few seconds and start them back one by one starting from the first control plane, everything starts working again, then after a few hours everything repeats again.