emissary-ingress v3.9.1 crash on EKS 1.29

[Space-Banana-42]

Describe the bug
emissary-ingress pod trapped in CrashLoopBackOff status with the error msg:

time="2024-09-26 17:23:50.6035" level=info msg="Started Ambassador (Version 3.9.1)" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.Main file="/go/cmd/entrypoint/entrypoint.go:96" CMD=entrypoint PID=1
time="2024-09-26 17:23:50.6433" level=info msg="AMBASSADOR_CLUSTER_ID=237a40bf-acd3-510d-8f2f-9a6846a40481" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.Main file="/go/cmd/entrypoint/entrypoint.go:100" CMD=entrypoint PID=1
time="2024-09-26 17:23:50.6438" level=info msg="started command [\"diagd\" \"/ambassador/snapshots\" \"/ambassador/bootstrap-ads.json\" \"/ambassador/envoy/envoy.json\" \"--notices\" \"/ambassador/notices.json\" \"--port\" \"8004\" \"--kick\" \"kill -HUP 1\"]" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:183" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=17
time="2024-09-26 17:23:50.6439" level=info msg="not logging input read from file \"/dev/stdin\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:185" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=17 dexec.stream=stdin
time="2024-09-26 17:23:50.6439" level=info msg="not logging output written to file \"/dev/stdout\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:188" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=17 dexec.stream=stdout
time="2024-09-26 17:23:50.6439" level=info msg="not logging output written to file \"/dev/stderr\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:191" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=17 dexec.stream=stderr
time="2024-09-26 17:23:50.6449" level=info msg="Ambex 3.9.1 starting, snapdirPath /ambassador/snapshots" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.Main file="/go/pkg/ambex/main.go:648" CMD=entrypoint PID=1 THREAD=/ambex
time="2024-09-26 17:23:50.6451" level=info msg="Wrote PID" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.Main file="/go/pkg/ambex/main.go:692" CMD=entrypoint PID=1 THREAD=/ambex fields.file=ambex.pid file="/go/pkg/ambex/main.go:692" pid=1
time="2024-09-26 17:23:50.6452" level=info msg="Listening on tcp:127.0.0.1:8003" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.runManagementServer file="/go/pkg/ambex/main.go:226" CMD=entrypoint PID=1 THREAD=/ambex/management-server
time="2024-09-26 17:23:50.6453" level=info msg="AMBASSADOR_RECONFIG_MAX_DELAY set to 1" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.WatchAllTheThings file="/go/cmd/entrypoint/watcher.go:47" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-09-26 17:23:50.6454" level=info msg="Saved snapshot v0" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.csDump file="/go/pkg/ambex/main.go:351" CMD=entrypoint PID=1 THREAD=/ambex/main-loop
time="2024-09-26 17:23:50.6456" level=info msg="Pushing snapshot v0" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.updaterWithTicker file="/go/pkg/ambex/ratelimit.go:159" CMD=entrypoint PID=1 THREAD=/ambex/updater
time="2024-09-26 17:23:50.6467" level=warning msg="Warning, unable to watch httproutes.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:156" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-09-26 17:23:50.6467" level=warning msg="Warning, unable to watch gatewayclasses.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:156" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-09-26 17:23:50.6467" level=warning msg="Warning, unable to watch gateways.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:156" CMD=entrypoint PID=1 THREAD=/watcher
2024-09-26 17:23:51 diagd 3.9.1 [P17TMainThread] INFO: AMBASSADOR_FAST_RECONFIGURE enabled, initializing cache
2024-09-26 17:23:51 diagd 3.9.1 [P17TMainThread] INFO: WILL NOT update Mapping status
2024-09-26 17:23:51 diagd 3.9.1 [P17TMainThread] INFO: thread count 17, listening on 127.0.0.1:8004
2024-09-26 17:23:51 diagd 3.9.1 [P17TMainThread] INFO: Ambassador 3.9.1 booted
[2024-09-26 17:23:51 +0000] [17] [INFO] Starting gunicorn 21.2.0
[2024-09-26 17:23:51 +0000] [17] [INFO] Listening at: http://127.0.0.1:8004 (17)
[2024-09-26 17:23:51 +0000] [17] [INFO] Using worker: gthread
[2024-09-26 17:23:51 +0000] [22] [INFO] Booting worker with pid: 22
2024-09-26 17:23:51 diagd 3.9.1 [P22TAEW] INFO: starting Scout checker and timer logger
2024-09-26 17:23:51 diagd 3.9.1 [P22TAEW] INFO: starting event watcher
I0926 17:23:51.852609       1 request.go:697] Waited for 1.111553543s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/networking.k8s.io/v1/ingressclasses?limit=500&resourceVersion=0
time="2024-09-26 17:23:52.9524" level=error msg="goroutine \"/watcher/loop\" exited with error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:369\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nreflect.flag.mustBeAssignableSlow\n\t/usr/local/go/src/reflect/value.go:269\nreflect.flag.mustBeAssignable\n\t/usr/local/go/src/reflect/value.go:259\nreflect.Value.Set\n\t/usr/local/go/src/reflect/value.go:2254\nk8s.io/apimachinery/pkg/runtime.structFromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:556\nk8s.io/apimachinery/pkg/runtime.fromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:359\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructuredWithValidation\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:247\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:274\nk8s.io/apimachinery/pkg/runtime.(*Scheme).unstructuredToTyped\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:539\nk8s.io/apimachinery/pkg/runtime.(*Scheme).convertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:453\nk8s.io/apimachinery/pkg/runtime.(*Scheme).ConvertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:438\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.convertAnnotationObject\n\t/go/pkg/snapshot/v1/annotations.go:102\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.ValidateAndConvertObject\n\t/go/pkg/snapshot/v1/annotations.go:84\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.(*KubernetesSnapshot).PopulateAnnotations\n\t/go/pkg/snapshot/v1/annotations.go:46\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1.2\n\t/go/cmd/entrypoint/watcher.go:448\ngithub.com/emissary-ingress/emissary/v3/pkg/debug.(*Timer).Time\n\t/go/pkg/debug/timer.go:164\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1\n\t/go/cmd/entrypoint/watcher.go:447\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate\n\t/go/cmd/entrypoint/watcher.go:544\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.watchAllTheThingsInternal.func2\n\t/go/cmd/entrypoint/watcher.go:278\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:385\ngithub.com/datawire/dlib/derrgroup.(*Group).Go.func2\n\t/go/vendor/github.com/datawire/dlib/derrgroup/errgroup.go:132\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/watcher/loop
time="2024-09-26 17:23:52.9525" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_logger"
time="2024-09-26 17:23:52.9526" level=info msg="  final goroutine statuses:" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:84" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-09-26 17:23:52.9526" level=info msg="    /watcher/consul  : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-09-26 17:23:52.9526" level=info msg="    /watcher/loop    : exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-09-26 17:23:52.9526" level=info msg="    /watcher/notifyCh: exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-09-26 17:23:52.9527" level=error msg="goroutine \"/watcher\" exited with error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:369\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nreflect.flag.mustBeAssignableSlow\n\t/usr/local/go/src/reflect/value.go:269\nreflect.flag.mustBeAssignable\n\t/usr/local/go/src/reflect/value.go:259\nreflect.Value.Set\n\t/usr/local/go/src/reflect/value.go:2254\nk8s.io/apimachinery/pkg/runtime.structFromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:556\nk8s.io/apimachinery/pkg/runtime.fromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:359\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructuredWithValidation\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:247\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:274\nk8s.io/apimachinery/pkg/runtime.(*Scheme).unstructuredToTyped\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:539\nk8s.io/apimachinery/pkg/runtime.(*Scheme).convertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:453\nk8s.io/apimachinery/pkg/runtime.(*Scheme).ConvertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:438\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.convertAnnotationObject\n\t/go/pkg/snapshot/v1/annotations.go:102\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.ValidateAndConvertObject\n\t/go/pkg/snapshot/v1/annotations.go:84\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.(*KubernetesSnapshot).PopulateAnnotations\n\t/go/pkg/snapshot/v1/annotations.go:46\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1.2\n\t/go/cmd/entrypoint/watcher.go:448\ngithub.com/emissary-ingress/emissary/v3/pkg/debug.(*Timer).Time\n\t/go/pkg/debug/timer.go:164\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1\n\t/go/cmd/entrypoint/watcher.go:447\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate\n\t/go/cmd/entrypoint/watcher.go:544\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.watchAllTheThingsInternal.func2\n\t/go/cmd/entrypoint/watcher.go:278\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:385\ngithub.com/datawire/dlib/derrgroup.(*Group).Go.func2\n\t/go/vendor/github.com/datawire/dlib/derrgroup/errgroup.go:132\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-09-26 17:23:52.9527" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD="/ambex:shutdown_logger"
time="2024-09-26 17:23:52.9529" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD=":shutdown_logger"
time="2024-09-26 17:23:52.9529" level=info msg="sending SIGINT" func="github.com/datawire/dlib/dexec.(*Cmd).Start.func1" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:211" CMD=entrypoint PID=1 THREAD=/diagd
[2024-09-26 17:23:52 +0000] [17] [INFO] Handling signal: int
time="2024-09-26 17:23:52.9562" level=info msg="Memory Usage 0.12Gi (6%)\n    PID 1, 0.13Gi: busyambassador entrypoint \n    PID 17, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 \n    PID 22, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 " func="github.com/emissary-ingress/emissary/v3/pkg/memory.(*MemoryUsage).Watch" file="/go/pkg/memory/memory.go:43" CMD=entrypoint PID=1 THREAD=/memory
[2024-09-26 17:23:53 +0000] [22] [INFO] Worker exiting (pid: 22)
[2024-09-26 17:23:53 +0000] [17] [INFO] Shutting down: Master
time="2024-09-26 17:23:53.2018" level=info msg="finished successfully: exit status 0" func="github.com/datawire/dlib/dexec.(*Cmd).Wait" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:255" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=17
time="2024-09-26 17:23:53.2019" level=info msg="  final goroutine statuses:" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:84" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2020" level=info msg="    /ambex                   : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2020" level=info msg="    /diagd                   : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2020" level=info msg="    /envoy                   : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2020" level=info msg="    /external_snapshot_server: exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2021" level=info msg="    /healthchecks            : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2021" level=info msg="    /memory                  : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2021" level=info msg="    /snapshot_server         : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2021" level=info msg="    /watcher                 : exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2021" level=error msg="shut down with error error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field" func=github.com/emissary-ingress/emissary/v3/pkg/busy.Main file="/go/pkg/busy/busy.go:87" CMD=entrypoint PID=1

To Reproduce

Env

EKS v1.29
Emissary-ingress app v3.9.1
Emissary-ingress chart v8.9.1

Installation command

kubectl apply -f https://app.getambassador.io/yaml/emissary/3.9.1/emissary-crds.yaml
kubectl wait --timeout=90s --for=condition=available deployment emissary-apiext -n emissary-system
helm upgrade --install emissary-ingress datawire/emissary-ingress \
  --namespace emissary \
  --create-namespace \
  --version=8.9.1 

Kindly suggest what could be the error.