Non-leader contour controller pod memory keeps increasing until OOM

[Levi080513]

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

1. Install contour by Helm, the contour.replicaCount is 2, and the envoy service config is this:

envoy:
  service:
    type: LoadBalancer

2. One of the contour controller pod memory keep increasing until OOM.

What did you expect to happen:

All contour controller pod works well.

Anything else you would like to add:

The pod memory heap (generated by https://github.com/cloudwego/goref):

The pod memory goroutine:

Checked the goroutine of the pod, the service events consumer is hangs on ServiceStatusLoadBalancerWatcher.notify.

On non-leader contour controller pod, loadBalancerStatusWriter will not start, it will cause ServiceStatusLoadBalancerWatcher.notify blocking writing to channel.

And then, all service events are cached in pendingNotifications.ringGrowing and will not be consumed. And as new events are generated, more events are cached in pendingNotifications.ringGrowing, and the memory continues to increase, final OOM.

Environment:

• Contour version: 1.27.0
• Kubernetes version: (use kubectl version): 1.26.15
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

[github-actions]

Hey @Levi080513! Thanks for opening your first issue. We appreciate your contribution and welcome you to our community! We are glad to have you here and to have your input on Contour. You can also join us on our mailing list and in our channel in the Kubernetes Slack Workspace