Clients experience downtime during NGF upgrade

[pleshakov]

Describe the bug
During an upgrade of NGF, external clients can experience downtime.

To Reproduce

See the test setup and steps published in #1120

As a result, during upgrade client HTTP and HTTPs traffic got affected:

note: time scale is MM:SS

HTTP:

HTTPS:

Expected behavior

External clients don't experience downtime during an upgrade of NGF.

Your environment

Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3-gke.100", GitCommit:"6466b51b762a5c49ae3fb6c2c7233ffe1c96e48c", GitTreeState:"clean", BuildDate:"2023-06-23T09:27:28Z", GoVersion:"go1.20.5 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}

Old NGF version:

"version":"0.6.0"
"commit":"803e6d612a9574362bda28868d4410943ffaf66a"
"date":"2023-08-31T16:32:37Z"

New NGF version:

version: "edge"
commit: "5324908e6e1145bec5f2f0ab80b312a809ad1744"
date: "2023-10-13T18:29:23Z"

Additional context
See #1120

[kate-osborn]

@pleshakov can you describe the bug more in this ticket? I'm not sure which result you are pointing to in the linked PR.

[brianehlert]

I am also assuming multiple gateway replicas and a rolling upgrade in the deployment design.

[mpstefan]

Should be addressed by solving #1154 and #1155

[mpstefan]

Fixed thanks to the above issues