Fix flapping of mesh gateway connect-service watches

[crhino]

Fixes #7503 , at least 1 specific cause of flakes.

What is the Problem?

While investigating flakey tests, I found the root cause to be a bug in our mesh gateway watch logic.

The bug happens like so:

1. A Mesh Gateway sets up an initial watch for all of the services in its datacenter, so that it can proxy traffic to them.

2. Upon receiving an update event for this watch, the proxycfg package will iterate over the entire list of services and set up a cacetype.HealthServicesName watch for any service that is not being watched already. During this loop, we also add the serviceID to a map so that we can cancel any unneeded watches, however we only added the serviceID to the map if we do not have a watch already. This is the bug.

3. After iterating over the services, we then iterate over a map currently watched services and cancel any services that are not in the map we constructed in step 2. This causes currently available services to be canceled because of the bug in step 2.

An example of the issue:

1. We have a mesh gateway which has received a list of services, like so:

web
api
consul
mesh-gateway
db
redis
mysql

and generates health-service watches for them.

2. Before these health-service watches return, another service-list watch fires and returns a new list of services, like so:

web
api
consul
mesh-gateway
db
redis
mysql
dashboard

This new list of services will currently generate a new health-services watch for the dashboard service, and cancel the watches for the other services.

This can manifest itself in delayed endpoint updates for services, where a service health check might have updated some time in the past but Envoy continues to think it is healthy/unhealthy. Or in pathological cases, this can manifest in the service never even being added as an Envoy cluster in the first place, if no other service-list updates come in.

Verification

While investigating, I ran this for loop to get the gateways-local integration test to fail consistently:

for i in $(seq 1 30); do FILTER_TESTS="gateways-local" ENVOY_VERSIONS="1.13.1" make test-envoy-integ > test.out; if [ $? -ne 0 ]; then break; fi; echo; done

After applying the fix, I was able to run this to completion without any failures.

[mkeeler]

[banks]

👏 Great work! Debugging flakey tests can be frustrating but surprising often does turn up actual bugs with enough perseverance. Thanks for digging into this one!

[banks]

Also, should we backport this to 1.7.x Is seems like a simple change to backport and a nasty bug in gateways that culd bite peope in real deployments too?