Flaky tests: ClientWatchEDS and ClusterNameInAddressAttributes

[easwars]

--- FAIL: Test (13.00s)
    --- FAIL: Test/ClientWatchEDS (10.03s)
        tlogger.go:116: INFO clusterresolver.go:81 [xds] [xds-cluster-resolver-lb 0xc0003d4e00] Created  (t=+46.801µs)
        tlogger.go:116: INFO clusterresolver.go:159 [xds] [xds-cluster-resolver-lb 0xc0003d4e00] Receive update from resolver, balancer config: {
              "discoveryMechanisms": [
                {
                  "cluster": "test-cluster-name",
                  "edsServiceName": "cluster-1"
                }
              ]
            }  (t=+107.103µs)
        tlogger.go:116: INFO resource_resolver.go:227 [xds] [xds-cluster-resolver-lb 0xc0003d4e00] EDS watch started on cluster-1  (t=+118.603µs)
        tlogger.go:116: INFO clusterresolver.go:159 [xds] [xds-cluster-resolver-lb 0xc0003d4e00] Receive update from resolver, balancer config: {
              "discoveryMechanisms": [
                {
                  "cluster": "test-cluster-name",
                  "edsServiceName": "foobar-1"
                }
              ]
            }  (t=+143.803µs)
        tlogger.go:116: INFO resource_resolver.go:227 [xds] [xds-cluster-resolver-lb 0xc0003d4e00] EDS watch started on foobar-1  (t=+152.503µs)
        tlogger.go:116: INFO resource_resolver.go:244 [xds] [xds-cluster-resolver-lb 0xc0003d4e00] EDS watch canceled on cluster-1  (t=+163.604µs)
        tlogger.go:116: INFO clusterresolver.go:159 [xds] [xds-cluster-resolver-lb 0xc0003d4e00] Receive update from resolver, balancer config: {
              "discoveryMechanisms": [
                {
                  "cluster": "test-cluster-name",
                  "edsServiceName": "foobar-2"
                }
              ]
            }  (t=+190.004µs)
        tlogger.go:116: INFO resource_resolver.go:227 [xds] [xds-cluster-resolver-lb 0xc0003d4e00] EDS watch started on foobar-2  (t=+199.404µs)
        tlogger.go:116: INFO resource_resolver.go:244 [xds] [xds-cluster-resolver-lb 0xc0003d4e00] EDS watch canceled on foobar-1  (t=+207.604µs)
        tlogger.go:116: INFO resource_resolver.go:244 [xds] [xds-cluster-resolver-lb 0xc0003d4e00] EDS watch canceled on foobar-2  (t=+250.405µs)
        tlogger.go:116: INFO clusterresolver.go:311 [xds] [xds-cluster-resolver-lb 0xc0003d4e00] Shutdown  (t=+258.305µs)
        grpctest.go:39: Leaked goroutine: goroutine 196 [IO wait]:
            internal/poll.runtime_pollWait(0x7f6e3094ff40, 0x72, 0xffffffffffffffff)
            	/opt/hostedtoolcache/go/1.16.13/x64/src/runtime/netpoll.go:227 +0x55
            internal/poll.(*pollDesc).wait(0xc000150d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
            	/opt/hostedtoolcache/go/1.16.13/x64/src/internal/poll/fd_poll_runtime.go:87 +0x45
            internal/poll.(*pollDesc).waitRead(...)
            	/opt/hostedtoolcache/go/1.16.13/x64/src/internal/poll/fd_poll_runtime.go:92
            internal/poll.(*FD).Read(0xc000150d80, 0xc00037f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
            	/opt/hostedtoolcache/go/1.16.13/x64/src/internal/poll/fd_unix.go:166 +0x1d5
            net.(*netFD).Read(0xc000150d80, 0xc00037f000, 0x1000, 0x1000, 0x7f6e30593768, 0x16c1ee0, 0xc0000abc70)
            	/opt/hostedtoolcache/go/1.16.13/x64/src/net/fd_posix.go:55 +0x4f
            net.(*conn).Read(0xc0000100a8, 0xc00037f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
            	/opt/hostedtoolcache/go/1.16.13/x64/src/net/net.go:183 +0x91
            net/http.(*persistConn).Read(0xc0000d7440, 0xc00037f000, 0x1000, 0x1000, 0x4076d0, 0x60, 0x0)
            	/opt/hostedtoolcache/go/1.16.13/x64/src/net/http/transport.go:1922 +0x77
            bufio.(*Reader).fill(0xc0000f6a80)
            	/opt/hostedtoolcache/go/1.16.13/x64/src/bufio/bufio.go:101 +0x108
            bufio.(*Reader).Peek(0xc0000f6a80, 0x1, 0xc000300000, 0x16, 0xc0000bc723, 0xffffffffffffffff, 0xc000065600)
            	/opt/hostedtoolcache/go/1.16.13/x64/src/bufio/bufio.go:139 +0x4f
            net/http.(*persistConn).readLoop(0xc0000d7440)
            	/opt/hostedtoolcache/go/1.16.13/x64/src/net/http/transport.go:2083 +0x1a8
            created by net/http.(*Transport).dialConn
            	/opt/hostedtoolcache/go/1.16.13/x64/src/net/http/transport.go:1743 +0xc77
        grpctest.go:39: Leaked goroutine: goroutine 197 [select]:
            net/http.(*persistConn).writeLoop(0xc0000d7440)
            	/opt/hostedtoolcache/go/1.16.13/x64/src/net/http/transport.go:2382 +0xf7
            created by net/http.(*Transport).dialConn
            	/opt/hostedtoolcache/go/1.16.13/x64/src/net/http/transport.go:1744 +0xc9c
        grpctest.go:60: Leak check disabled for future tests
FAIL
FAIL	google.golang.org/grpc/xds/internal/balancer/clusterresolver	16.832s

https://github.com/grpc/grpc-go/runs/4958237379?check_suite_focus=true

[easwars]

Another one: https://github.com/grpc/grpc-go/runs/4958534588?check_suite_focus=true

[easwars]

Crap, I can't reproduce this even once in 1000 runs on forge. But seems to have happened twice in a row in a single PR here.

[easwars]

https://github.com/grpc/grpc-go/runs/4958731483?check_suite_focus=true

[easwars]

https://github.com/grpc/grpc-go/runs/4983977390?check_suite_focus=true

[easwars]

https://github.com/grpc/grpc-go/runs/5010349058?check_suite_focus=true

[easwars]

https://github.com/grpc/grpc-go/runs/5010777514?check_suite_focus=true

[menghanl]

reproduced without gRPC: https://github.com/menghanl/cloud-compute-metadata-leak/runs/5043626024?check_suite_focus=true

Filed an google cloud issue: googleapis/google-cloud-go#5430

[dfawley]

Worked around in grpc-go via #5182. Cloud issue will track more permanent fix, if possible.