Flaky test ListSnapshots [Controller Server] should return empty when the specify source volume id is not exist

[saad-ali]

The following test fails in travis randomly:

•••••••••••••••••[32m••••••••••••••••••••••••••STEP: connecting to CSI driver
------------------------------
• Failure in Spec Setup (BeforeEach) [60.002 seconds]
ListSnapshots [Controller Server]
/home/travis/gopath/src/github.com/kubernetes-csi/csi-test/pkg/sanity/tests.go:44
  should return empty when the specify source volume id is not exist [BeforeEach]
  /home/travis/gopath/src/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:1195
  Expected error:
      <*errors.errorString | 0xc420327130>: {
          s: "Connection timed out",
      }
      Connection timed out
  not to have occurred
  /home/travis/gopath/src/github.com/kubernetes-csi/csi-test/pkg/sanity/sanity.go:97
------------------------------
••••••••••••••

[pohly]

Looks like the issue that was reported by @darkowlzz in PR #97 and PR #98.

It's not specific to csi-test: the sidecar containers and kubelet can also run into intermittent connection problems. IMHO this needs further discussion and a consistent approach for retrying either the connection attempt in the custom dialer for Unix domain sockets or the gRPC request (as in PR #97). This is getting complicated enough that we'll probably want to have the code in a common package, which would need to be created first.

In the meantime I'll fix PR #98 so that it also works when embedding the sanity test in a large Ginkgo suite.