grpcproxy: TestLaunchDuplicateMemberShouldFail: unexpect successful launch

[heyitsanthony]

via https://jenkins-etcd-public.prod.coreos.systems/job/etcd-proxy/620/console

--- FAIL: TestLaunchDuplicateMemberShouldFail (0.80s)
	member_test.go:81: unexpect successful launch

Possibly related to #7239

[gyuho]

Root cause: duplicate member launched and got Client.Timeout exceeded while awaiting headers error(https://github.com/coreos/etcd/blob/master/etcdserver/server.go#L342,https://github.com/coreos/etcd/blob/master/etcdserver/cluster_util.go#L66-L73) from peer handler(https://github.com/coreos/etcd/blob/master/etcdserver/api/v2http/peer.go). Existing members should have started serving /members in its peer handlers, and the duplicate member fetches client URLs from these /members endpoints and finds out itself conflicts with existing members.

Problem: it is possible that duplicate member's GET request to /members times out, and etcd mistakenly thinks there's no conflict, so successfully bootstraps. Since duplicate member tries to boot with same token, it won't be caught even after successful boot.

Assume this should fail with TCP port conflicts in regular set-up. Not sure how this is handled in unix ports.

[gyuho]

I feel like just bumping up the bootstrap time-out for tests is enough. Problem was all GET requests to peers' /members endpoint timed out. But in real world, this never happens. Duplicate config member will never boot with such conflicting TCP ports.

[heyitsanthony]

A member could have more than one ip/port so there won't be a bind conflict. Maybe split-brain issues if the member is killed, has its wal dir removed, then tries to boot as a new cluster while partitioned. This path needs some more testing...

I'm thinking:

• infof the bootstrap error so there's some indication the member is assuming it's not bootstrapped
• a test that partitions a duplicate member from a booted cluster so it bootstraps, then confirms it shuts down when the partition is lifted
• either a boosted bootstrap timeout for LaunchDuplicateMemberShouldFail or confirm the duplicate eventually exits