*: handle request timeout error to '/member' when bootstrapping

[gyuho]

Fix #7267.

In slow CPU machine, it often takes more than 10ms, so times out.

When log is enabled, I get a bunch of

2017-05-04 01:28:59.260298 W | etcdserver: could not get cluster response from unix://127.0.0.1:2100723422: Get unix://127.0.0.1:2100723422/members: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2017-05-04 01:28:59.260471 W | etcdserver: could not get cluster response from unix://127.0.0.1:2100723422: Get unix://127.0.0.1:2100723422/members: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[heyitsanthony]

I set my bootstrap timeout to a nanosecond and was able to reproduce #7267.

If the test is failing because c.Launch(t) is failing to bootstrap, shouldn't c.Launch(t) be fataling out before the test reaches m.Launch(t)? Bad error handling somewhere?

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@4301f49). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #7873   +/-   ##
=========================================
  Coverage          ?   76.02%           
=========================================
  Files             ?      332           
  Lines             ?    26203           
  Branches          ?        0           
=========================================
  Hits              ?    19921           
  Misses            ?     4863           
  Partials          ?     1419

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4301f49...c73abaa. Read the comment docs.

[gyuho]

@heyitsanthony

It's failing here with m.Launch(t) (member that's supposed to fail when launching)

https://github.com/coreos/etcd/blob/master/etcdserver/server.go#L342

if isMemberBootstrapped(cl, cfg.Name, prt, cfg.bootstrapTimeout()) {
	return nil, fmt.Errorf("member %s has already been bootstrapped", m.ID)
}

Normal fresh cluster expects isMemberBootstrapped to return false because it has not been bootstrapped yet. It returns false with Client.Timeout exceeded while awaiting headers errors.

Duplicate config member, however, expects isMemberBootstrapped to return true. The duplicate member should be found via /member endpoint. But also possible to return false, when request to /member times out, with Client.Timeout exceeded while awaiting headers errors.

So it's failing in m.Launch(t) (cloned member duplicate launch), because request timed out (10ms) and isMemberBootstrapped returned false.

We need to differentiate between time-out from slow network and time-out from non-existing /member endpoint?

[heyitsanthony]

OK, isMemberBootStrapped should probably return the error (instead of just false) so etcd doesn't try to boot?