WaitForLeader times out before an election takes place

[nstott]

On a new cluster, If i do the following,
start up one node, wait more then 5 seconds, start up a second node
then the first server will hit the timeout here:
https://github.com/influxdb/influxdb/blob/master/meta/store.go#L402
and will wait endlessly here:
https://github.com/influxdb/influxdb/blob/master/cmd/influxd/run/server.go#L298

The first server is stuck trying to elect a leader. Once the other nodes come online then a leader is elected and the nodes are created properly, but the first node never completes the initialization and needs to be restarted

It seems to me as though there shouldn't be a timeout there. if the node doesn't exist and we need to meta.CreateLocalNode, then we shouldn't timeout prematurely

[pauldix]

@jwilder I think this can be done in conjunction with the work on cluster joining #2966. Since @benbjohnson is going to be working on compression in the 0.9.2 release cycle, can you take a look at both this and the joining issue for 0.9.2?

[beckettsean]

Possibly related: https://groups.google.com/d/msgid/influxdb/df98f329-8274-45b0-ae9c-744375171ae9%40googlegroups.com, https://groups.google.com/d/msgid/influxdb/a8941fa8-054a-4fd4-9c64-573d1c30bcae%40googlegroups.com

[otoolep]

Believe fixed by #3256

[otoolep]

Cherry-picked to 0.9.2.