load balancing: add discovered server in a random position of the server list

[matthiashanel]

Feature Request

Saw this through code inspection.
The connect list is shuffled.
New server are added to the end.
This makes server, the client learns about, less likely to be picked during a re-connect.

Use Case:

Any situation where we have 2 server running, then start a third, then kill one of the former server.
Clients are equally distributed between server (a,b)
We start server c.
Because server c gets added to the end of the list it will result in the client having one of these two lists:
a, b, c (current is a)
b, a, c (current is b)

if now we restart either a or b, all clients will connect to the other one, not c.

Proposed Change:

insert the new server at a random location in the list.

Who Benefits From The Change(s)?

This is useful to reduce thundering heard and better load balancing.

Alternative Approaches

[derekcollison]

I agree with this. We should just reshuffle?

[ColinSullivan1]

Reshuffle imo, but remove the last connected server first. ;)

[matthiashanel]

@derekcollison it would be enough to insert at a random location.
(probably correcting for what the current server)

[ColinSullivan1]

Shuffle, then add last connected at the end imo. Alternatively, shuffle, then insert randomly anywhere except the front of the list.

[derekcollison]

We could get elaborate for sure.. ;)

End result should be backup servers always shuffled when added to (don't think required for removal) with current connected at the end of the list.