Skip to content

Commit

Permalink
Documentation: add FAQs on membership operation
Browse files Browse the repository at this point in the history
Copy Anthony's answer from:
#6103
#6114
  • Loading branch information
gyuho committed Dec 16, 2016
1 parent 531c306 commit 851b0bb
Showing 1 changed file with 16 additions and 0 deletions.
16 changes: 16 additions & 0 deletions Documentation/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,22 @@ With longer latencies, the default etcd configuration may cause frequent electio

etcdctl provides a `snapshot` command to create backups. See [backup][backup] for more details.

#### Always remove first when replacing member?

When replacing an etcd node, we recommend to remove the member first and then add its replacement.

etcd employs distributed consensus based on a quorum model; (n+1)/2 members, a majority, must agree on a proposal before it can be committed to the cluster. These proposals include key-value updates and membership changes. This model totally avoids any possibility of split brain inconsistency. The downside is permanent quorum loss is catastrophic.

How this applies to membership: If a 3-member cluster has 1 downed member, it can still make forward progress because the quorum is 2 and 2 members are still live. However, adding a new member to a 3-member cluster will increase the quorum to 3 because 3 votes are required for a majority of 4 members. Since the quorum increased, this extra member buys nothing in terms of fault tolerance; the cluster is still one node failure away from being unrecoverable.

Additionally, that new member is risky because it may turn out to be misconfigured or incapable of joining the cluster. In that case, there's no way to recover quorum because the cluster has two members down and two members up, but needs three votes to change membership to undo the botched membership addition. etcd will by default reject member add attempts that could take down the cluster in this manner.

On the other hand, if the downed member is removed from cluster membership first, the number of members becomes 2 and the quorum remains at 2. Following that removal by adding a new member will also keep the quorum steady at 2. So, even if the new node can't be brought up, it's still possible to remove the new member through quorum on the remaining live members.

#### Why so strict about membership change?

etcd sets `strict-reconfig-check` in order to reject reconfiguration requests that would cause quorum loss. Abandoning quorum is really risky (especially when the cluster is already in a bad way). We're aware that losing quorum is painful, but disabling quorum on membership could lead to full fledged cluster inconsistency and that would be even worse in many applications ("disk geometry corruption" being a candidate for most terrifying).

### Performance

#### How should I benchmark etcd?
Expand Down

0 comments on commit 851b0bb

Please sign in to comment.