cli: add command or flag to actively remove node from cluster

[cuongdo]

We need some kind of command or flag to actively remove a node from the cluster when there's no intention to bring that node back up.

At a minimum, this would:

• Ensure that the node not show up in the admin UI in the future
• Migrate replicas off the stores on that node in a clean way
• Prevent the stream of log entries that occur when you currently kill a node. For example:

I160421 11:32:01.362135 /Users/cdo/go/src/google.golang.org/grpc/clientconn.go:317  Failed to dial cdo-cockroach.local:26259: grpc: timed out trying to connect; please retry.
W160421 11:32:10.651793 ts/db.go:98  error writing time series data: failed to send RPC: too many errors encountered (1 of 1 total): rpc error: code = 4 desc = context deadline exceeded

• Shut down the CockroachDB instance on the node.
• Probably log an entry in the event log

[tbg]

This somewhat decomposes into a few more granular features:

• drain a node (i.e. let all leader leases expire and don't obtain new
  ones, disable all queues, maybe prevent new SQL connections - but keep
  making Raft progress) and wait until the slate is clean
  Used for clean shutdown (and pretty much always part of any type of
  shutdown)
• drain data off a node (i.e. have all or some subset of ranges migrate
  away, and wait until that has happened)
  Used for decommissioning, but could also be used to change hard-drives
  cleanly (i.e. drain data off a store).
• drain for update/migration (see the {F,Unf}reeze proposal in [RFC] Proposer-evaluted KV #6166)
  and finally
• decommission (permanently remove), which is "most of the above".

On Thu, Apr 21, 2016 at 11:43 AM Cuong Do [email protected] wrote:

We need some kind of command or flag to actively remove a node from the
cluster when there's no intention to bring that node back up.

At a minimum, this would:

• Ensure that the node not show up in the admin UI in the future
• Migrate replicas off the stores on that node in a clean way
• Prevent the stream of log entries that occur when you currently kill
  a node. For example:

I160421 11:32:01.362135 /Users/cdo/go/src/google.golang.org/grpc/clientconn.go:317 Failed to dial cdo-cockroach.local:26259: grpc: timed out trying to connect; please retry.
W160421 11:32:10.651793 ts/db.go:98 error writing time series data: failed to send RPC: too many errors encountered (1 of 1 total): rpc error: code = 4 desc = context deadline exceeded

• Shut down the CockroachDB instance on the node.
• Probably log an entry in the event log

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#6198

-- Tobias

[kwando]

So... there is no way of retire a node in a cluster?

[tbg]

There is (or at least should be), but this issue was closed prematurely.

[kwando]

That's what I figured =)

[tbg]

Yep, thanks for pointing this out.

More precisely, here are intermediate steps for draining a cluster:

• use ZoneConfigs to initiate the data of the node moving elsewhere
• wait until that has happened
• turn off the node.

The first two steps are currently too manual, especially the waiting. Once we have them figured out, it might be worth to package the result into a cli command.

[petermattis]

Currently ZoneConfigs are the only way to drain a node, but I think that is an inappropriate use. We really want a simpler mechanism to mark a node as draining.

[tbg]

Why is that an inappropriate use? ZoneConfigs describe what data is where and how often, and when I want all data data of a node to move somewhere else, it seems like exactly the appropriate use - building a second mechanism for a special case of that seems unwise.

Where value can be added is in making that process simple and intuitive, but nothing is required there that we don't want anyway. For example, we certainly want a tool to be able to tell you whether a given replication zone has all of its constraints satisfied (i.e. is everything replicated in the right way), the high level version of that giving you the cluster's overall status. This tool would tell you exactly when it's safe to take that node you're trying to get rid of down.
Also, an explicit remove command can announce to the cluster that the node isn't coming back, which can avoid triggering certain alerts.

I agree that it's not clear how the ZoneConfig would in general support the removal of nodes (it seems easy enough to do in any standard situation, but harder if the zone config is completely trivial or if there are a gazillion zones). What else did you have in mind?

[petermattis]

ZoneConfigs are settable by the user and there might be dozens or hundreds. Changing all of them in order to drain a node doesn't seem reasonable. Conceptually, what we'd want is something where there is an implicit -draining attribute on every ZoneConfig and marking a node as draining adds that attribute. We'd still go through the standard allocator/rebalancing logic. Re-reading this, perhaps we're saying the same thing.

[tbg]

Ok, sure sounds like it. Glad we're on the same page.

[kaeverens]

any progress on this? I'd like to cleanly remove a server from my cluster.

in the meantime, is it safe to simply turn one of the servers off? would the others eventually give up on it and stop trying to talk to it?

[kaeverens]

as a note, I see some work an RFC on this:
https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/drain_modes.md

[bdarnell]

Yes, it's safe to simply turn one of the servers off. The data will be re-replicated onto other nodes.

[spencerkimball]

I'm moving this off of 1.0.

[EamonZhang]

mark

[tbg]

Remaining here is the ui work (hiding nodes that are successfully decommissioned in the ui, at least after a while) and adding an event (that shows up in the ui) when a node is decommissioning/decommissioned.

See the last commit in #17419 for beginnings of this.

We'll also want another pass over the cli commands (#17419) before writing the documentation.

cc @jseldess

[jseldess]

Thanks for the update, @tschottdorf.

[cuongdo]

@couchand can you help @tschottdorf get up to speed with the UI changes?

@tschottdorf we're spreading out UI work, because we don't have nearly enough engineers conversant with the front-end to centralize admin UI development for new features. I think this should be a gentle intro given the minimal admin UI change needed for this

[tbg]

@cuongdo sgtm, but note that I still have version migrations on my plate and so I won't get to this for perhaps another two weeks.

[couchand]

@tschottdorf @cuongdo I'm happy to take a few minutes this week to talk through what will be needed here (but it has to be this week, I'm out for a month after Thursday...).

Moving forward, let's work to get the implementation of UI pieces of major features started well in advance of feature freeze, to be sure that there's plenty of time for the design to bake. And ideally let's try to include some implementation details in RFCs where possible.

[cuongdo]

@benesch has agreed to work on the UI component for this

[tbg]

Thanks @benesch, appreciate it! You'll want to look at the ui-related commit in https://github.com/cockroachdb/cockroach/pull/17157/commits.

[tbg]

I'll call this done, since the only remaining item is tracked in #17677, though the test is flaky (#17995).