Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release-22.1: liveness: improve disk probes during node liveness updates #81476

Merged
merged 3 commits into from
Jun 8, 2022
Merged

release-22.1: liveness: improve disk probes during node liveness updates #81476

merged 3 commits into from
Jun 8, 2022

Conversation

blathers-crl[bot]
Copy link

@blathers-crl blathers-crl bot commented May 18, 2022 •
edited by erikgrinaker
Loading

/cc @cockroachdb/release

When NodeLiveness updates the liveness record (e.g. during
heartbeats), it first does a noop sync write to all disks. This ensures
that a node with a stalled disk will fail to maintain liveness and lose
its leases.

However, this sync write could block indefinitely, and would not respect
the caller's context, which could cause the caller to stall rather than
time out. This in turn could lead to stalls higher up in the stack,
in particular with lease acquisitions that do a synchronous heartbeat.

This patch does the sync write in a separate goroutine in order to
respect the caller's context. The write operation itself will not
(can not) respect the context, and may thus leak a goroutine. However,
concurrent sync writes will coalesce onto an in-flight write.

Additionally, this runs the sync writes in parallel across all disks,
since we can now trivially do so. This may be advantageous on nodes with
many stores, to avoid spurious heartbeat failures under load.

Touches #81100.

Release note (bug fix): Disk write probes during node liveness
heartbeats will no longer get stuck on stalled disks, instead returning
an error once the operation times out. Additionally, disk probes now run
in parallel on nodes with multiple stores.

Release justification: cluster availability improvement.

@blathers-crl blathers-crl bot requested review from a team as code owners May 18, 2022 20:49
@blathers-crl blathers-crl bot force-pushed the blathers/backport-release-22.1-81133 branch from 0b54478 to 6da8e80 Compare May 18, 2022 20:49
@blathers-crl blathers-crl bot requested review from andreimatei and tbg May 18, 2022 20:49
@blathers-crl
Copy link
Author

blathers-crl bot commented May 18, 2022

Thanks for opening a backport.

Please check the backport criteria before merging:

  • Patches should only be created for serious issues or test-only changes.
  • Patches should not break backwards-compatibility.
  • Patches should change as little code as possible.
  • Patches should not change on-disk formats or node communication protocols.
  • Patches should not add new functionality.
  • Patches must not add, edit, or otherwise modify cluster versions; or add version gates.
If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.
  • There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
  • The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
  • New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
  • The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

  • What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
  • Will this work in a cluster of mixed patch versions? Did we test that?
  • If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

@blathers-crl blathers-crl bot added blathers-backport This is a backport that Blathers created automatically. O-robot Originated from a bot. labels May 18, 2022
@cockroach-teamcity
Copy link
Member

This change is Reviewable

tbg
tbg approved these changes May 18, 2022
View reviewed changes
Copy link
Member

@tbg tbg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but maybe a week or two of bake on master?

@erikgrinaker
Copy link
Contributor

LGTM but maybe a week or two of bake on master?

Yep, for sure.

@blathers-crl blathers-crl bot requested a review from a team as a code owner June 8, 2022 17:37
@erikgrinaker
liveness: improve disk probes during node liveness updates
85c352a 
When `NodeLiveness` updates the liveness record (e.g. during
heartbeats), it first does a noop sync write to all disks. This ensures
that a node with a stalled disk will fail to maintain liveness and lose
its leases.

However, this sync write could block indefinitely, and would not respect
the caller's context, which could cause the caller to stall rather than
time out. This in turn could lead to stalls higher up in the stack,
in particular with lease acquisitions that do a synchronous heartbeat.

This patch does the sync write in a separate goroutine in order to
respect the caller's context. The write operation itself will not
(can not) respect the context, and may thus leak a goroutine. However,
concurrent sync writes will coalesce onto an in-flight write.

Additionally, this runs the sync writes in parallel across all disks,
since we can now trivially do so. This may be advantageous on nodes with
many stores, to avoid spurious heartbeat failures under load.

Release note (bug fix): Disk write probes during node liveness
heartbeats will no longer get stuck on stalled disks, instead returning
an error once the operation times out. Additionally, disk probes now run
in parallel on nodes with multiple stores.
@erikgrinaker
liveness: move stopper to NodeLivenessOptions
af44457 
Release note: None
@erikgrinaker
liveness: run sync disk write in a stopper task
97054ae 
This patch runs the sync disk write during node heartbeats in a stopper
task. The write is done in a goroutine, so that we can respect the
caller's context cancellation (even though the write itself won't).
However, this could race with engine shutdown when stopping the node,
violating the Pebble contract and triggering the race detector. Running
it as a stopper task will cause the node to wait for the disk write to
complete before closing the engine.

Of course, if the disk stalls then node shutdown will now never
complete. This is very unfortunate, since stopping the node is often
the only mitigation to recover stuck ranges with stalled disks. This is
mitigated by Pebble panic'ing the node on stalled disks, and Kubernetes
and other orchestration tools killing the process after some time.

Release note: None
@erikgrinaker erikgrinaker force-pushed the blathers/backport-release-22.1-81133 branch from 3e03cc0 to 97054ae Compare June 8, 2022 17:39
@erikgrinaker erikgrinaker merged commit 79594b2 into release-22.1 Jun 8, 2022
@erikgrinaker erikgrinaker deleted the blathers/backport-release-22.1-81133 branch June 8, 2022 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tbg tbg tbg approved these changes

@andreimatei andreimatei Awaiting requested review from andreimatei

Assignees

@erikgrinaker erikgrinaker

Labels
blathers-backport This is a backport that Blathers created automatically. O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@cockroach-teamcity @erikgrinaker @tbg