Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exit chain creation routine before shutting down chain router #2140

Merged
merged 1 commit into from
Oct 5, 2023

Conversation

StephenButtolph
Copy link
Contributor

@StephenButtolph StephenButtolph commented Oct 5, 2023

Why this should be merged

Shutting down the chain manager can currently happen concurrently with chain creation: https://github.com/ava-labs/avalanchego/blob/master/chains/manager.go#L1421-L1433. This results in a weird state where a new chain may be added after the chain router is shutdown. Additionally, the rest of the node shutdown process may continue assuming that all chains had finished gracefully shutting down. This can cause unexpected PANICs, FATALs, and ERRORs during shutdown.

How this works

Ensure the chain creation goroutine has exited before shutting down the chain router.

How this was tested

  • This bug was originally found by antithesis and this patch no longer replicates the unexpected behavior in their simulations.
  • CI

@StephenButtolph StephenButtolph added the bug Something isn't working label Oct 5, 2023
@StephenButtolph StephenButtolph added this to the v1.10.12 milestone Oct 5, 2023
@StephenButtolph StephenButtolph self-assigned this Oct 5, 2023
@@ -1393,11 +1396,14 @@ func (m *manager) StartChainCreator(platformParams ChainParameters) error {
m.createChain(platformParams)

m.Log.Info("starting chain creator")
m.chainCreatorExited.Add(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normally I'd use an exit channel for this given the wg will only ever have value 1?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking around in avalanchego - I think it's much more common for us to use a wg than a channel for this. Even with a single goroutine

@StephenButtolph StephenButtolph added this pull request to the merge queue Oct 5, 2023
Merged via the queue into dev with commit 386065f Oct 5, 2023
16 checks passed
@StephenButtolph StephenButtolph deleted the fix-chain-manager-shutdown branch October 5, 2023 05:44
@StephenButtolph StephenButtolph added the antithesis Related to an issue reported by Antithesis label Mar 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
antithesis Related to an issue reported by Antithesis bug Something isn't working
Projects
No open projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants