Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

debug: add version constraint to avoid pprof panic #12807

Merged
merged 18 commits into from
Apr 28, 2022
Merged

Conversation

davemay99
Copy link
Contributor

This PR introduces go-version constraint checks to avoid capturing the threadcreate pprof profile on Nomad versions 0.11.0 to 0.11.2. The code to obtain the version number for a given server or node ID isn't pretty, but in lieu of a better method it should suffice for now.

Fixes #12804.

@davemay99 davemay99 added this to the 1.3.0 milestone Apr 28, 2022
@davemay99 davemay99 requested review from tgross and lgfa29 and removed request for tgross April 28, 2022 04:56
tgross
tgross previously requested changes Apr 28, 2022
Copy link
Member

@tgross tgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @davemay99. The code here is roughly correct (although I've left some comments), but I have a concern about this approach in a more general sense:

We're trying to rip out all the backcompat code that supports unsupported versions of Nomad (ex #12791). So I'm honestly not convinced we should do this at all, especially given that this version of Nomad is vulnerable to being crashed by anyone hitting the threadcreate pprof endpoint, as I noted in #12804 (comment).

If we are going to do it, I think this approach is too narrow: I'd propose we skip the entire pprof on any version of Nomad earlier than 0.12.0 (that's when we switched to go modules and at least have a reasonable story for discovering what's going on with dependencies).

command/operator_debug_test.go Show resolved Hide resolved
command/operator_debug.go Outdated Show resolved Hide resolved
command/operator_debug.go Outdated Show resolved Hide resolved
command/operator_debug_test.go Outdated Show resolved Hide resolved
command/operator_debug.go Show resolved Hide resolved
@tgross
Copy link
Member

tgross commented Apr 28, 2022

@lgfa29 @davemay99 in the interest of moving things along, I've pushed updates for my comments onto this PR, along with the changelog entry and rebased on main (the branch was pretty old). But now I shouldn't be the one to 👍 the PR so I'm adding @schmichael to the reviewers so that either @lgfa29 or @schmichael can get to it first 😀

@davemay99
Copy link
Contributor Author

If we are going to do it, I think this approach is too narrow: I'd propose we skip the entire pprof on any version of Nomad earlier than 0.12.0 (that's when we switched to go modules and at least have a reasonable story for discovering what's going on with dependencies).

This PR was driven by a specific scenario and is the only incompatibility that I am aware of. The remainder of the pprofs work as expected and still provide valuable debugging context for all versions of Nomad.

@tgross Are you open to the following 2 changes? I would be happy to provide the commit.

  1. revert the version constraint back to ">= 0.11.0, <= 0.11.2"
  2. only skip the threadcreate pprof

Copy link
Contributor

@jazzyfresh jazzyfresh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made some feedback comments but can ignore

.changelog/12807.txt Outdated Show resolved Hide resolved
if !c.Check(v) {
return nil
}
return fmt.Errorf("unsupported version=%s matches version filter %s", version, minimumVersionPprofConstraint)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return fmt.Errorf("unsupported version=%s matches version filter %s", version, minimumVersionPprofConstraint)
return fmt.Errorf("unsupported version=%s matches version filter %s", version, versionConstraint)

should show the versionConstraint passed in since it used for checking

command/operator_debug.go Show resolved Hide resolved
.changelog/12807.txt Outdated Show resolved Hide resolved
@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Nomad panic when capturing threadcreate pprof on version 0.11.0 to 0.11.2
4 participants