Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

client: avoid trying to remove unreadable cgroup #17450

Merged
merged 1 commit into from
Jun 7, 2023

Conversation

shoenig
Copy link
Member

@shoenig shoenig commented Jun 7, 2023

During shutdown of a client with drain_on_shutdown there is a race between
the Client ending the cgroup and the task's cpuset manager cleaning up
the cgroup. During the path traversal, skip anything we cannot read, which
avoids the nil DirEntry we try to dereference now.

Fixes #17439

During shutdown of a client with drain_on_shutdown there is a race between
the Client ending the cgroup and the task's cpuset manager cleaning up
the cgroup. During the path traversal, skip anything we cannot read, which
avoids the nil DirEntry we try to dereference now.
@shoenig shoenig force-pushed the skip-unreadable-cgroup branch from 0fc7df3 to f348b4f Compare June 7, 2023 19:11
@shoenig shoenig added backport/1.3.x backport to 1.3.x release line backport/1.4.x backport to 1.4.x release line backport/1.5.x backport to 1.5.x release line labels Jun 7, 2023
@shoenig shoenig added this to the 1.5.x milestone Jun 7, 2023
@shoenig shoenig marked this pull request as ready for review June 7, 2023 19:38
@shoenig shoenig requested review from lgfa29 and tgross June 7, 2023 19:39
Copy link
Member

@tgross tgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I see we're backporting this to 1.3.x and 1.4.x which don't have drain_on_shutdown. Is this a problem for any case where an allocation is being shutdown during a client shutdown? If so, should we note that in the changelog?

@shoenig
Copy link
Member Author

shoenig commented Jun 7, 2023

Oh whoops, I forgot drain_on_shutdown is so recent; will remove those backports since I don't think we're affected without it

@shoenig shoenig removed backport/1.3.x backport to 1.3.x release line backport/1.4.x backport to 1.4.x release line labels Jun 7, 2023
@shoenig shoenig merged commit 225693a into main Jun 7, 2023
@shoenig shoenig deleted the skip-unreadable-cgroup branch June 7, 2023 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport/1.5.x backport to 1.5.x release line
Projects
None yet
Development

Successfully merging this pull request may close these issues.

panic: during agent shutdown with drain_on_shutdown, got this panic
2 participants