Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: pkg/rootless: do not use shortcut with --tmpdir #18057

Closed
wants to merge 1 commit into from

Conversation

Luap99
Copy link
Member

@Luap99 Luap99 commented Apr 5, 2023

When using --tmpdir for the podman cli we use this as location for the pause.pid file. However the c shortcut code has no idea about this option and always assumes XDG_RUNTIME_DIR/libpod/tmp. This can cause us to join the wrong user namesapce which leads to very weird issues as mounts are missing there.

Fixes #17903

Does this PR introduce a user-facing change?

None

When using --tmpdir for the podman cli we use this as location for the
pause.pid file. However the c shortcut code has no idea about this
option and always assumes XDG_RUNTIME_DIR/libpod/tmp. This can cause us
to join the wrong user namespace which leads to very weird issues as
mounts are missing there.

Fixes containers#17903

Signed-off-by: Paul Holzinger <[email protected]>
@openshift-ci openshift-ci bot added release-note-none approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Apr 5, 2023
@Luap99
Copy link
Member Author

Luap99 commented Apr 5, 2023

@giuseppe PTAL

@edsantiago It might be good to check the integration test runtime to see if it will be slower? Without the shortcut we need to rexec every time which may be noticeable.

@vrothberg
Copy link
Member

Could we move the pause.pid to another location (i.e., not under TMP)?

@Luap99
Copy link
Member Author

Luap99 commented Apr 5, 2023

Could we move the pause.pid to another location (i.e., not under TMP)?

I would need to figure what commit ab88632 really tried to fix with that.

cc @mheon

@mheon
Copy link
Member

mheon commented Apr 5, 2023 via email

@Luap99
Copy link
Member Author

Luap99 commented Apr 5, 2023

That wasn’t a behavior change (just changed how we got the final path, the
intended destination did not change)

No it changed the path, likely not intentionally but the previous path was $runtimedir/libpod/pause.pid, where runtimedir was either $XDG_RUNTIME_DIR or /tmp/podman-run-$UID and your commit changed it to $runtimedir/libpod/tmp/pause.pid.
Commit 11badab shows that the path was changed to $runtimedir/libpod/tmp/pause.pid because it needed to be fixed in the c code to reflect that.

But the big difference is that the new behavior now also respected --tmpdir while the previous one never did. So the fact is that this commit changed it from one pause process to one per tmpdir expect the the tmpdir thing never worked correctly with the c code.

If you have two Libpods with different tmpdirs on the
same user, is it important that both have different user namespaces?

I do not know, the only problem would be if one is using podman system migrate kill the pause process whikle other containers are still running but there should be a fallback to join a container namespace in any case so it likely won't matter.
It would also be much better to only have one because with this change we are very likely leaking thousands of pause processes in the integration tests.

@Luap99 Luap99 changed the title pkg/rootless: do not use shortcut with --tmpdir WIP: pkg/rootless: do not use shortcut with --tmpdir Apr 5, 2023
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 5, 2023
Copy link
Member

@giuseppe giuseppe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Apr 6, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: giuseppe, Luap99

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Luap99
Copy link
Member Author

Luap99 commented Apr 6, 2023

I am working on patch to only use a single pause process, this should be much better otherwise we leak many processes in CI. Also this sort of keeps the current behaviour assuming the --tmpdir command was made after a normal podman command without that flag.

@giuseppe
Copy link
Member

giuseppe commented Apr 6, 2023

The root issue is that we are trying to be too clever in how we compute $XDG_RUNTIME_DIR when it is not set and this is problematic, especially from the C code.

would it be simpler if we use only $XDG_RUNTIME_DIR/libpod/tmp/pause.pid without taking into account tmpdir? If the C code doesn't find that file, it won't join the namespace immediately and we will do it from the Go code.

@Luap99
Copy link
Member Author

Luap99 commented Apr 6, 2023

Yes lets go with #18083 which should fix it in a better way, given that we never need more than a single pause process.

@Luap99 Luap99 closed this Apr 6, 2023
Luap99 added a commit to Luap99/libpod that referenced this pull request Apr 11, 2023
Currently --tmpdir changes the location of the pause.pid file. this
causes issues because the c code in pkg/rootless does not know about
that. I tried to fix this[1] by fixing the c code to not use the
shortcut. While this fix worked it will result in many pause processes
leaking in the integrration tests.

Commit ab88632 added this behavior but following the disccusion it was
never the intention that we end up having more than one pause process.
The issues that was trying to fix was caused by somthing else AFAICT,
the main problem seems to be that the pause.pid file parent directory
may not be created when we try to create the pid file so it failed with
ENOENT. This patch fixes it by creating this directory always and revert
the change to no longer depend on the tmpdir value.

With this commit we now always use XDG_RUNTIME_DIR/libpod/tmp/pause.pid
for all podman processes. This allows the c shortcut to work reliably
and should therefore improve perfomance over my other approach.

A system test is added to ensure we see the right behavior and that
podman system migrate actually stops the pause process. Thanks to Ed
Santiago for the improved test to make it work for both `catatonit` and
`podman pause`.

This should fix the issues with namespace missmatches that we can see in
CI as flakes.

[1] containers#18057

Fixes containers#18057

Signed-off-by: Paul Holzinger <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 3, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. release-note-none
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pod stats: unknown FS magic on "/run/user/4902/netns/netns-etc-etc"
4 participants