-
Notifications
You must be signed in to change notification settings - Fork 64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Write logs to fifo from a driver run as non-root user #189
Comments
Hi @rina-spinne 👋 I don't have a lot of experience with running rootless workloads like in Podman, but I imagine you would have to then collect logs outside Nomad. Have you tried using |
Collecting the logs outside nomad might be a workaround for this issue. Can't use journald on void but looks like is possible to disable the log collection and use another tool for that. While this is not a dealbreaker, I wanted to have a quick way of retrieving logs from a single task. Is running the driver as root the only way of achieving this? |
I'm going to transfer this issue over to the podman driver repo, as I don't think this is something we can handle in Nomad itself. |
Hi, Nomad 1.4.3 nomad config podman.hcl
When I try the example podman job:
Task failed:
I can run any podman container, it works fine. `podman
For me revelant logs:
I cannot find solution to add conmon in /sys/fs/cgroup/cgroup.subtree_control. I don't know! I also try Nomad this podman in root mode, it works! Does someone already run Nomad with Podman in rootless ? Thanks! |
Hi @fred-gb 👋 I believe your issue may be related to cgroups v2 support. Can you try running with cgroups v1 and see if it fixes the problem? And (quite) late but Happy New Year for you as well 😄 |
@rina-spinne Have you found a solution for this? I'm currently using |
Nomad version
Nomad v1.2.4 (55e5c49b99a6fd2bf925e7fd98d95829776c331f)
Operating system and Environment details
Void Linux
Issue
While trying to setup the podman driver with a non-root user all containers fail to start due to the podman driver not being able to write on the log fifo files. The nomad client is running as root.
This happens due to both the
<alloc-dir>
and the internal<alloc-dir>/<alloc-id>/alloc/log/.<task>.stdout.fifo
being owned by root and having only user permissions. Even if I grant the podman user the permissions to access to the<alloc-dir>
, the fifos are being created with the limited permissions.What I can get from the driver documentation is that drivers are expected to be able to write to the alloc logs fifo files but because of the permission design it looks like it's not possible to run any driver with a non-root user.
This works well when running podman as root.
Reproduction steps
Start the podman service as a non-root user and try to run any job
Expected Result
Containers starting when running with a podman service with a non-root user
Actual Result
Job fails to start due to podman not being able to write to log's fifo
Job file (if appropriate)
https://github.com/hashicorp/nomad-driver-podman/blob/main/examples/jobs/redis_deprecated.nomad
Nomad Client logs (if appropriate)
Relevant part of the podman logs
The text was updated successfully, but these errors were encountered: