option to manage volume permissions

[josemaia]

Hello,

In Kubernetes/Openshift, when you mount a volume (through CSI or otherwise), you can configure its security context, and there determine, among other things, what Linux user and group the volume will be mounted as:

By default, Kubernetes recursively changes ownership and permissions for the contents of each volume to match the fsGroup specified in a Pod's securityContext when that volume is mounted.

Is there anything similar in Nomad? As far as I can tell, the only alternative is something like running the following pretask to ensure the configuration is what you want:

    task "prep-disk" {
      driver = "docker"
      volume_mount {
        volume      = "nexus-volume"
        destination = "/nexus-data/"
        read_only   = false
      }
      config {
        image        = "busybox:latest"
        command      = "sh"
        args         = ["-c", "chown -R 200:200 /nexus-data/"]
      }
      resources {
        cpu    = 200
        memory = 128
      }

      lifecycle {
        hook    = "prestart"
        sidecar = false
      }
    }

If this is the recommended way, is there any way it could be added as an example in the documentation? It would appear to me to be a relatively common use case. Thanks!

[tgross]

Hi @josemaia! I noticed this same problem with Sonatype Nexus container in the other issue we're working on with you. What you have here is unfortunately the only way to do this right now. I'm not entirely convinced that K8s is really doing the right thing here in allowing the job operator to recursively change ownership on the volume by default, but we'd need to look into it a bit. I'm going to mark this as an enhancement for future storage work.

[m1keil]

Just hit this wall myself. It would be really nice if this would be documented somewhere.

[zhiguangwang]

Most stateful workloads whose docker image run as non-root user would hit this issue. For example:

• prometheus
• grafana
• loki
• elasticsearch

[apollo13]

I'm not entirely convinced that K8s is really doing the right thing here in allowing the job operator to recursively change ownership on the volume by default, but we'd need to look into it a bit.

Me neither, that seems to be able to cause more problems than it is supposed to fix. Also recursively changing might add quite a bit of time to the pre-start operations (especially for large volumes). The better option (where possible) is imo to supply this information to the CSI plugin, like one can do with my nfs plugin for this exact reason: https://gitlab.com/rocketduck/csi-plugin-nfs/-/blob/main/nomad/example.volume -- this way it will create the volume with the proper modes and uid/gid.

[tgross]

The better option (where possible) is imo to supply this information to the CSI plugin, like one can do with my nfs plugin for this exact reason: https://gitlab.com/rocketduck/csi-plugin-nfs/-/blob/main/nomad/example.volume -- this way it will create the volume with the proper modes and uid/gid.

Agreed.

The other thing that comes up with this feature request, which has been on my mind of late, is user namespace remapping. Who "owns" the uid/gid here? The plugin is what does the mounting and any chown, but the plugin has either the host's uid/gid (if configured as we currently require) or its own uid/gid map (if configured with userns remapping itself), neither of which are the uid/gid map in the task that mounts the volume, much less some other task in another job entirely!

[apollo13]

True, namespace remapping is becoming more & more common and there is no easy solution to that. Sure recursive chmod/chown is an option, but if at all it should be optional since doing that by default can often be unnecessary or simply be wrong. And it still leaves the question about who should be doing the chown, because as you said different tasks in a group might have different uids etc…

Supplying a uid/gid during volume creation is not perfect either (I guess most CSI plugins don't support it, but then again many CSI plugins simply do not work easily with nomad either ;)).

[apollo13]

More information on what k8s does:
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods
as well as their support for pushing that down into the CSI layer:
https://kubernetes-csi.github.io/docs/support-fsgroup.html

[acaloiaro]

I'm not entirely convinced that K8s is really doing the right thing here in allowing the job operator to recursively change ownership on the volume by default, but we'd need to look into it a bit. I'm going to mark this as an enhancement for future storage work.

I agree, this behavior doesn't seem entirely correct. While at the same time, as a task author it is highly desirable to not have to think about how uid/gids map from the host that bootstrapped the volume's permissions, to the container/system needing access.

My very rough idea would be to have an ACL that controls whether tasks claiming a volume can designate the owning uid/guid of the files contained within. This may open up a cross-platform can of worms not worth opening.

And there are performance considerations as well. Recursive chowning is not particularly efficient.

[Lindsay-Mathieson]

Any progress with this? Currently it makes the ceph-csi driver impossible to use with MS Sql Server as it runs as user mssql:10001 with no permissions to the csi folder.

Edit: Got it to work with @josemaia task prep work around, thanks!

[116davinder]

Hopefully, this will be fixed or documented somewhere properly.

[tgross]

@116davinder this is a feature request. That a feature doesn't exist isn't something to document (an infinite number non-existing features aren't documented either). In any event, the problem to solve is non-trivial and isn't currently on the roadmap. The issue will be updated if we decide to work on it.

[116davinder]

@tgross, if users like me have to implement these hacky solutions then it is a problem for me to consider using nomad.
I completely, understand that it is non-trival solution as you mentioned but not selected within 2-3 years of development, doesn't seem right to me at least.

[zaphar]

I just ran into this as well. It would be really great if there was an officially supported way to do this. It's probably the only downside I have right now to using Nomad instead of k8s. And I really would prefer to use Nomad.

[kaspergrubbe]

Hello from 2024 :) I just hit this too!

[zip-fa]

same here

[tgross]

Folks, please just do an emoji reaction on the issue description (ex. 👍) if you'd just like to +1 the feature without additional context. That way you're not sending a notification to a few dozen people.