Artifact getter breaks when Nomad is run as non-root

[the-nando]

Nomad version

Nomad v1.5.1+ent

Issue

The artifact getter now runs as a subprocess since v1.5.0. We run nomad as a non-root user and the getter fails with:

fork/exec /usr/bin/nomad: operation not permitted

The error originates from this syscall:

[pid 18934] setgroups(0, NULL <unfinished ...>
[pid 16745] epoll_pwait(4,  <unfinished ...>
[pid 18934] <... setgroups resumed>)    = -1 EPERM (Operation not permitted)

Which comes from setting the GID and UID in SysProcAttr:
https://github.com/hashicorp/nomad/blob/v1.5.1/client/allocrunner/taskrunner/getter/util.go#L140
https://github.com/hashicorp/nomad/blob/v1.5.1/client/allocrunner/taskrunner/getter/util_linux.go#L29

The code was intended to be used to drop privileges and run the getter as nobody when the nomad binary runs as root but that has been reverted in 95359b8.
Unfortunately when nomad is run as a non user, setting the GID for the child doesn't work as the user lacks cap_setgid.
A workaround is:

# setcap 'cap_setgid=ep' /usr/sbin/nomad 

or removing setting the attributes altogether as they are technically non required and they'll be inherited from the parent:

# diff --git a/client/allocrunner/taskrunner/getter/util.go b/client/allocrunner/taskrunner/getter/util.go

index 1a71017f3..8edf5a449 100644
--- a/client/allocrunner/taskrunner/getter/util.go
+++ b/client/allocrunner/taskrunner/getter/util.go
@@ -137,7 +137,7 @@ func (s *Sandbox) runCmd(env *parameters) error {
 	cmd.Stdin = env.reader()
 	cmd.Stdout = output
 	cmd.Stderr = output
-	cmd.SysProcAttr = attributes()
+	// cmd.SysProcAttr = attributes()

 	// start & wait for the subprocess to terminate
 	if err := cmd.Run(); err != nil {

Same issue was addressed here for non-Linux systems: 78d8ae0

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.