Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

alloc fs doesn't have access to mounted volumes #7365

Open
tgross opened this issue Mar 17, 2020 · 4 comments
Open

alloc fs doesn't have access to mounted volumes #7365

tgross opened this issue Mar 17, 2020 · 4 comments

Comments

@tgross
Copy link
Member

tgross commented Mar 17, 2020

The various Nomad filesystem commands (and their APIs) don't have access to the contents of mounted volumes. This means that while tasks can access host volume content (and CSI volume content), nomad alloc fs cannot. The operator can view the files with nomad alloc exec, but as this requires a higher level of privilege with its own ACL, this workaround isn't great.

Example jobspec that writes to a volume mount location in the task dir. (The exact location doesn't matter so long as its somewhere within the alloc dir.)

# a job that mounts an EBS volume and writes its alloc ID as a file
job "use-ebs-volume" {
  datacenters = ["dc1"]

  group "group" {
    volume "test" {
      type   = "csi"
      source = "ebs-vol0"
    }

    task "task" {
      driver = "docker"

      config {
        image   = "busybox:1"
        command = "/bin/sh"
        args    = ["-c", "touch /local/test/${NOMAD_ALLOC_ID}; sleep 3600"]
      }

      volume_mount {
        volume      = "test"
        destination = "${NOMAD_TASK_DIR}/test"
        read_only   = false
      }

      resources {
        cpu    = 500
        memory = 128
      }
    }
  }
}

Results below. Note that this all works as expected when there's no volume_mount stanza.

▶ nomad alloc fs d7db81c8 task/local
Mode        Size     Modified Time         Name
drwxr-xr-x  4.0 KiB  2020-03-17T15:22:46Z  test/

▶ nomad alloc fs d7db81c8 task/local/test
Mode  Size  Modified Time  Name

Exec into the allocation and see everything we'd expect to see:

▶ nomad alloc exec d7db81c8 /bin/sh
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                   7.7G      3.7G      4.0G  49% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     1.9G         0      1.9G   0% /sys/fs/cgroup
shm                      64.0M         0     64.0M   0% /dev/shm
/dev/xvda1                7.7G      3.7G      4.0G  49% /local
tmpfs                     1.0M         0      1.0M   0% /secrets
/dev/xvda1                7.7G      3.7G      4.0G  49% /alloc
/dev/xvdba               39.2G     48.0M     39.2G   0% /local/test
/dev/xvda1                7.7G      3.7G      4.0G  49% /etc/resolv.conf
/dev/xvda1                7.7G      3.7G      4.0G  49% /etc/hostname
/dev/xvda1                7.7G      3.7G      4.0G  49% /etc/hosts
tmpfs                     1.9G         0      1.9G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/timer_stats
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                     1.9G         0      1.9G   0% /proc/scsi
tmpfs                     1.9G         0      1.9G   0% /sys/firmware

/ # cd /local/test
/local/test # ls -lah
total 24K
drwxr-xr-x    3 root     root        4.0K Mar 17 15:22 .
drwxrwxrwx    3 nobody   nogroup     4.0K Mar 17 15:22 ..
-rw-r--r--    1 root     root           0 Mar 17 15:13 5564b554-893e-41e3-8d2b-62316c35dbf6
-rw-r--r--    1 root     root           0 Mar 17 15:07 5f54d08e-2fdf-9b4b-2f6b-4a7b52cb8e9f
-rw-r--r--    1 root     root           0 Mar 17 15:11 68ad6e66-5da4-04d2-0ede-ec9c96d6d204
-rw-r--r--    1 root     root           0 Mar 17 14:31 b7f43bb5-15a3-b8ee-ed44-0bc3156e8f78
-rw-r--r--    1 root     root           0 Mar 17 15:04 d56136d6-550b-c379-1d2f-a29207a3232d
-rw-r--r--    1 root     root           0 Mar 17 15:22 d7db81c8-4c4c-a10c-fa47-1e7ee2e5854d
-rw-r--r--    1 root     root           0 Mar 17 14:59 e7f39ca6-6df4-f4b8-13c3-61ca2c00c209
-rw-r--r--    1 root     root           0 Mar 17 15:15 ed9a2dd6-bbe7-9498-dfac-6a0dc1bf6ab0
drwx------    2 root     root       16.0K Mar 17 14:31 lost+found

@tgross tgross added this to the 0.11.0 milestone Mar 17, 2020
@tgross tgross self-assigned this Mar 17, 2020
@tgross tgross removed their assignment Mar 25, 2020
@tgross tgross removed this from the 0.11.0 milestone Mar 30, 2020
@tgross tgross removed the volumes label Mar 31, 2020
@tgross tgross added this to the 0.11.1 milestone Apr 9, 2020
@tgross tgross modified the milestones: 0.11.1, 0.11.2 Apr 22, 2020
@tgross
Copy link
Member Author

tgross commented May 8, 2020

Potentially interrelated: #7799

@tgross tgross assigned tgross and unassigned tgross May 26, 2020
@langmartin langmartin self-assigned this May 31, 2020
@tgross tgross added this to the unscheduled milestone Jul 9, 2020
@tgross
Copy link
Member Author

tgross commented Oct 7, 2020

This turns out to be another case of the mounting behaviors described in #8919

@tgross
Copy link
Member Author

tgross commented Nov 5, 2020

While working on some E2E test improvements following the 0.12.7 security patch and landing those docs, I've found that the volumes themselves are visible but their contents are not. This may be similar to the handling of the secrets directory. Going to re-open this.

@tgross tgross reopened this Nov 5, 2020
@tgross tgross removed this from the unscheduled milestone Nov 5, 2020
@tgross
Copy link
Member Author

tgross commented Jan 19, 2021

Turns out this is a long-known issue with the file system API: #914

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants