Mounted task directory volumes empty inside the container

[sssilver]

Nomad version

Nomad v1.4.3 (f464aca)

Operating system and Environment details

Ubuntu 22.04.1 LTS
Linux ironforge 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Issue

Volumes mounting ${NOMAD_TASK_DIR} into another directory inside the container are empty.

Reproduction steps

1. Define a job that renders a template into ${NOMAD_TASK_DIR}
2. Mount ${NOMAD_TASK_DIR} into any path inside the container using volumes or mount stanzas in task.config stanza
3. exec into the container to observe that the directory is empty
4. Confirm that the rendered template exists inside of ${NOMAD_TASK_DIR}

Expected Result

The rendered template is accessible inside the container in the mounted directory outside of ${NOMAD_TASK_DIR}.

Job file (if appropriate)

job "writefreely" {
  datacenters = ["aus01"]

  group "writefreely" {
    service {
      name     = "writefreely"
      provider = "nomad"
      port     = "http"

      tags = ["ingress"]
    }

    network {
      port "http" {
        to = 8080
      }
    }

    ephemeral_disk {
      size = 200
    }

    task "writefreely" {
      driver = "docker"

      config {
        image = "algernon/writefreely:0.13.2-1"
        ports = ["http"]

        volumes = ["data:/mounted_data"]

        // This does not work as well:
        // mount {
        //   type   = "bind"
        //   source = "${NOMAD_TASK_DIR}/data"
        //   target = "/mounted_data"
        // }
      }

      template {
        destination = "${NOMAD_TASK_DIR}/data/config.ini"
        change_mode = "restart"
        data = "# Sample content"
      }
    }
  }
}

In Nomad:

$ nomad fs 3f5dc70f /writefreely/local/data

Mode        Size   Modified Time              Name
-rw-r--r--  507 B  2022-12-07T22:51:43-06:00  config.ini

The file is there, as expected!

Inside the Docker container:

$ docker exec -it bf9ceda0d94d /bin/sh

/writefreely # ls -al /volume_test/
total 8
drwxr-xr-x    2 root     root          4096 Dec  8 04:51 .
drwxr-xr-x    1 root     root          4096 Dec  8 04:51 ..

The file isn't there although /volume_test exists -- unexpected!

[the-maldridge]

I think you're missing a local/ in your mount config, and what's happening here is that you're creating a new top level directory data next to local and secrets rather than mounting the one your template is in which is local/data.

[sssilver]

Interesting.

volumes = ["${NOMAD_TASK_DIR}/data:/volume_test"] creates, to my great surprise, a directory named config.ini inside of /volume_test, instead of the actual file with the actual contents.

volumes = ["${NOMAD_TASK_DIR}/data/config.ini:/volume_test/config.ini"] results in an empty /volume_test, as in the initial issue.

If I replace the volumes stanza with mount, I get an empty directory named config.ini again.

        mount {
          type   = "bind"
          source = "${NOMAD_TASK_DIR}/data"
          target = "/volume_test"
        }

Inside my container:

# stat /volume_test/config.ini/

  File: /volume_test/config.ini/
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: 811h/2065d	Inode: 14811141    Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-07 06:44:23.223749057 +0000
Modify: 2022-12-07 06:43:54.059707652 +0000
Change: 2022-12-07 06:43:54.059707652 +0000

# ls -n /volume_test/config.ini
total 0

[sssilver]

Some more research suggests that:

The underlying reason causing the file being shared with -v to appear as a directory instead of a file is that Docker could not find the file on the host. So Docker creates a new directory in the container with the name being the name of the non existing file on the host as docker thinks that the user just want to share a volume/directory that will be created in the future.

But given that I haven't even mentioned mounting config.ini file and only specified the directory in which it's located, what is this if not poltergeist?

[tgross]

Hi @sssilver! To help explain this, let's look at the following job, which serves the index.html contents as you'd expect:

job "example" {
  datacenters =  ["dc1"]

  group "web" {

    network {
      mode = "bridge"
      port "www" {
        to = 8001
      }
    }

    task "http" {

      driver = "docker"

      config {
        image   = "busybox:1"
        command = "httpd"
        args    = ["-v", "-f", "-p", "8001", "-h", "/var/www"]
        ports   = ["www"]
        volumes = [
          "local:/var/www",                 // works: relative path
          "${NOMAD_TASK_DIR}:/var/taskdir", // doesn't work: abs path
          "data:/var/data",                 // doesn't work: wrong relative path
        ]

      }

      template {
        data        = "<html>hello, world</html>"
        destination = "local/index.html"
      }

      template {
        data        = "<html>goodbye, world</html>"
        destination = "${NOMAD_TASK_DIR}/data/goodbye.html"
      }

      resources {
        cpu    = 128
        memory = 128
      }

    }
  }
}

Note that the template destination is relative to the task working directory. If you look at the Filesystem concepts docs you'll see this section:

«taskname»/local/: This directory is the location provided to the task as the NOMAD_TASK_DIR. Note this is not the same as the "task working directory". This directory is private to the task.

So for the job above, the allocation directory looks like this:

├── alloc
│   ├── data
│   ├── logs
│   └── tmp
└── http
    ├── local
    │   ├── index.html
    │   └── data
    │       └── goodbye.html
    ├── secrets
    └── tmp

$ nomad alloc fs d133 http/local
Mode        Size  Modified Time              Name
drwxr-xr-x  3 B   2023-01-03T13:36:22-05:00  data/
-rw-r--r--  25 B  2023-01-03T13:36:22-05:00  index.html

$ nomad alloc fs d133 http/local/data
Mode        Size  Modified Time              Name
-rw-r--r--  27 B  2023-01-03T13:36:22-05:00  goodbye.html

Now let's look at our volumes configuration:

• "local:/var/www": this works fine because it's a relative path from local, as @the-maldridge has pointed out.
• "${NOMAD_TASK_DIR}:/var/taskdir": this doesn't work because it's been given $NOMAD_TASK_DIR, which is an absolute path because it's intended to be consumed by the workload inside the container. The volume source is an absolute path to /local on the host, which you almost certainly don't have.
• "data:/var/data": this doesn't work because data is relative to the task directory, not relative to the NOMAD_TASK_DIR that's under it at local.

This all results in the following mounts on the container:

$ docker inspect fdc | jq '.[0].Mounts'
[
  {
    "Type": "bind",
    "Source": "/srv/nomad/data/alloc/74b4126b-927c-a4a0-a7e9-9faf530f8a07/http/data",
    "Destination": "/var/data",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "bind",
    "Source": "/srv/nomad/data/alloc/74b4126b-927c-a4a0-a7e9-9faf530f8a07/hosts",
    "Destination": "/etc/hosts",
    "Mode": "",
    "RW": true,
    "Propagation": "private"
  },
  {
    "Type": "bind",
    "Source": "/srv/nomad/data/alloc/74b4126b-927c-a4a0-a7e9-9faf530f8a07/alloc",
    "Destination": "/alloc",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "bind",
    "Source": "/srv/nomad/data/alloc/74b4126b-927c-a4a0-a7e9-9faf530f8a07/http/local",
    "Destination": "/local",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "bind",
    "Source": "/srv/nomad/data/alloc/74b4126b-927c-a4a0-a7e9-9faf530f8a07/http/secrets",
    "Destination": "/secrets",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "bind",
    "Source": "/srv/nomad/data/alloc/74b4126b-927c-a4a0-a7e9-9faf530f8a07/http/local",
    "Destination": "/var/www",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "bind",
    "Source": "/local",
    "Destination": "/var/taskdir",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  }
]

The last mystery is the phantom config.ini directory, which as you've pointed out, Docker knows nothing about. Note in the Filesystem docs that templates are rendered before the task is started by the task driver (which includes creating all the bind-mounts).

So the template is getting rendered at $datadir/alloc/$allocdir/writefreely/local/data/config.ini and then a bind mount is getting created with a source from /local/data on the host. I wasn't able to reproduce the phantom directory with the following:

...
        volumes = [
          "${NOMAD_TASK_DIR}/data:/var/www",
        ]
      }

      template {
        data        = "<html>hello, world</html>"
        destination = "${NOMAD_TASK_DIR}/data/index.html"
      }
...

But again, use the right paths and it shouldn't really matter.

[tgross]

Looks like this has been answered as expected behavior (given the right paths), so I'm going to close this out.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.