don't work volume option subpath in docker swarm mode

[bobro99]

Hi all!
I use docker swarm to access the directory inside the created volume (volume is test_volume, direcotry is test). Docker swarm consists of one machine, on which the volume is located. I use the same compose-file. docker-compose up mounts the required volume folder inside the container
docker stack deploy --prune --compose-file docker-compose.yml test_2 This command only mounts the volume itself, but not the folder inside it, as I would expect from the command docker run --mount 'type=volume,src=test_volume,dst=/srv,volume-subpath=test' -it --entrypoint sh alpine:3.18.5
My docker-compose.yml

version: '3.7'
volumes:
  test_volume:
    external: true

services:
  alpine-test:
    image: alpine:3.18.5
    deploy:
      placement:
        constraints:
          - node.hostname == mynodename
      restart_policy:
        condition: on-failure
    volumes:
      - type: volume
        source: test_volume
        target: /srv
        # volume options
        volume:
          subpath: test
    command: "tail -f /dev/null"

docker info:

# docker system info
Client: Docker Engine - Community
 Version:    26.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 6
  Running: 1
  Paused: 0
  Stopped: 5
 Images: 1
 Server Version: 26.1.1
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: lhxizmfg9fx9qnk0v3h07d2o3
  Is Manager: true
  ClusterID: oma6m5qjj3golwzivz86smz79
  Managers: 1
  Nodes: 1
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 172.30.3.192
  Manager Addresses:
   172.30.3.192:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 3.10.0-1160.25.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 11.58GiB
 Name: mynodename
 ID: b822c40a-2837-40a0-b3b1-507850527b12
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 54
  Goroutines: 179
  System Time: 2024-05-07T22:12:55.242252032+03:00
  EventsListeners: 1
 Username: dubzap
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[DEPRECATION NOTICE]: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
In future versions this will be a hard failure preventing the daemon from starting! Learn more at: https://docs.docker.com/go/api-security/
[DEPRECATION NOTICE]: API is accessible on http://0.0.0.0:2376 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
In future versions this will be a hard failure preventing the daemon from starting! Learn more at: https://docs.docker.com/go/api-security/

[bohdan-shulha]

I have a similar issue. Any guidance on what is wrong?

docker volume subpath starting container failed cannot access path /var/lib/docker/volumes/<path> lstat /var/lib/docker/volumes/<path> no such file or directory

(this is true, the subpath doesn't exist, but I expect Docker to create one).

I was getting this error on Docker for Desktop (Mac M1) and now on the real Linux machine with the latest Docker.

OrbStack works fine on M1.

In my case, I have crafted the API call myself (i.e. without using the Docker CLI):

Jun 11 16:24:52 us-east-1 ptah_agent[1439623]: 16:24:52.924 [debug] DOCKER REQUEST: %{body: %{Name: "ptah_caddy", EndpointSpec: %{Ports: [%{Protocol: "tcp", PublishedMode: "ingress", PublishedPort: 80, TargetPort: 80}, %{Protocol: "tcp", PublishedMode: "ingress", PublishedPort: 443, TargetPort: 443}]}, Mode: %{Replicated: %{Replicas: 1}}, TaskTemplate: %{ContainerSpec: %{Env: ["CADDY_ADMIN=0.0.0.0:2019"], Hostname: "caddy.ptah", Image: "caddy:2.8-alpine", Mounts: [%{Target: "/data", Type: "volume", Source: "ptah_caddy", VolumeOptions: %{Subpath: "data"}}, %{Target: "/config", Type: "volume", Source: "ptah_caddy", VolumeOptions: %{Subpath: "config"}}]}, Networks: [%{Target: "ptah-net", Aliases: ["caddy.ptah"]}], Placement: %{Constraints: ["node.id == dprtsc18phm839ah6yndyf73b"]}}}, method: :post, url: "/services/create", status_map: %{}}

Jun 11 16:24:52 us-east-1 ptah_agent[1439623]: 16:24:52.930 [debug] DOCKER RESPONSE: status: 201, body: "{\"ID\":\"vb4cvzs07pnuanr1fqri19zua\",\"Warnings\":null}\n"

[bohdan-shulha]

Related issues:

• New volume mount subpath - does not create sub directory if it doesn't ee moby/moby#47842

[rkj]

+1 here, surprised this happens. It works fine with docker compose, but the same config when used with a docker stack deploy ignores subpath and mounts top level of the volume.

Is this because of old compose spec? I do not see subpath mentioned in https://github.com/docker/compose/blob/v1/docs/Compose%20file%20reference%20(legacy)/version-3.md#volume-configuration-reference

[david-arteaga]

I also ran into this same behavior.

Is there any workaround to get the same behavior of using a subpath when mounting a volume?
My use case (though I assume most are similar) is having a volume with multiple directories and mounting only a single directory into another container B (which is exactly what subpath is supposed to be for).

Container B should only have access to the directory under the subpath, not to the whole volume.

[Kolahzary]

I just spent the whole day trying to figure out why my stack is not working as I expected, and at the end I realized it's about docker ignoring "subpath".
I ended up creating 7 different volumes attached to different directories of the same AWS EFS instance and binding each of them to one of the folders in my container.

services:
  nifi:
    image: apache/nifi:2.0.0
    volumes:
      - efs-nifi-logs:/opt/nifi/nifi-current/logs
      - efs-nifi-flowfile_repository:/opt/nifi/nifi-current/flowfile_repository
      - efs-nifi-database_repository:/opt/nifi/nifi-current/database_repository
      - efs-nifi-content_repository:/opt/nifi/nifi-current/content_repository
      - efs-nifi-provenance_repository:/opt/nifi/nifi-current/provenance_repository
      - efs-nifi-state:/opt/nifi/nifi-current/state
      - efs-nifi-conf:/opt/nifi/nifi-current/conf

volumes:
  efs-nifi-logs:
    driver_opts:
      type: nfs4
      o: 'nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,addr=x.x.x.x,rw'
      device: ':/nifi/logs'
  efs-nifi-flowfile_repository:
    driver_opts:
      type: nfs4
      o: 'nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,addr=x.x.x.x,rw'
      device: ':/nifi/flowfile_repository'
  efs-nifi-database_repository:
    driver_opts:
      type: nfs4
      o: 'nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,addr=x.x.x.x,rw'
      device: ':/nifi/database_repository'
  efs-nifi-content_repository:
    driver_opts:
      type: nfs4
      o: 'nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,addr=x.x.x.x,rw'
      device: ':/nifi/content_repository'
  efs-nifi-provenance_repository:
    driver_opts:
      type: nfs4
      o: 'nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,addr=x.x.x.x,rw'
      device: ':/nifi/provenance_repository'
  efs-nifi-state:
    driver_opts:
      type: nfs4
      o: 'nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,addr=x.x.x.x,rw'
      device: ':/nifi/state'
  efs-nifi-conf:
    driver_opts:
      type: nfs4
      o: 'nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,addr=x.x.x.x,rw'
      device: ':/nifi/conf'

• there are lots of other files in the /opt/nifi/nifi-current directory that should not be persisted, only the listed folders is persistable.

Are there any plans to start supporting subdirectories for swarm?

[david-arteaga]

I at least think there should be a warning message when deploying a stack that has a volume with subpath to docker swarm saying it will be ignored.
Or at least mention it explicitly in the docs next to the subpath option. It’s really not a great experience debugging something for a long time only to find out it was never going to work.