Unable to use absolute host volume with Windows/Docker

[marcjonesuk]

Nomad version

Nomad v0.7.1 (0b295d3)

Operating system and Environment details

Windows Server Core 2016/Windows 10

Issue

I am trying to use an absolute path for a volume on the host but it looks like nomad is treating it as relative. I've tried various permutations for the volume string now but none of them have worked.

I am trying to achieve the equivalent of:

 docker run -v "c:\test:c:\test" [image]

Reproduction steps

Dockerfile:

FROM microsoft/nanoserver
RUN mkdir c:\test
CMD ping google.com

Job:

job "test" {
  datacenters = ["dc1"]
  type = "service"
  group "testgroup" {
    count = 1
    task "testtask" {
      driver = "docker"

      config {
	    image = "[docker image built above]"
		volumes = [
		  "C:\\test:C:\\test"
		]
	  }
    }
  }
}

Nomad Server logs (if appropriate)

2018-03-02T08:21:07.996Z [DEBUG] plugin: plugin process exited: path=C:\Users\marc.jones\Downloads\nomad_0.7.1_windows_amd64\nomad.exe
    2018/03/02 08:21:07.996544 [WARN] client: failed to start task "testtask" for alloc "6d8423f0-1f85-0f92-cbd3-992d6fc5f073": Failed to create container: API error (500): {"message":"invalid volume specification: 'C:\\Users\\marc.jones\\AppData\\Local\\Temp\\NomadClient253893578\\6d8423f0-1f85-0f92-cbd3-992d6fc5f073\\testtask\\C:\\test:c:\\test'"}

[schmichael]

Seems likely the bug lies in this line of code splitting on the : in C:\ instead of the middle colon, so we treat the path as relative instead of absolute (hence the prepended task dir in the log output).

[shorne2017]

Within a Windows 2016 Docker Host machine, when I specify a Volume within the Nomad job, it errors out no matter what I try in the specification. So my question is, what is the correct way to specify a volume in Nomad for Windows?

Specifically, here's what I'm successfully able to do with a straight-up Docker run command (this works perfectly fine):
<docker.exe run -v 'D:\DC-test\ContainerLogs:"C:\logs"' $ImageRepro"/"$ImageName":"$ImageVersion>

However when I setup a Nomad job script this way it fails:
...
volumes = [
"D:\DC-test\ContainerLogs:"C:\logs"
]
(the above really hass two back-slashes even though it's displaying a single back-slash)

It fails with this error:
failed to start task "testcontainer" for alloc "b3cbda00-bcf7-6c76-a6a9-495928b2d611": Failed to create container: API error (500): {"message":"invalid bind mount spec "c:\\program files\\nomad\\data\\alloc\\b3cbda00-bcf7-6c76-a6a9-495928b2d611\\testcontainer\\D:\\DC-test\\ContainerLogs:C:\\logs": invalid volume specification: 'c:\program files\nomad\data\alloc\b3cbda00-bcf7-6c76-a6a9-495928b2d611\testcontainer\D:\DC-test\ContainerLogs:C:\logs'"}

(most of the back-slashes in the above are really four back-slashes even though it is only displaying two)

Notice that part, "invalid bind mount spec" - what is the correct way to specify this in Windows?

Thanks,
--Steve

[sujeetsm]

Hi @schmichael I am facing the exact same issue ..

1. It doesnt allow absolute path to be specified .. what ever path we mention in volumes tag it is basically appended to the path for Task which is basically the temp path create for that task
   C:\\Users\\UserName\\AppData\\Local\\Temp\\NomadClient952618186\\30910d5a-a99c-cf9d-d293-ddfb8a4a83be\\elastic\
   So If I was to give
   C:\\data:C:\\DataOnContainer
   it would by default append C:\Data to the temp path above and ... which it wont find obviously,

And then since it is always looking for the volume to be mounted in the temp folder user can basically never provide the correct path.
Do let know if their is way to mount volumes on windows using windows-container mode

Thanks
Sujeet

Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:05:28 2018
 OS/Arch:           windows/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.24)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:23:19 2018
  OS/Arch:          windows/amd64
  Experimental:     true

[angrycub]

We have identified the root cause of this issue and it is a bit more complicated than suspected when we initially commented on the GH issue.

There is commentary on the moby project that discusses the issue with using the -v style syntax with absolute paths on Windows boxes. This also led us to the resolution of the issue which is to add bind mount support to the Docker task driver's mount stanza.

However, this fix is more complicated than string parsing and requires new unit testing and an implementation that honors the current configuration behavior. Because of that, it requires significant changes to the docker driver. #4761 was submitted for Engineering review; however, the PR is currently blocked by a task driver refactor that is a key component of Nomad v0.9.0.

Once this refactor is complete, we will then be able to refactor the existing PR to incorporate these changes into the new version of the Docker driver.

[antoine-choimet-cbp]

Still got the problem in nomad 9.2.0 and windows 10 enterprise with docker windows containers.

[endocrimes]

@antoine-choimet-cbp The docker volumes configuration updates here should mean that 9.{1,2} support docker volumes on windows.

Could you provide logs and a repro case and we can take a look soon?

Thanks!

[antoine-choimet-cbp]

@endocrimes Hello, here the logs :

2019-06-06T11:42:05.893+0200 [ERROR] client.driver_mgr.docker: failed to create container: driver=docker error="API error (500): invalid volume specification: 'c:\temp\client2\alloc\b23f0c2a-35c5-0164-56ca-232270ed0bd0\alloc:c:\alloc'"
2019-06-06T11:42:05.893+0200 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=b23f0c2a-35c5-0164-56ca-232270ed0bd0 task=redis error="failed to create container: API error (500): invalid volume specification: 'c:\temp\client2\alloc\b23f0c2a-35c5-0164-56ca-232270ed0bd0\alloc:c:\alloc'"
2019-06-06T11:42:05.897+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=b23f0c2a-35c5-0164-56ca-232270ed0bd0 task=redis reason="Error was unrecoverable"

It's from the example to run the cluster with the redis use case, here the server.hcl

log_level = "DEBUG"  
data_dir = "c:/temp"  
server {  
  enabled = true  
  bootstrap_expect = 1  
}

And the example.nomad from nomad job init

for the versions :
$ docker -v Docker version 18.09.2, build 6247962
$ nomad -v Nomad v0.9.2 (028326684b9da489e0371247a223ef3ae4755d87)

[fairmonk]

This type of mapping is working in nomad v1.4.3

[tgross]

Thanks @fairmonk. I think this is safe to close then.