Support Linux images in Docker for Windows

[schmichael]

Nomad only supports Docker for Windows 17.03.1 in Windows Container mode. Attempting to run Linux containers will fail because Nomad will create Windows-style volume bind paths instead of Linux-style.

We should:

• Detect Docker for Windows mode in the fingerprinter
  • Requires updating go-dockerclient to support the undocumented Isolation key in the /info API - Support API version >=1.25 for Docker on Windows support fsouza/go-dockerclient#649
    • Isolation: "hyperv" means Windows mode
    • Isolation: "" means Linux mode
  • Add new attr.driver.docker.os_type to expose Docker mode to constraints
• Create volume paths based on the image's OS
  • Requires updating go-dockerclient to support the 1.29 InspectImage API - Support API version >=1.25 for Docker on Windows support fsouza/go-dockerclient#649
• Adjust signal handling to be based on the runtime os, rather than the nomad-client os.

Stretch goal:

• Vary the output of nomad init based on Docker for Windows mode (redis:3.2 for Linux, redis:nanoserver for Windows)

[KZachariassen]

Any news on this one, we could really use this!

[schmichael]

@krjensen I'm afraid we haven't prioritized it on our roadmap yet since a Windows node running Docker in Linux mode doesn't offer any functionality not already provided by running Docker on a Linux node. We were curious if there were enough people running in homogenous Windows environments to make Linux mode on Windows compelling.

[KZachariassen]

I totally understand that.

So why do I need this:
We are a .net developer house and has just begun to look at .net core that will run on Linux.

So every developer use windows 10 with docker, to build .net core applikations. We are amning to use nomad in produktion and there for the need to test it local, so a developer can write and test the .nodmad file. Many services will contain multiple containers, so they need to compose. I would prefer the need for only a nomad definition, that works both local and in production and not a situation where I need both a docker compose and a nomad definition.

[frundh]

We are also a Windows-shop facing the same issue when it comes to local development and testing, so this feature would be a nice addition.

Since this is only required for local development, I have experimented with some alternative solutions/hacks that I can share if anybody is interested:

Docker Image
https://github.com/frundh/nomad-mobyvm

Nomad.exe
https://github.com/frundh/nomad/releases/tag/v0.6.2.1-moby

[thasmo]

Would love to see this supported too, although I don't use it in a business-context.

[bartuer]

It will became pretty common to run Linux docker image on Windows, we seldom run Windows image on Windows inside Microsoft.

[automaticgiant]

just for tracking @schmichael, fsouza/go-dockerclient#649 is done so the remaining work is in-repo, and confined to the listed plan now, right?

[adamsmithkld]

This is even more important now that Docker for Windows supports side-by-side Linux and Windows containers (via the experimental feature). We are also a Windows development shop that builds Linux Docker containers. The only thing missing is Nomad support for Linux containers on Docker for Windows.

[renevo]

Is there any traction on this, currently even with drive sharing in windows, nomad is unable to allocate a job due to permission denied on mounting drives.

2018-08-08T21:12:09.945-0700 [DEBUG] plugin: plugin process exited: path=D:\tools\nomad.exe
    2018/08/08 21:12:09.945889 [WARN] client: failed to start task "redis" for alloc "6354d373-9fd5-c0f3-a96c-09bf0a737f6f": Failed to create container: API error (500): Mount denied:
The source path "D:/nomad.d/nomad/alloc/6354d373-9fd5-c0f3-a96c-09bf0a737f6f/alloc:c"

I am able to mount volumes when running containers natively with docker for windows (linux containers, non-experimental), but nomad isn't able to mount.

PS: I am running nomad on windows, instructing to use the pipefile.

data_dir = "d:\\nomad.d\\nomad"
datacenter = "dc1"

client {
    enabled = true
    network_interface = "Ethernet"

    options {
        "docker.endpoint" = "npipe:////./pipe/docker_engine"
    }
}

server {
    enabled = true

    bootstrap_expect = 1
    data_dir = "d:\\nomad.d\\nomad\\server"
}

I get the error:

failed to start task "redis" for alloc "a7feb856-514e-9d52-fb2b-ca87f55172d5": Failed to create container: API error (500): Mount denied: The source path "D:/nomad.d/nomad/alloc/a7feb856-514e-9d52-fb2b-ca87f55172d5/alloc:c" doesn't exist and is not known to Docker

However, the directory is definately available:

[jzupka]

exactly same to me

[sujeetsm]

upvote this one .. windows now supports homogeneous support for linux/windows containers.

[gblok]

How to make work nomad with Linux images in Docker for Windows?
It is already possible?

[cgbaker]

We don't have any immediate plans to support this, but we are tracking it in our backlog.

[atrauzzi]

Chiming in to say that I've just encountered this issue. A bit disappointed after reading the last comment.

Not being able to run linux containers via nomad on Windows completely blocks what I'm currently working on. Worse still is that the first container I tried to run doesn't even have bind mounts configured!

Microsoft and Docker have been putting a lot of work into ensuring that Docker Desktop gets as much out of the way as possible. Even so far as allowing Windows and linux containers to run at the same time.
It doesn't feel like too much to suggest that nomad add support for this scenario. Even if only to remain consistent with the direction of the ecosystem.

[schmichael]

Even so far as allowing Windows and linux containers to run at the same time.

That's fantastic and was not the case when this ticket was originally written! Originally Docker on Windows was modal and only supported one image type at a time. That may nullify the Add new attr.driver.docker.container_mode to expose Docker mode to constraints checkbox above and make implementation even easier.

[nbpalomino]

I'm also waiting for implementing this feature... currently trying to use Nomad with Docker for Windows on my Windows 10 box...

[ilyaiqoqo]

I'm also waiting I am waiting for this feature. we are running batch jobs that should be agnostic to where they run

[amfern]

I am able to run Linux container when the daemon is in windows containers mode, by enabling experimental features in docker edge.

I think nomad should also support this and not set the driver to an unhealthy state

[ilyaiqoqo]

I am attempting to make lcow work, I've extracted some patches i made into three PRs, would appreciate feedback

• fixes invalid network mode when running linux container on windows driver/docker: use default network mode #5999
• allows specifying relative binding with backslashes or forward slashes driver/docker: convert host bind path to os native #6000
• update destination regex to allow unix destination path in container driver/docker: support unix destination mount path in windows #6001

[karlem]

Is there any update on this issue please?

We're planning to use Nomad for very interesting use case but support of Linux containers on Windows is must.

Thanks

[tgross]

Hi @karlem! The patches we received from @ilyaiqoqo should be enough to allow support, but I admit it's not an area we have well-tested yet. I recently added Windows 2016 (Datacenter) to our end-to-end test suite, but that supports only Windows containers on Windows. We've got upcoming work planned to add Windows 2019 targets to e2e testing that will make sure we've got well-tested support there as well.

(I have some open issues tracking that work on a private GitHub board that I'll move over into this repo shortly so that you can keep an eye on them.)

[karlem]

Hi @tgross!
Thank you very much for your response. That sounds pretty good. I will keep an eye on the issues.

Have a good day!

[atrauzzi]

@tgross - Is that using Docker for Windows (the desktop app)?

[tgross]

No, it's the engine for the server. My understanding is that the underlying virtualization problems to solve are the same so support for Windows 2019 production environments should solve for those desktop use cases as well. But the desktop app isn't likely to be something we'll fully support with testing from a practical standpoint.

[jlarriba]

@lutetium174 The feature target was not WSL2, but LCOW, a very promising feature included in Windows Server 2019 that allowed to run Windows Containers natively along with Linux Containers via the Hyper-V Containers feature. The API of LCOW is very similar to the one used by Docker in Windows, so supporting it took very little effort. Even if it was not the target, I checked (manually, this was never officialy tested as @tgross said) that it also made Nomad work with the Linux Containers feature provided by the Docker Desktop Windows 10 application.

Even if last year LCOW looked like a promising technology and the "way-to-go" for Docker on Windows support, it never left experimental phase, and it seems to look more and more abandoned. Since then, the Docker team focused their efforts for Linux Containers on WSL2. See this issue.

WSL2 however, is a completely different technology that has nothing to do with LCOW, so it would require additional effort to add support to that in Nomad.

[tomqwpl]

I'm also very interested in this functionality. I'm sure my set up is common, Windows 10 running Docker Desktop in WLS2 mode. I've used docker on Windows for some time and have never, ever run a Windows docker image, I've always run docker in Linux mode, scheduling linux containers. For me it's kind of surprising that nomad doesn't just support this, naively I assumed nomad just talked to the local docker daemon to schedule containers.

[tgross]

For me it's kind of surprising that nomad doesn't just support this, naively I assumed nomad just talked to the local docker daemon to schedule containers.

As mentioned several times in this issue, Nomad is shouldn't need any particular support for launching Linux containers on Docker on Windows. It just hasn't been verified. There are lots of weird quirks to the WSL2 environment (ex. a monolithic kernel without modules we were at one point using to detect networking isolation support because that's how virtually every other Linux distro builds their kernel out). So we do want to make sure we have verifiable support for Windows Server 2019 environments. Windows 10 desktops, however, are not a production environment so it's just not likely to ever be a test target for us.

[tomqwpl]

@tgross So if I ignore the warning and try to schedule a linux container anyway, it should work? I've read through the issue but been confused by the status of this.

[lutetium174]

@tomqwpl Not in my experience. The problem I've found is that the docker driver is marked unhealthy and any attempted use of the driver in my jobs fails. You would need to switch to Windows containers.

[tomqwpl]

@lutetium174 Right, that was mind of my impression too. I think hence my confusion here. There seemed to be talk earlier of it just working at some point, but it certainly doesn't appear to now for that reason.

[gaetandezeiraud]

I'm also very interested.
You would need to switch to Windows containers. Don't work for me. Need Windows 10 Pro maybe? (I'm on Home on my new laptop) @lutetium174

[dg-eparizzi]

We have a solution that's a mix of dockerized services (Linux) and some native .NET framework apps. I was under the impression that Nomad would be able to orchestrate both types of jobs without issue but I'm seeing Linux containers are not supported on Windows hosts? Linux containers have been supported in Docker for Windows for a while now so I don't understand why is this an issue for Nomad.

Can someone clarify whether this is a potential approach? We're using Windows 10 Enterprise 2019 LTSC, so we're not even running Docker using WSL2, but Hyper-V instead...

[gthieleb]

I want to showcase to windows guys how easy it is to get started with nomad together with docker to run some linux container workload on a windows box.

Installation of consul / nomad is very easy in windows and in this showcase I'd like to have both of them running in dev mode:

nomad agent -dev
consul agent -dev

Furthermore I'd like to use docker desktop (wsl2 engine) and run nomad/consul in dev mode. But starting a nomad job apparently fails due to the error message of the driver:

Can you shed some light on why this should not be supported/not work? With WSL2 I don't have any chance to either start docker natively or register as a (dedicated linux) nomad client in consul.

[das974]

I has a similar need for showcasing Windows with the WSL 2 Docker backend. I worked around this by running the server in WSL2 and a client on the Windows host. This allows me to run .Net apps on the windows host and Linux Docker containers in the WSL backend.

You basically end up with 2 nodes with the same name.

Linux Nomad Server

Windows Nomad Client

Enabled only raw_exec for Windows

VSCode Tasks For Reference

This assumes you have downloaded nomad and is in the path (I use a separate task for this)

{
    "label": "Nomad WSL2",
    "type": "shell",
    "command": "bash",
    "args": [
        "-c",
        "sudo nomad agent -config=./nomad/linux/config -bind 0.0.0.0"
    ]
},
{
    "label": "Nomad Windows Client",
    "type": "shell",
    "command": "nomad",
    "args": [
        "agent",
        "-config=${workspaceFolder}\\nomad\\windows\\config"
    ]
}

[qagwaai]

any updates on this?

[FibreFoX]

I find it a bit frustrating, getting excited from the first blog-articles and tutorials about Nomad, which itself proclaimes to provide an easy developer experience, just to find out that this experience is "just" for Linux-based and MacOS(X) developer systems. Myself I am mostly using Windows as my daily driver, and having a mixed type of systems among our developers, I would like to have a shared configuration in our projects.

As "Docker-for-Windows" itself is provided in multiple ways (via HyperV or WSL) there probably is "just" a complexity-issue of the driver. I am confused why this is still an active issue at least for HyperV based Linux-containers.

Is there any plan to provide a workaround to this? Like "just" connecting to a docker-socket or remote-port?

I will look forward to use Nomad (mostly because I dislike the whole Kubernetes ecosystem due to its complexity and giant fragmenting YAML files), but will have to create separate VM for it. Or is there a "simple solution" via Vagrant? (I mean, its a HashiCorp-product, maybe the chances are good, hehe)

EDIT: ... RTFM ... there are instructions for a local Vagrantfile setup: https://www.nomadproject.io/docs/install/quickstart#run-nomad-in-vagrant ... sorry for the noise, but it would be cool if it could work without Vagrant (I think this is nice, but only a workaround).

[agajdosi]

This is very disappointing for such a beautiful product.

[Seikilos]

I also was happy to test drive it, only to find out, that docker on windows with linux containers does not work.
Since I am building linux docker images and use them, the vagrant option is not really a good one because I have to switch hyper visor modes and reboot the machine every time to either use vagrant with virtual box or be able to use docker.

[Seikilos]

A little update after toying around with nomad and only be using raw_exec. I would prefer nomad over k8s but regarding linux containers on a windows machine I will probably stick to Rancher Desktop, which brings a working k3s out of the box and Rancher Dashboard comes along to provide a nice web ui.

I thought about trying nomad on a vps but since I can only can connect via public internet web interface to it (without building a vpn), I would be getting fully hit by "running nomad in production" with setting up mTLS and ACL, probably spending days figuring everything out. Then having issues like single node nomad server running into recovery mode, wiping all data and start again.

If I recall correctly, Armon Dadgar CTO of Hashicorp said, he regards nomad as a competitor to k8s (and I agree that k8s is complex and fragmented by various providers).
I do not know, how big the "windows developer for linux containers" segment is (I would think it is largely growing since windows streamlines developing C#, etc. for linux containers), but as of now, I am one and nomad is unfortunately of no use for me. 😥

[gthieleb]

@Seikilos did you tested the solution of @das974 with using a seperate docker installation in WSL2 for the Linux Workload? . There is also the option to use podman [1] (or podman desktop) within WSL2 which also offers you running container workload with a (similar command line like docker) in Linux. For nomad a podman plugin exists [2].

[1] https://podman.io/docs/installation
[2] https://github.com/hashicorp/nomad-driver-podman

[Seikilos]

@Seikilos did you tested the solution of @das974 with using a seperate docker installation in WSL2 for the Linux Workload? . There is also the option to use podman [1] (or podman desktop) within WSL2 which also offers you running container workload with a (similar command line like docker) in Linux. For nomad a podman plugin exists [2].

[1] https://podman.io/docs/installation [2] https://github.com/hashicorp/nomad-driver-podman

Hi, @gthieleb
thanks for replying. I did and it did not work, but it was user error. Now the workaround of nomad being an agent in wsl seems to work. I explain what I did wrong to help others:

When I tried this, I had no docker driver in my WSL2 linux. But the issue was: The docker enabled WSL Linux is an ubuntu 20.04. I later installed a WSL2 Ubuntu 22.04 via app store for other things.
The default setup was

• Ubuntu 20.04 with docker support
• Ubuntu 22.04 vanilla, no docker

So actually starting nomad client on 20.04 seems to work for now.

Another issue, we windows users are very likely to hit is: After rebooting our machine, the nomad server will go into recovery and lose all the data from the previous run. AFAIK this has something to do with its identity. The IP is part of it and since vEthernet WSL keeps assigning a new IP each time, I have to add a static IP to my windows host every time by calling

netsh interface ip add address "vEthernet (WSL)" 172.19.200.1 255.255.255.0

My client.hcl file then basically looks like

client {
  enabled = true
  servers = ["172.19.200.1:4647"]
}

I have yet to test, whether this works.

[tgross]

Hi folks! The Docker ecosystem has moved a little bit since this issue was originally opened, so I wanted to summarize the current state of the world and recommendations for running containers on Windows.

• As of Docker 23.0, Docker has removed support for the original implementation for Linux containers on Windows (called "LCOW"). Nomad never really supported this well.
• Running Linux containers on Docker Engine on Windows Server via WSL2 should work, assuming Nomad is running under WSL2 as well.
• Running Linux containers on Docker Desktop on Windows should also work, with the caveats described in our FAQ for Docker Desktop
• Running Windows containers on Windows Server with either Hyper-V or Process Isolation mode should work. We consider this to be the expected production environment for Nomad clients on Windows.

We're going to close this issue out as resolved. If you run into issues with current versions of Nomad and the specific scenarios described above, please feel free to open a new issue describing the symptoms you're seeing (and which scenario you're deploying in) and we'll try to get that sorted. Thanks everyone!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.