WSL2 and Hybrid Support

[prom3theu5]

Hi. Currently you can switch to Windows containers, and enable experimental features, which gives you the ability to run a Hybrid setup on the same machine, i.e. You gain access to the Platform option to be able to run windows and Linux containers at the same time

If you switch to Linux containers, you lose the ability to start Windows containers.

I'm on insiders 2004, and have WSL2 working with docker set to linux, but ultimately lose the windows only containers I need to run when I do this.

Is there a way to Specify Windows containers, but also gain the performance benefits of WSL2 for my linux containers?
Don't want to have to fire up a Hyper-V VM and create a swarm between host and this guest to achieve it, and the windows components I need to run will not run under Mono

[simonferquel]

Hi, currently we are facing 2 issues with that:

• There is a remaining bug in 2004 (fixed in insider fast branch) which prevent us to run both windows containers and wsl2 at the same time reliably. (see [wsl2] Import failed with: The operation timed out because a response was not received from the virtual machine or container. microsoft/WSL#4726).
• once the bug is fixed, we plan to leverage docker CLI contexts to run and expose both daemons at the same time and allow you to switch from one to the other using those contexts:

# assuming default context is Linux
$> docker run -d -p 127.0.0.2:80:80 nginx
$> docker --context=dd-win run -d -p 127.0.0.3:80:80 aspnet-iis

[simonferquel]

Is this kind of UX something that would fit your use case ?

[prom3theu5]

Hi Simon
That would be perfectly acceptable for our use case :)

Do you think there will be eventual compose or swarm support?
Thanks

[prochnowc]

@simonferquel Is there a way to try this out with current Docker Desktop Edge release ?

[bplasmeijer]

Hi @simonferquel ; we did spoke before about the docker context.
The new compose support 1.126-rc4 supports the docker context now.

There is a high demand for Hybrid, simultaneously build and run, docker/roadmap#79

@craigloewen-msft can you help on microsoft/WSL#4726 ?

[simonferquel]

We don't have started working on this. We still need to figure how it will work in term of UI etc (what about containers dashboard, settings UI etc.), and in term of lifecycle (when should a particular backend be started ? on docker context use ? At startup ?)

[bplasmeijer]

We don't have started working on this. We still need to figure how it will work in term of UI etc (what about containers dashboard, settings UI etc.), and in term of lifecycle (when should a particular backend be started ? on docker context use ? At startup ?)

We should use the context argument where needed and have a default context set.
If the requested context is sleeping, spin-up the context.
The GUI will combine both context in one single view, and filter on the context.

One happy world, Windows, and Linux containers don't separate.

[nunix]

not supported by docker, but here is a workaround for using "both" daemons at the same time:
https://wsl.dev/hybridswarm/

in the long run, if LCOW evolves into LCOW2, then it would be more streamlined for the users and also in terms of UI.

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[georgechang]

/remove-lifecycle stale

[bplasmeijer]

any update @simonferquel ?

[simonferquel]

Still in backlog, not yet prioritized. (But we really want to do it!)
Cc @nebuk89

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[cerebrate]

/remove-lifecycle stale

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[cerebrate]

/remove-lifecycle stale

[SetTrend]

Perhaps my imagination is wrong, but shouldn't this be a quick one?

I would expect you to launch some flimsy standard Linux distribution in a WSL 2 Hyper-V machine with containers enabled and address that from the local Docker client.

[TBBle]

@SetTrend That part already works in Docker Desktop's WSL2 support, although they use a trimmed-down custom Linux distribution instead of a "flimsy standard Linux distribution" to host the Linux dockerd inside WSL2.

What this issue wants is to support addressing both the dockerd in WSL2 and the dockerd in Windows at the same time, possibly using the --context command-line flag, and also all the work needed to make the UI play nicely with this setup, rather than completely switching modes as it does now.

[SetTrend]

Yes, I apprehend.

Maybe I'm blind here, but frankly, what's the extraordinary challenge in particular? There are just two daemons to address, one local and one remote (the WSL 2 one), and merge them in the GUI to appear like one (or let them appear as two - no one cares). The only challenge I see is to be able to do correct port forwarding and get the docker network commands work properly.

If the docker command line was based on a provider design pattern, multiple, arbitrary daemons could be addressed and appear as one.

One could think of introducing namespaces in order to address different providers' images and containers (highlighted in red below):

NB:

• New alias property suggested to add to docker login for introducing a namespace.
• Commands without namespace reach out to all signed in providers (e.g. docker container ls)
• Commands targeting particular images or containers would be enhanced by providing a namespace if more than one provider is signed in to.

[TBBle]

That structure roughly already exists (where you have 'namespaces', in the open-source part of Docker CLI, called 'contexts', except the Docker CLI commands only operate on one context at a time, rather than the docker container ls example you gave above which is operating on all of them.

And people have demonstrated functionality rather like the flow described in #6311 (comment), but it takes a bit of manual setup of the contexts, and the UI only knows about one of the contexts at a time, and the API Proxy is only functional for one at a time. And on startup, Docker Desktop normally only starts the daemon for the mode you're in, so you need to switch modes so both daemons are running.

If you want to experiment for yourself, the two named pipes are npipe:////./pipe/docker_engine_windows and npipe:////./pipe/docker_engine_linux (not sure how stable these are, I've seen npipe:////./pipe/docker_engine_wsl2 from otherr people). The API proxy is on npipe:////./pipe/docker_engine, and I'd recommend keeping Docker Desktop in Linux Containers mode and using the API Proxy's pipe for Linux, as the API Proxy features like path-translation are probably more important for Linux containers inside WSL2 than Windows containers running natively.

There's a version of the setup at docker/roadmap#78 (comment), but the original source has disappeared, and it actually sets up Docker from scratch both in Windows and in an Ubuntu distro in WSL2, without Docker Desktop, so doesn't really solve this issue at all (and will be an absolute pain to get right for volume mounts from Windows, but that was the "people who enjoy that kind of pain" approach anyway.)

The actual work to be done to deliver this feature in Docker Desktop, AFAIK, is teaching the UI and the Docker Desktop API proxy (and any other internal systems) in the non-open-source part of Docker Desktop for Windows to setup and support this mode of functionality, smoothing off the rough edges as seen in the complexity of setting up a similar workflow in Docker Swarm (See #6311 (comment)).

All that said, just because something is not particularly complex, doesn't mean it gets done next. I don't know how the Docker developers prioritise their time, but I am pretty confident that they don't have nearly as much as they'd like to have, so relatively-low-value (small target market) new features like this are probably low on their list.

[SetTrend]

@TBBle: 👍 Sure, I apprehend.

Nonetheless, I seriously would expect Microsoft (who I suppose is the owner of this repository) to sponsor that effort. They are so keen in advertising Docker in each of their cloud and development products, so I suggest they should strive to do their best to make it work flawlessly on Windows to not make it look like a bait-and-switch offer.

[TBBle]

This repository (and Docker Desktop for Windows) belongs to Docker Inc. I'm not sure why you would suppose Microsoft owns it?

[SetTrend]

I see. Please excuse the imputation. However, I still suggest MS should sponsor it, so we get a satisfying product fast. It's their intrinsic business interest to have the Windows platform fully supported in this area.

From my proposal above I believe it didn't get clear that there'd also be a volume/port sharing shim included in the suggested container provider, so remote ports and volumes supposedly could be mapped to local ports and drives. That way you could host a website on Azure in a container and reach it by, e.g. localhost:5050. Same would be true, then, for a Linux VM, joined to the local Windows Docker client by the suggested provider. I'm rather suggesting an orchestration feature, that would yield (a) more transparency, and (b) more versatility.

[conioh]

Contrary to some bizarre comments, installing Docker "from scratch" is really easy, something like 5 PowerShell lines, or one script invocation. You can also use the install script by Mirantis but it has some issues (doesn't work in PowerShell 7, sets global flags that break your PS session after the script ends, incredibly long and convoluted) and I prefer my own.

Setting up Docker inside a WSL distribution is another command or two and then you get the behavior Simon showed above, without waiting for an uncertain future. (And it's better for a host of other reason too.)

[SetTrend]

For me, the most important part is that me and my team, we're Windows engineers, not Linux engineers. For getting our work done, (the one we're getting paid for and the one we're paying MS subscriptions for), we solely need WSL 2 for prefactured containers that aren't available for Windows.

I don't want to dig into the world of Linux programming, and, with the invention of WSL 2, I expect both types of containers to run smoothly along in Windows and for Windows to fully support Docker features. That was the advertised promise. That's what we've planned our projects for, that's what we're paying our subscriptions for, and that's what we expect.

[conioh]

I suggest you adjust your expectations from Microsoft to avoid being constantly disappointed. This applies to them in general and not necessarily to this issue.

Though in this case you should remember that Docker is a third party offering and your payments to Microsoft, one-time or through a subscription, can't reasonably provide a basis for such an expectation.

In my opinion running apk add docker can't be considered "dig[ging] into the world of Linux programming" but I understand some people may have different opinions about that.

I suppose some people might also expect their IT departments to install for them Visual Studio with the correct components, and if they fail to build a project because an optional component is missing (for example, Spectre-mitigated libraries or a newer version of the Windows SDK) they expect their IT department to fix that for them rather than just add the missing component.

I'm aware of this approach but I don't subscribe to it and my proposal isn't intended for people who follow it.

Notice however that the way provided by Microsoft, the ones you are paying, to install Docker on Windows Server is using a PowerShell script/command and Mirantis who probably also has paying customers offers installation on Windows 10 via the script I linked above.

If you require a graphical, windowed installation wizard I guess you'll have to keep waiting.

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[cerebrate]

/remove-lifecycle stale

[cerebrate]

/lifecycle frozen