Should we continue supporting building Linux container images under Windows?

[adamrehn]

I've never attempted to build Linux container images under Windows with ue4-docker myself (I always use native Linux for this), but I've been seeing a number of questions lately from users who are doing so and it's become evident that the WSL2 backend for Docker Desktop under Windows suffers from a couple of limitations that make it poorly suited to this use case:

• As encountered in ue4-docker build 4.27.0 --cuda=10.2 --no-engine FAILED！NEED HELP！ #203 and discussed upstream in Very slow network speed on WSL2 microsoft/WSL#4901, network speeds inside WSL2 can be extremely slow depending on the configuration of the host Windows system. This is far from ideal when building container images that download many gigabytes of source code and binary dependencies.

• As encountered by users who have been attempting to build the generated Dockerfiles for ue4-minimal that ship with Unreal Engine 4.27, the default disk size for the WSL2 backend is too small to build images with ue4-docker. Unlike the old Hyper-V backend, the WSL2 backend does not provide an easy way to increase the disk space available to the Docker daemon, which means users need to follow the fairly involved set of steps documented in How to change 250G disk size limit. Where is it configured/defined ? docker/for-win#9672 before they can build images.

Users can of course configure Docker Desktop to use the legacy Hyper-V backend rather than the WSL2 backend, but the newer backend provides better UX (for non ue4-docker use cases) and receives active feature updates and maintenance, so it's probably safe to assume that users will prefer to use WSL2 going forward.

Given these limitations and the poor user experience that they cause, I can see two potential options to address the problem:

• Drop support: adopt the position that this is an unsupported configuration and advise users against it.

• Improve support: thoroughly document the problems and their resolutions, and potentially investigate ways to automate the expansion of the WSL2 disk size so we can perform this step in the ue4-docker setup command.

@slonopotamus @TBBle what do you think?

[TBBle]

My leaning is 'don't support', but my own use-cases are generally towards CI, so that may be experiential bias.

Windows Server doesn't support WSL2 nor Docker Desktop for Windows, so if we do want to support this, it's strictly a Windows 10 (and Windows 11...) issue, where we already have several caveats.

Linking to the MS docs on expanding your virtual disk with specific pointer to the Docker Desktop for Windows partition image that needs to be expanded is reasonable.

The networking issue appears to be related to "Large Send Offload", but MS's registry documentation link is dead, and the various UI options for this setting apparently revert on reboot. It might be possible through netsh though.

I wouldn't bother trying to automate support for these issues though. Diagnostics might be possible, but at least the network performance change would require host system administrator permissions to make changes in the network configuration. I suspect the resize operation would too, since the disk image lives in %PROGRAMDATA%\Docker which is limited to Administrators.

[slonopotamus]

I personally always treated WSL as a "hackish VM that cuts corners here and there to speedup things but loses compatibility due to that". If one wants to build Linux builds on Windows, they can just spin up a proper VM. So I vote on "don't support but accept patches that extend support for this configuration in case someone submits them".

Windows Server doesn't support WSL2

Are you sure about this? (Note that I haven't tried that personally)

[TBBle]

Windows Server doesn't support WSL2

Are you sure about this? (Note that I haven't tried that personally)

Yup, totally sure: microsoft/WSL#6301 (comment)

It was disabled in Windows Server when it went from vNext (when that article was written) to Windows Server 2022 LTSC.

[slonopotamus]

Okay then, I suggest we just claim somewhere in docs: if you want to build Linux ue4-docker images on a Windows box, just use a VM (Hyper-V, VMWare, VirtualBox or whatever) and install Linux-distro-of-your-choice there.

[TBBle]

That would also keep us out of the firing-line for "Docker Desktop for Windows doesn't release disk space back to Windows when you remove Linux container images", i.e. docker/for-win#244 for Hyper-V backend and microsoft/WSL#4699 for WSL2 backend, both of which have accumulated years of "Here's some random magic that worked for me" comments, that are sometimes for the other backend, and occasionally are non-obvious versions of "delete your entire container image collection".

[DVLP]

Windows Server does support Docker Desktop for Windows but not in WSL2 mode, Hyper-V version works fine.

[TBBle]

It may work on Windows Server, but is not supported:

Can I run Docker Desktop on Windows Server?
No, running Docker Desktop on Windows Server is not supported.

[DVLP]

It may work on Windows Server, but is not supported:

Can I run Docker Desktop on Windows Server?
No, running Docker Desktop on Windows Server is not supported.

So it's Docker that doesn't support Windows Server not the other way around.

[TBBle]

Microsoft doesn't support Docker Desktop at all; Docker, Inc. supports Docker Desktop, it's their product.

As it happens, Microsoft do support Docker Engine on Windows Server, by licensing Mirantis Container Runtime, (formerly Docker Enterprise) from Mirantis for Windows Server licensees. However, this does not support Linux containers, as the Docker Desktop WSL2 or Hyper-V backends are not present.

Microsoft support LCOW (at least, they're still fixing bugs and adding features) but Docker have dropped or deprecated their LCOW experimental integration (HCS v1), and so LCOW is basically moribund until at least Docker moves to containerd on Windows (integrating HCS v2), and the LCOW integration in Docker is redone against containerd, assuming someone is sufficiently motivated to own that bridging work. My guess is that MS themselves are using LCOW and containerd under containerd-cri for Kubernetes, so Docker integration may not be a high priority for them.

[adamrehn]

Docs changes are up: https://gitlab.com/adamrehn/docs.adamrehn.com/-/commit/374d5035365a9c10429294dd3bda35aa9bbff289

You can see the affected pages here:

• https://docs.adamrehn.com/ue4-docker/configuration/supported-host-configurations
• https://docs.adamrehn.com/ue4-docker/configuration/configuring-windows-10

[adamrehn]

@slonopotamus @TBBle in addition to the documentation explicitly stating that this scenario is unsupported, did we want to have ue4-docker print a warning message to that effect when it detects that users are building Linux containers under Windows?

[TBBle]

I think so, yeah. That way there's an even better chance of that information being in people's issue reports when they copy and paste error logs into a GitHub issue. Assuming we can detect it reliably, probably under the [ue4-docker build] LINUX CONTAINER SETTINGS section of the output.

I guess detection would work on the Operating System value and/or Kernel Version in the server status of docker info, assuming that's exposed usefully in the API. That wouldn't be 100% percent reliable though, as a WSL2 user could be bypassing Docker Desktop to directly talk to the Docker daemon, and running a custom (non-microsoft-standard-WSL2) kernel. That case would be impossible to distinguish from building against a real but remote Linux host's dockerd.

So any such output would be best-effort, but that should be good enough to catch low-hanging fruit, i.e. "Ooops, I left Docker Desktop in Linux Containers mode".

[slonopotamus]

microsoft/WSL#4555 has several ways of detecting we're inside WSL. The simplest one seems to be $WSL_DISTRO_NAME env variable.

[slonopotamus]

So any such output would be best-effort, but that should be good enough to catch low-hanging fruit, i.e. "Ooops, I left Docker Desktop in Linux Containers mode".

That cannot happen. You have to explicitly pass --linux to ue4-docker build to tell that you want to build Linux engine on a Windows host.

[adamrehn]

Yeah, we already detect when ue4-docker itself is running on a Windows host and communicating with a Linux daemon (which triggers the check for the --linux flag), so the only detection logic that needs to be added is for the case that ue4-docker itself is running inside WSL2. The environment variable approach sounds good.

[adamrehn]

Circling back to this a few years later, the check and associated warning message have now been added in commit 4d1cec3.