Windows Host Compute Service bug breaks docker (and other) sandboxes bigger than 20G on Windows 1903

[mitchcapper]

This is being filed here as it is one of two wrappers microsoft talks about for it at: https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/containerd

This one clearly is the primary one being used as well. If there is somewhere better for this bug then happy to file there.

I am not sure if fixing this bug will require a platform update (or if it is already fixed in an insider build). A great temporary work around would be if there was somewhere (registry or otherwise) where the default sandbox layer size could be changed from 20G to something else by the end user who encounters this issue.

This is not a docker problem (can be reproduced outside of docker), this is a Windows Host Compute Service bug starting in 1903. It is a pretty major blocker as it prevents(for example) all docker windows containers bigger than 20G (new or existing I believe, so users upgrading will then be broken).

It can be reproduced with docker as illustrated in bugs(will be referenced as 0x57 when its this):

• The hcsshim on Windows 10 1903 always fails to build Docker image #624 (mixed between this error and one other, fix for other 0x1 error does NOT fix this)

• hcsshim::PrepareLayer failed in Win32: The parameter is incorrect moby/moby#36831 (mostly this error a bit mixed with 0x1 error)

• An error occurred while attempting to build Docker image docker/for-win#3884 (mixed between this 0x57 error and the 0x1 error again)

• Windows 1903 fails when storage-opt used docker/for-win#4100 this error only

• Docker build windows 19.03 --storage-opt size>20G moby/moby#39524 this error only

by setting the storage option size=20481M or bigger either on docker daemon.json OR in the run command with docker run --storage-opt size=20481M

Docker will then fail with:

hcsshim::PrepareLayer - failed failed in Win32: The parameter is incorrect. (0x57)

Or it can be recreated using Microsoft.Windows.ComputeVirtualization library which essentially does what hcsshim does and sits on top of the vmcompute.dll dll.

While the call to ExpandSandboxSize with a size greater than 20480M succeeds (and the vhdx can be checked to reflect the new larger size) the further call to PrepareLayer will fail. The MountSandbox command from the ComputeVirtualization lib does that call after calling ActivateLayer.

To replicate in short hand:

ContainerStorage.CreateSandbox(child_path, layers);
ContainerStorage.ExpandSandboxSize(child_path, 20480 * 1024 * 1024L);
ContainerStorage.MountSandbox(child_path, layers);

This will succeed when set to 20480 and fail at 20481. I have attached a sample application sandbox.zip with a generic blank base layer that can be used to test this(use any base layer you want though). The app must be run as Admin as it is the only one with permissions to run some of the host compute commands. Extract sandbox.zip to c:\temp\sandbox (or update the base dir you extracted to in the MainWindow.xaml.cs). Change MainWindow.xaml.cs between 20480 and 20481 to see it work or fail.

There is very little (none?) official documentation around vmcompute.dll or even Microsoft.Windows.ComputeVirtualization but looking at the Microsoft.Windows.ComputeVirtualization prepareLayer call it seems very generic without any special or odd flags I can find.

[jterry75]

Closing out since #718 provide a workaround for now while we wait for the platform fix to be sent via WU.