Downlevel support for Windows 11 doesn't seem to work for some images

[jsturtevant]

On windows 11:

cmd /c ver
Microsoft Windows [Version 10.0.22000.258]      

I am able to run some WS 2022 images even thought they have the correct build numbers:

docker run --isolation=process --rm mcr.microsoft.com/windows/nanoserver:ltsc2022 cmd /c echo hello                    
 hello                                    

 ./manifest-tool-linux-amd64 inspect --raw mcr.microsoft.com/windows/nanoserver:ltsc2022 | jq ‘.[] .Platform’
{
“architecture”: “”,
“os”: “”
}
{
“architecture”: “amd64”,
“os”: “windows”,
“os.version”: “10.0.20348.288”
}      

But not others:

docker run --isolation=process --rm k8s.gcr.io/e2e-test-images/busybox:1.29-2 cmd /c echo hello
docker: Error response from daemon: container 5ff61c6cb106ce89cb0117f810cab29e02cd19c750d9c66a1918aeca5d3176be encountered an error during hcsshim::System::Start: failure in a Windows system call: The container operating system does not match the host operating system. (0xc0370101).   

/manifest-tool-linux-amd64  inspect --raw k8s.gcr.io/e2e-test-images/busybox:1.29-2 | jq '.[] .Platform'
{
  "architecture": "",
  "os": ""
}
{
  "architecture": "amd64",
  "os": "linux"
}
{
  "architecture": "arm",
  "os": "linux",
  "variant": "v7"
}
{
  "architecture": "arm64",
  "os": "linux"
}
{
  "architecture": "ppc64le",
  "os": "linux"
}
{
  "architecture": "s390x",
  "os": "linux"
}
{
  "architecture": "amd64",
  "os": "windows",
  "os.version": "10.0.17763.2114"
}
{
  "architecture": "amd64",
  "os": "windows",
  "os.version": "10.0.19041.1165"
}
{
  "architecture": "amd64",
  "os": "windows",
  "os.version": "10.0.19042.1165"
}
{
  "architecture": "amd64",
  "os": "windows",
  "os.version": "10.0.20348.169"
}

[jsturtevant]

found something suspicious:

PS C:\Program Files\containerd> docker history k8s.gcr.io/e2e-test-images/busybox:1.29-2
IMAGE CREATED CREATED BY SIZE COMMENT
32f74229bd0d 2 months ago ENTRYPOINT ["cmd.exe" "/s" "/c"] 0B buildkit.dockerfile.v0
<missing> 2 months ago ENV PATH=C:\dig\;C:\bin;C:\curl;C:\Windows\S… 0B buildkit.dockerfile.v0
<missing> 2 months ago USER ContainerAdministrator 0B buildkit.dockerfile.v0
<missing> 2 months ago ADD hostname /bin/hostname.exe # buildkit 2.13MB buildkit.dockerfile.v0
<missing> 2 months ago COPY /Users/ContainerAdministrator/AppData/L… 3.82kB buildkit.dockerfile.v0
<missing> 2 months ago COPY /PowerShell/pwsh.exe /Program Files/Pow… 276kB buildkit.dockerfile.v0
<missing> 2 months ago COPY /PowerShell /Program Files/PowerShell #… 148MB buildkit.dockerfile.v0
<missing> 2 months ago COPY /Windows/System32/ntdsapi.dll /Windows/… 149kB buildkit.dockerfile.v0
<missing> 2 months ago COPY /Windows/System32/en-US/nltest.exe.mui … 22kB buildkit.dockerfile.v0
<missing> 2 months ago COPY /Windows/System32/nltest.exe /Windows/S… 527kB buildkit.dockerfile.v0
<missing> 2 months ago COPY /vcruntime140.dll /Windows/System32/ # … 95.7kB buildkit.dockerfile.v0
<missing> 2 months ago COPY /dig-dir /dig # buildkit 7.51MB buildkit.dockerfile.v0
<missing> 2 months ago COPY /netcat-dir/nc64.exe /bin/nc.exe # buil… 45.3kB buildkit.dockerfile.v0
<missing> 2 months ago COPY /curl-dir /curl # buildkit 2MB buildkit.dockerfile.v0
<missing> 2 months ago COPY /busybox-dir /bin # buildkit 1.15MB buildkit.dockerfile.v0
<missing> 2 months ago COPY /tmp-dir /tmp # buildkit 0B buildkit.dockerfile.v0
**<missing> 3 months ago Apply image 1809-amd64 257MB**

[brasmith-ms]

After some investigation we found that this is likely due to an issue with the runtime automatically selecting an a version from multi-arch images when it sees that the OS build does not match. It's unclear as to where this bug is occurring but it seems clear that this relates to running multi-arch images on Windows 11.

[jsturtevant]

maybe @claudiubelu has some ideas? same happens with containerd

[lippertmarkus]

maybe related to #170 ?

[claudiubelu]

It does look suspicious, and at a first glance it does seem that the right image is not picked up from the manifest list.

Can you try running the ltsc2022 image directly? gcr.io/k8s-staging-e2e-test-images/busybox:1.29-2-windows-amd64-ltsc2022. If it works, it means that docker just picked the first image from the manifest list that matched windows/amd64, disregarding the OS Version. Could it be that docker is out of date? What version are you using?

[jsturtevant]

Looks like it is because of mutli arch manifests as in #170. pulling gcr.io/k8s-staging-e2e-test-images/busybox:1.29-2-windows-amd64-ltsc2022 worked.

I am using docker 20.10.8 and also had it happen with containerd 1.5.7. Using the direct image worked for both.

[ghost]

This issue has been open for 30 days with no updates.
@brasmith-ms, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates.
@brasmith-ms, please provide an update or close this issue.

[TBBle]

Yeah, this looks like a duplicate of #170. I'd suggest closing this in favour of that.

[cwilhit]

Closing in favor of #170