Windows support

[awakecoding]

I am looking for a way to use containerd on Windows Server 2019 with a Docker CLI replacement like this one. I see there are Linux builds, but has a Windows port been attempted?

Long story short, containerd for Windows should handle graceful shutdowns for containers, but the current version of Docker is still using HCS v1 + some really broken stuff under the hood, meaning Windows containers have never supported graceful shutdowns (microsoft/Windows-Containers#16).

There is no clear migration path for Docker to adopt containerd on Windows in the short term, so I would be fine with installing containerd on Windows (https://kubernetes.io/docs/setup/production-environment/container-runtimes/#install-containerd) but then I just need a proper CLI on top of it, other than 'ctr' which is very primitive and marked as unsupported/unstable.

I am not familiar with Go, but how difficult would it be to attempt a Windows port of this tool?

[AkihiroSuda]

Contribution is wanted 😃

[jsturtevant]

I took a quick look and it would be doable but would require some work, there are assumption that this would run on linux like (there are probably more):

nerdctl/exec.go

Line 229 in 06f3a7c

allCaps, err := cap.Current()

nerdctl/run.go

Lines 762 to 766 in 435f533

	for _, c := range cap.Known() {
	// "CAP_SYS_ADMIN" -> "sys_admin"
	s := strings.ToLower(strings.TrimPrefix(c, "CAP_"))
	fmt.Fprintln(w, s)
	}

nerdctl/pkg/lockutil/lockutil.go

Line 33 in 435f533

if err := Flock(dirFile, unix.LOCK_EX); err != nil {

nerdctl/pkg/containerinspector/containerinspector.go

Line 73 in 435f533

fn := func(_ ns.NetNS) error {

[jsturtevant]

I got started on this. Will likely want to break this down into many different PRs

[AkihiroSuda]

Thanks!

[jsturtevant]

Got it to build and connect 🚀

PS C:\projects\nerdctl\_output> .\nerdctl.exe ps -a
CONTAINER ID    IMAGE                                        COMMAND                   CREATED          STATUS                   PORTS    NAMES
multi-arch      k8s.gcr.io/pause:3.4.1                       "/pause.exe"              3 days ago       Created
multi-arch2     k8s.gcr.io/pause:3.4.1                       "/pause.exe"              8 minutes ago    Created
multi-arch3     k8s.gcr.io/pause:3.4.1                       "/pause.exe"              8 minutes ago    Up
test            mcr.microsoft.com/windows/nanoserver:20H2    "echo hello"              9 days ago       Exited (1) 9 days ago
test-1          mcr.microsoft.com/windows/nanoserver:20H2    "echo c:\\License.txt"    9 days ago       Exited (1) 9 days ago
test2           k8s.gcr.io/pause:3.4.1                       "/pause.exe"              7 days ago       Created

Needs some more work to be able to run containers

[jsturtevant]

I was able to get nerdctl run to work! But I did run into issues running multi-arch docker manifests (containerd/containerd#5297) which should be resolved by containerd/containerd#5298.

.\nerdctl.exe run --name nerdclttest1 mcr.microsoft.com/windows/nanoserver:20H2
Microsoft Windows [Version 10.0.19042.867] 
(c) 2019 Microsoft Corporation. All rights reserved.                                                                                                                                                                                            C:\Windows\system32>                     

Although it is rough, I opened #164 just to show the work involved. I would propose breaking this down into at least two PRs for easier reviewing:

1. split out the Logic that is tied to Linux to their separate files so that we can build the windows binary. This would allow producing and .exe but functionality would be limited. From my testing nerdctl ps would work and maybe some other basic commands.
2. Add in run support which requires a bit more refactoring.
3. After that we can approach each other functionality iteratively from there.

What do you think @AkihiroSuda?

[AkihiroSuda]

SGTM, thanks

[awakecoding]

I am revisiting this ticket since Rancher Desktop 0.5 just added initial nerdctl support. My understanding is that the latest version of nerdctl doesn't support Windows containers, unless one builds a branch from @jsturtevant from source + installs containerd.exe manually?

[AkihiroSuda]

I am revisiting this ticket since Rancher Desktop 0.5 just added initial nerdctl support. My understanding is that the latest version of nerdctl doesn't support Windows containers, unless one builds a branch from @jsturtevant from source + installs containerd.exe manually?

Yes. Help wanted to move #197 forward

[jsturtevant]

should we close this? I've opened a few other issues to track the next steps in windows support, otherwise this could be used as an epic?

[jsturtevant]

https://github.com/containerd/nerdctl/issues?q=is%3Aopen+is%3Aissue+label%3Aplatform%2FWindows

[AkihiroSuda]

I think we can keep opening this

[brwilkinson]

I was wondering what the current status is for Windows support ?

Based on below, can I assume this issue can be closed now?

• https://learn.microsoft.com/en-us/virtualization/windowscontainers/quick-start/set-up-environment?tabs=containerd

Was there anything in particular outstanding?

[TBBle]

https://github.com/containerd/nerdctl/issues?q=is%3Aissue+label%3Aplatform%2FWindows%2FNon-WSL2+is%3Aopen are the outstanding issues. (Same list as #28 (comment), but the label has been renamed since that comment was made.)

I have no strong opinion on whether we should keep this issue open or not. This issue does appear to lack clear (unmet) success criteria though.

[AkihiroSuda]

This issue does appear to lack clear (unmet) success criteria though.

We still need to support builds and compose AFAICS.
It would be also nice to have a double clickable “SETUP.EXE” too.

[TBBle]

#627 exists for building, it is currently blocked on microsoft/Windows-Containers#34 and through to buildkit and related. Once we have buildkitd for WCOW stable, hopefully the integration won't be too far from how it's used on non-Windows.

---

I know approximately nothing about compose, nor the compose support in nerdctl. I don't yet see a ticket for it to work on Windows, so I'm guessing not much time has been spent on that yet, given we're still lacking build support.

---

Are you thinking of a simple installer for nerdctl.exe, or more like the "Full" binary release, bundling the whole (Windows-supported) stack of dependencies with service setup etc?

Or perhaps something different? We could recreate something like MS's existing containerd/nerdctl script, perhaps.

[AkihiroSuda]

Are you thinking of a simple installer for nerdctl.exe, or more like the "Full" binary release, bundling the whole (Windows-supported) stack of dependencies with service setup etc?

I’m expecting the latter one

[SamuelMarks]

I can help with the installer and porting Go code cross-platforms.

FYI: That linked issue just got an update 9 hours ago:

// Update: Buildkit 0.13.0 is now released with windows binaries - Release v0.13.0 · moby/buildkit (github.com) 🎉 Thank you all who have continued to test the releases, WCOW support is still experimental but we are on the way to stabilization. We will appreciate any feedback; this will aid in our prioritization especially on closing the Linux/Windows feature-parity gap. Feel free to open the issue at moby/buildkit.

// UPDATE: Stabilization efforts still going on. We have had several releases since v0.13.0, now at v0.14.1. Aiming for GA by September 2024. Feel free to give it a try and open any issues on moby/buildkit.

microsoft/Windows-Containers#34 (comment)

[TBBle]

At this point, the hardest remaining thing for "Windows support" is getting compose working, and I say that simply because I have no context on compose, so don't know how far the current code is from "working" if we simply remove any checks for Windows.

That said, building a full-stack installer (containerd, nerdctl, buildkitd, CNI plugins, is there a composed?) is a fiddly task, so having that done even before we have compose support ready would definitely make it easier for people to test and start working with the nerdctl stack. And really, that's more important than closing any specific ticket, I think.

Oh, we need to look closely at the CNI setup, nerdctl generates CNI 1.0.0 configs, but the last-released WinCNI nat plugin can't read those. The WinCNI source has been updated, but no new releases shipped. So we might need to bundle a local build of the WinCNI plugin in such a package. (Also, the generated config might be slightly wrong. I worked through this while testing nerdctl/buildkitd support on Windows, see microsoft/Windows-Containers#34 (comment) for example. However, I was working from an existing Docker-created NAT network, so that's not necessarily the complete target experience.)

[TBBle]

On the topic of installers, I just came across https://github.com/microsoft/containers-toolkit which appears to be the start of centralising various containerd-nerdctl-buildkitd stack install scripts in a PowerShell module. I haven't tried it, but it might be a reasonable alternative or placeholder for the full setup.exe install stack we've talked about here.

[slonopotamus]

Hmmm, maybe it's time to extend Stevedore a bit 😈

[slonopotamus]

How do you handle the situation with nerdctl requesting CNIVersion=1.0.0 in C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist while CNI plugins only support 0.3.0? There is microsoft/windows-container-networking#101, but it was never released.

UPD: I missed that @TBBle talks about it in #28 (comment). Maybe we actually should just fork and release the thing while waiting for upstream.

[slonopotamus]

That said, building a full-stack installer (containerd, nerdctl, buildkitd, CNI plugins, is there a composed?) is a fiddly task, so having that done even before we have compose support ready would definitely make it easier for people to test and start working with the nerdctl stack. And really, that's more important than closing any specific ticket, I think.

Look what I've got for you!

Basic usage instructions

[slonopotamus]

Maybe we actually should just fork and release WinCNI while waiting for upstream.

And... done!

[SamuelMarks]

@slonopotamus Great to see!

PS: Ping me if you need/want a hand

[TBBle]

A quick note while I think of it: If I remember correctly, Windows only allows one NAT network to be created. So we might want to look at being able to read an existing NAT network config back from HCN/HNS when generating the CNI config, rather than just stamping out a pre-determined config as I recall we do now.

That will make life easier for systems where Docker is also installed, particularly if the user wants Docker to continue to work. Alternatively, matching whatever Docker's defaults are for the NAT network might bridge that gap for the majority of cases; assuming they're not dynamic... and assuming we're not already doing that and my experience wasn't an outlier.

[slonopotamus]

Windows only allows one NAT network to be created

I'm not sure about that:

[slonopotamus]

Well... Things are not fully functional. nerdctl container gets default gateway from C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist but all the rest network settings come from pre-existing nat network.

>nerdctl run --rm -it mcr.microsoft.com/windows/nanoserver:ltsc2022 ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : minwinpc
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No

Ethernet adapter vEthernet (27157d866c9ea519d931a2cf112db5037656d84b7f81ed7d160a7dd00b084a89_nat):


   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Container Adapter
   Physical Address. . . . . . . . . : 00-15-5D-16-73-AA
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::fa0b:424f:9b9:a66b%60(Preferred)
   IPv4 Address. . . . . . . . . . . : 172.20.60.34(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 10.4.0.1
   DNS Servers . . . . . . . . . . . : 192.168.0.1
   NetBIOS over Tcpip. . . . . . . . : Disabled

Interestingly, neither nerdctl- nor docker-based container can access IPv6 address.

UPD: Okay, I can force nerdctl to create a separate NAT network by adjusting name in C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist. And IPv4 + DNS starts to work in it (IPv6 still doesn't, but that's a separate story). The caveat is that I now need to pass --network <network_name> to each nerdctl run.

Can anyone explain what's the purpose of nerdctl/default-network label in C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist? It seems to be ignored, nerdctl run just defaults to nat: https://github.com/containerd/nerdctl/blob/v2.0.0-rc.0/cmd/nerdctl/container_run.go#L109-L116

[TBBle]

I must have misremembered about the NAT network limitation then, sorry. I'm sure there was some limitation, but maybe it was in Docker or something? (It may have been in the older virtual network API, too.)

nerdctl container gets default gateway from C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist but all the rest network settings come from pre-existing nat network.

This is a known behaviour issue to be fixed in the WinCNI plugins: If you name an existing network, it uses it as-is, it does not update it to match the CNI configuration. (The default gateway lives on the net connection to the network, not the network itself).

See microsoft/windows-container-networking#98

There's also an issue around the WinCNI plugins faultily matching their network binary to their type rather than their name, when they are creating a network: microsoft/windows-container-networking#57 It's possible this is why I recalled multiple NAT networks being a problem...

[slonopotamus]

There's also an issue around the WinCNI plugins faultily matching their network binary to their type rather than their name, when they are creating a network: microsoft/windows-container-networking#57

I believe that issue is outdated. I was able to use a different network name in nerdctl-nat.conflist. It properly created a separate nat network and used it via nerdctl run --network my_awesome_nat. The nuisance is that I have to say --network my_awesome_nat because nerdctl run doesn't lookup network name with nerdctl/default-network label and blindly uses nat, as I said before.

microsoft/windows-container-networking#98

I'm not sure it is a good idea. It will break things. I'd solve the issue from a completely different angle. If a network is already created, why user needs to write config file at all? Just inspect network and use settings from it.

[slonopotamus]

BTW, there is a new release of win-cni: https://github.com/microsoft/windows-container-networking/releases/tag/v0.3.1

[apostasie]

Hey folks.

Is this issue still serving a purpose?
Should we open follow-ups and/or close this?

[TBBle]

Last time we talked about this:

This issue does appear to lack clear (unmet) success criteria though.

We still need to support builds and compose AFAICS. It would be also nice to have a double clickable “SETUP.EXE” too.

Since then, we have build support (i.e. BuildKit), but not yet Compose (in fact, I don't even see a ticket for it) and I'm not a Compose user, so I haven't looked into it to see what exactly that entails. The Compose implementation is already included in the Windows binary, so maybe it "just works"? I'm really hoping someone who uses Compose on Windows Containers will magically appear and report bugs in what we have now, and or tell us it works fine...

We don't have a full-stack installer in the containerd org yet, but Stevedore can cover that at the moment; it isn't mentioned in the README's component list, but it does include BuildKit, so it should cover all the use-cases we have for a full-stack installer AFAIK.