[K3s][Windows Port] Build script, multi-call binary, and Flannel

[seanyen]

Proposed Changes

This is the initial set of K3s agent for Windows port. This pull request is focused on:

1. Modifying the build script for Windows build
2. Modifying the multi-call binary logic to accommodate Windows convention.
3. Modifying the Flannel and kube-proxy setup code for Windows node to run.

Types of Changes

New feature

Verification

Run GOOS=windows CXX=x86_64-w64-mingw32-g++ CC=x86_64-w64-mingw32-gcc SKIP_VALIDATE=true ./scripts/ci on Ubuntu machine to proof build Windows port.

NOTE: You may need to install the following packages to build:

sudo apt-get install mingw-w64
sudo apt-get install zstd

Testing

I am testing it on a Windows dev box with Hyper-V enabled, and I created two VMs:

1. Linux VM (in my case, I am using CBL-Mariner 2.0, but Ubuntu should work too).
2. Windows VM (Windows 11 Dev VM from the Hyper-V quickstart).

Two VMs are on the same subnet (in my case, I am using Default Switch).

On Linux VM, I deployed a k3s control plane node with all the default configuration and necessary firewall rules. Additionally, I pass --disable-network-policy to k3s server command, since Windows node doesn't support network policy yet.

On Windows VM, I prepared a k3s-config.yaml file and the content is as follows:

token: '<join token>'
server: 'https://<control plane VIP>:6443'
kubelet-arg=cgroups-per-qos: 'false'
kubelet-arg=enforce-node-allocatable: ''
kubelet-arg=resolv-conf: ''
pause-image: '<the windows pause image matching to your Windows VM>'

Example for pause-image: mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2022-amd64
List available pause images: curl -s -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" https://mcr.microsoft.com/v2/oss/kubernetes/pause/manifests/3.8

And additionally, I added the following firewall rules on Windows VM:

New-NetFirewallRule -Name k3s -DisplayName 'k3s' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 10250
New-NetFirewallRule -Name k3sOverlayTraffic4789UDP -DisplayName "K3s Overlay Traffic 4789 UDP" -Enabled True -Direction Inbound -Protocol UDP -Action Allow -LocalPort 4789

And finally, I ran K3s on Windows VM by the following command:

.\k3s.exe agent --config k3s-config.yaml

Linked Issues

#7258

User-Facing Change

None.

Further Comments

None.

[codecov-commenter]

Codecov Report

Attention: 23 lines in your changes are missing coverage. Please review.

Comparison is base (ba750e2) 47.27% compared to head (6e5f9ea) 51.08%.
Report is 3 commits behind head on master.

❗ Current head 6e5f9ea differs from pull request most recent head 2279e13. Consider uploading reports for the commit 2279e13 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7259      +/-   ##
==========================================
+ Coverage   47.27%   51.08%   +3.80%     
==========================================
  Files         143      145       +2     
  Lines       14806    14994     +188     
==========================================
+ Hits         6999     7659     +660     
+ Misses       6692     6118     -574     
- Partials     1115     1217     +102     

Flag	Coverage Δ
e2etests	48.34% <30.76%> (?)
inttests	44.28% <30.30%> (-0.33%)	⬇️
unittests	19.51% <0.00%> (-0.21%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
pkg/daemons/executor/embed.go	68.32% <100.00%> (-1.25%)	⬇️
pkg/daemons/executor/embed_linux.go	100.00% <100.00%> (ø)
pkg/agent/run.go	45.12% <25.00%> (+1.18%)	⬆️
pkg/cli/crictl/crictl.go	0.00% <0.00%> (ø)
pkg/kubectl/main.go	47.22% <0.00%> (-4.30%)	⬇️
pkg/agent/flannel/setup.go	36.36% <22.22%> (-2.15%)	⬇️

... and 44 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[brandond]

Thank you for the PR! I have a couple nits on some of the inline os-specific logic, I'm excited to get this merged once those are addressed!

[brandond]

@seanyen are you still working on this?

[seanyen]

@seanyen are you still working on this?

yes, sorry I was little bit slow on this. I am distracted by other works and will be back to this one in 2 week-ish.

[cwayne18]

@seanyen very much appreciate the work on this! Please note we can't merge unsigned commits due to our DCO check, can you please make sure to commit with -s

[seanyen]

@seanyen very much appreciate the work on this! Please note we can't merge unsigned commits due to our DCO check, can you please make sure to commit with -s

sure thing! I am still working on the feedback and definitely will make sure all checks are green before getting your attention again.

[cwayne18]

Thanks so much!

[brandond]

You can update your PR to use the new tag: https://github.com/rancher/plugins/releases/tag/v1.3.0-k3s1

k3s/scripts/version.sh

Line 56 in fe9604c

VERSION_CNIPLUGINS="v1.2.0-k3s1"

[brandond]

lgtm, but looks like it needs a rebase

[seanyen]

lgtm, but looks like it needs a rebase

Thanks. I did a rebase. Let me know anything else to take care.

[brandond]

Did you go mod tidy after rebasing? It looks like there are some missing imports.

[brandond]

Multiple builds are failing with:

+ go build -tags urfave_cli_no_docs -ldflags '
    -X github.com/k3s-io/k3s/pkg/version.Version=v1.27.3+k3s-bbee5d7d
    -X github.com/k3s-io/k3s/pkg/version.GitCommit=bbee5d7d
    -w -s
 -extldflags '\''-static'\''' -o dist/artifacts/k3s ./cmd/k3s
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.

@dereknola do you think this might be related to your change to exclude some of the git repo structure from being copied into the build image?

[seanyen]

Cool, now I saw all checks are all green now. Thank @dereknola.

[manuelbuil]

Looks good! I just have a couple of comments

[seanyen]

@seanyen A quick rebase will be necessary.

Thanks! The rebase was done.

[cwayne18]

YOLO 🎉🎉

[fcabrera23]

Congrats! Thank you all for the hard work

[cwayne18]

Thanks so much Francisco and Sean for all your work on this, we're very excited to continue working with y'all on this!

[amenome]

Silly question - ran through the build process (make download && make generate && make), but didn't get a k3s.exe out of it - what's the proper step-by-step procedure to build the k3s.exe?

Thanks!

[amenome]

Okay, I think maybe I just didn't read closely enough. Looks like we need to run this on a WSL prompt:

wget https://go.dev/dl/go1.21.9.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.21.9.linux-amd64.tar.gz

sudo apt-get -y install mingw-w64
sudo apt-get -y install zstd
git clone https://github.com/k3s-io/k3s.git
cd k3s
GOOS=windows CXX=x86_64-w64-mingw32-g++ CC=x86_64-w64-mingw32-gcc SKIP_VALIDATE=true ./scripts/ci 

Tested and worked! Had to downgrade to go 1.21.9 though!

[zc-devs]

Thank you for your work and instructions. I managed to compile and run it.

Building

Cross-compiling K3s 1.29 or 1.30 on Ubuntu 22.04 requires packages:

apt update
apt install git curl yq golang mingw-w64 zstd
# optional nano or other editor

Pull the code:

git clone --depth 1 --branch release-1.30 https://github.com/k3s-io/k3s.git
cd k3s/

Using standard build scripts I get error:

yq: error: argument files: can't open '.dependencies[] | select(.name == "golang: upstream version").version': [Errno 2] No such file or directory: '.dependencies[] | select(.name == "golang: upstream version").version'

Therefore, I patched scripts/version.sh

-VERSION_GOLANG="go"$(curl -sL "${DEPENDENCIES_URL}" | yq e '.dependencies[] | select(.name == "golang: upstream version").version' -)
+VERSION_GOLANG="go"$(curl -sL "${DEPENDENCIES_URL}" | yq '.dependencies[] | select(.name == "golang: upstream version").version' -)

Then

GOOS=windows CXX=x86_64-w64-mingw32-g++ CC=x86_64-w64-mingw32-gcc SKIP_VALIDATE=true ./scripts/ci
ls  dist/artifacts/

Running

Trying to run K3s 1.30 I got error:

.\k3s.exe agent --config k3s-config.yaml
time="2024-12-14T02:20:47-08:00" level=info msg="Acquiring lock file C:\\Users\\Administrator\\.rancher\\k3s\\data\\.lock"
time="2024-12-14T02:20:47-08:00" level=info msg="Preparing data dir C:\\Users\\Administrator\\.rancher\\k3s\\data\\761fab38c25deea9988c5556f47af1f585e3868bd55f9e461288f6b07e17ac88"
time="2024-12-14T02:20:50-08:00" level=fatal msg="Failed to validate golang version: incorrect golang build version - kubernetes v1.30.8-rc1 should be built with go\"1.22.9\", runtime version is go1.22.9"
exit status 1

Therefore, I patched pkg/cli/cmds/golang.go

+	upstreamGolang := strings.Replace(version.UpstreamGolang, "\"", "", 2)
-	if v, _, _ := strings.Cut(runtime.Version(), " "); version.UpstreamGolang != v {
+	if v, _, _ := strings.Cut(runtime.Version(), " "); upstreamGolang != v {
		return fmt.Errorf("incorrect golang build version - kubernetes %s should be built with %s, runtime version is %s", k8sVersion, version.UpstreamGolang, v)
	}

Containers feature needs to be enabled

Windows Server 2022

Got error

I1214 03:02:32.477605    2336 kube.go:146] Node controller sync successful
I1214 03:02:32.477605    2336 vxlan_windows.go:126] VXLAN config: Name=flannel.4096 MacPrefix=0E-2A VNI=4096 Port=4789 GBP=false DirectRouting=false
time="2024-12-14T03:02:32-08:00" level=error msg="flannel exited: failed to register flannel network: failed to query HCN version number: this is expected on pre 1803 builds.: hnsCall failed in Win32: The specified module could not be found. (0x7e)"
exit status 1

Windows Server 2025

Enable containers feature

k3s-config:

token: '<token from server node from /var/lib/rancher/k3s/server/node-token>'
server: 'https://<server-node-ip>:6443'
node-external-ip: '<windows-agent-node-ip>'
kubelet-arg=cgroups-per-qos: 'false'
kubelet-arg=enforce-node-allocatable: ''
kubelet-arg=resolv-conf: ''
pause-image: 'mcr.microsoft.com/oss/kubernetes/pause:3.10-windows-ltsc2022-amd64'

Set firewall, run K3s.

Run test deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: windows-app-deployment
spec:
  selector:
    matchLabels:
      app: windows-app
  replicas: 1
  template:
    metadata:
      labels:
        app: windows-app
    spec:
      containers:
        - name: windows-app
          image: mcr.microsoft.com/oss/kubernetes/pause:3.10-windows-ltsc2022-amd64
      nodeSelector:
        kubernetes.io/os: windows

And it works:

---

Useful links:

1. Kubernetes on Windows
2. Windows container version compatibility

[imnotjames]

commenting here because this is what I found when searching on google for k3s windows..

Using standard build scripts I get error:

If it's anything like me, that's probably because you're using the python-yq wrapper around jq, not go-yq.

This also may have caused the problem you ran into with the go version when running k3s -- if you want to keep python-yq rather than switching to go-yq the patch in scripts/version.sh should be:

-VERSION_GOLANG="go"$(curl -sL "${DEPENDENCIES_URL}" | yq e '.dependencies[] | select(.name == "golang: upstream version").version' -)
+VERSION_GOLANG="go"$(curl -sL "${DEPENDENCIES_URL}" | yq -r '.dependencies[] | select(.name == "golang: upstream version").version' -)

Note switching out e (short for "eval") with -r ("raw", omitting the quotes!)

While it would be FANTASTIC to have a build of the windows agent available somewhere this works okay to try it all out!