[Release-1.26] Windows port of K3s agent role

[manuelbuil]

Backport: #7258

[mdrahman-suse]

Validated on release-1.26 branch with commit 4e77f40

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-1031-aws x86_64)
Windows 2022 

Cluster Configuration:

1 Linux server, 1 Linux agent and 1 Windows agent

Config.yaml:

write-kubeconfig-mode: 644
cluster-init: true
node-external-ip: <public-ip>
disable-network-policy: true
token: summerheat

Additional files

pod_client.yaml (clusterIP service)

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: client
  name: client-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      containers:
      - image: ranchertest/mytestcontainer
        imagePullPolicy: Always
        name: client-curl
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - client
            topologyKey: kubernetes.io/hostname
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
---
apiVersion: v1
kind: Service
metadata:
  name: client-curl
  labels:
    app: client
    service: client-curl
spec:
  type: ClusterIP
  selector:
    app: client
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 80

win-app.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: windows-app-deployment
spec:
  selector:
    matchLabels:
      app: windows-app
  replicas: 2
  template:
    metadata:
      labels:
        app: windows-app
    spec:
      containers:
        - name: windows-app
          image: mbuilsuse/pstools:v0.2.0
          ports:
            - containerPort: 3000
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - windows
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: windows-app-svc
  name: windows-app-svc
  namespace: default
spec:
  type: NodePort
  ports:
    - port: 3000
      nodePort: 30096
      name: http
  selector:
    app: windows-app

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s on server and agent nodes
3. Build k3s.exe as suggested here: [K3s][Windows Port] Build script, multi-call binary, and Flannel #7259 (comment)
4. Copy k3s.exe on Windows node
5. SSH in to Windows node, activate powershell and run the commands in Windows node

New-NetFirewallRule -Name k3s -DisplayName 'k3s' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 10250
New-NetFirewallRule -Name k3sOverlayTraffic4789UDP -DisplayName "K3s Overlay Traffic 4789 UDP" -Enabled True -Direction Inbound -Protocol UDP -Action Allow -LocalPort 4789

6. Setup config

Set-Content -Path c:/Users/Administrator/k3s-config.yaml -Value @"
token: '<K10-token-from-server>'
server: 'https://<server-public-ip>:6443'
node-external-ip: '<public-ip>'
kubelet-arg=cgroups-per-qos: 'false'
kubelet-arg=enforce-node-allocatable: ''
kubelet-arg=resolv-conf: ''
pause-image: 'mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2022-amd64'
"@

7. Run k3s.exe using command .\k3s.exe agent --config k3s-config.yaml
8. Ensure Windows node joins the cluster
9. Apply the deployments and ensure they are up and running

Validations

$ k3s -v
k3s version v1.26.9+k3s-4e77f40d (4e77f40d)
go version go1.20.8

$ kubectl get node,pods -A -o wide
NAME               STATUS   ROLES                       AGE   VERSION                INTERNAL-IP     EXTERNAL-IP    OS-IMAGE                         KERNEL-VERSION    CONTAINER-RUNTIME
node/agent1        Ready    <none>                      25m   v1.26.9+k3s-4e77f40d   xxx.xx.xx.153   x.xxx.xx.57    Ubuntu 22.04.2 LTS               5.15.0-1031-aws   containerd://1.7.7-k3s1.26
node/ip-ac1f2610   Ready    <none>                      16m   v1.26.9+k3s-4e77f40d   xxx.xx.xx.229   x.xx.xxx.119   Windows Server 2022 Datacenter   10.0.20348.350    containerd://1.7.7-k3s1.26
node/server1       Ready    control-plane,etcd,master   29m   v1.26.9+k3s-4e77f40d   xxx.xx.xx.145   x.xxx.xx.67    Ubuntu 22.04.2 LTS               5.15.0-1031-aws   containerd://1.7.7-k3s1.26

NAMESPACE     NAME                                          READY   STATUS            RESTARTS   AGE    IP          NODE          NOMINATED NODE   READINESS GATES
default       pod/client-deployment-7dc79494b4-5rd8x        1/1     Running           0          12m    xx.xx.1.3   agent1        <none>           <none>
default       pod/client-deployment-7dc79494b4-z7rmw        1/1     Running           0          12m    xx.xx.0.9   server1       <none>           <none>
default       pod/windows-app-deployment-68f979c8c4-gzz65   1/1     Running           0          12m    xx.xx.2.3   ip-ac1f2610   <none>           <none>
default       pod/windows-app-deployment-68f979c8c4-kb2jh   1/1     Running           0          12m    xx.xx.2.4   ip-ac1f2610   <none>           <none>
kube-system   pod/coredns-59b4f5bbd5-849kl                  1/1     Running           0          28m    xx.xx.0.6   server1       <none>           <none>
kube-system   pod/helm-install-traefik-crd-z5kcp            0/1     Completed         0          28m    xx.xx.0.5   server1       <none>           <none>
kube-system   pod/helm-install-traefik-jf27q                0/1     Completed         1          28m    xx.xx.0.3   server1       <none>           <none>
kube-system   pod/local-path-provisioner-76d776f6f9-6h5gm   1/1     Running           0          28m    xx.xx.0.4   server1       <none>           <none>
kube-system   pod/metrics-server-68cf49699b-5nr8j           1/1     Running           0          28m    xx.xx.0.2   server1       <none>           <none>
kube-system   pod/svclb-traefik-a131f824-nnf6g              0/2     SysctlForbidden   0          6m1s   <none>      ip-ac1f2610   <none>           <none>
kube-system   pod/svclb-traefik-a131f824-pjs5r              2/2     Running           0          25m    xx.xx.1.2   agent1        <none>           <none>
kube-system   pod/svclb-traefik-a131f824-qn4s4              2/2     Running           0          28m    xx.xx.0.7   server1       <none>           <none>
kube-system   pod/traefik-f75d47c4b-xck74                   1/1     Running           0          28m    xx.xx.0.8   server1       <none>           <none>

Additional Observations

• svclb pod in SysctlForbidden status: [K3s][Windows Port] Build script, multi-call binary, and Flannel #7259 (comment)
• With Windows 2019, observed the error below when tried to deploy windows workload. (Due to wrong pause image)
  The error was not observer on Windows 2022

$ kubectl describe -n default pod/windows-app-deployment-6964ff4fb8-574ck
...
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               4m41s                  default-scheduler  Successfully assigned default/windows-app-deployment-6964ff4fb8-574ck to ip-ac1f0e4f
  Warning  FailedCreatePodSandBox  4m26s (x2 over 4m40s)  kubelet            Failed to create pod sandbox: rpc error: code = NotFound desc = failed to get sandbox image "mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2019-amd64": failed to pull image "mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2019-amd64": failed to pull and unpack image "mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2019-amd64": failed to resolve reference "mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2019-amd64": mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2019-amd64: not found