chore: Hyperv and upstream Containerd package support (#3688)

* hyperv and upstream support * update docs and scripts for hyperv Co-authored-by: Matt Boersma <[email protected]>
Azure · Aug 13, 2020 · 046db34 · 046db34
1 parent dedcef0
commit 046db34
Show file tree

Hide file tree

Showing 26 changed files with 815 additions and 377 deletions.
diff --git a/docs/topics/features.md b/docs/topics/features.md
@@ -514,7 +514,7 @@ We are investigating possible risks & mitigations for when VMs are deprovisioned
 
 Kubernetes 1.18 introduces alpha support for the ContainerD runtime on Windows Server 2019. This is still a work-in-progress tracked in [kubernetes/enhancements#1001](https://github.com/kubernetes/enhancements/issues/1001). This feature in AKS-Engine is for testing the in-development versions of ContainerD and Kubernetes, and is not for production use. Be sure to review [open issues](https://github.com/azure/aks-engine/issues?q=containerd+label%3Awindows+is%3Aopen) if you want to test or contribute to this effort.
 
-Currently it requires URLs to custom ContainerD and CNI plugin builds.
+Containerd now has supported builds at https://github.com/containerd/containerd/releases/tag/v1.4.0-rc.0.  You can find nightly builds of Containerd at https://github.com/marosset/windows-cri-containerd/releases/download/nightly/windows-cri-containerd.zip.
 
 ### Deploying multi-OS clusters with ContainerD
 
@@ -525,15 +525,125 @@ These parameters are all required.
 
 ```json
       "kubernetesConfig": {
-        "networkPlugin": "kubenet",
+        "networkPlugin": "azure",
         "containerRuntime": "containerd",
-        "windowsContainerdURL": "...",
-        "windowsSdnPluginURL": "..."
+        "windowsContainerdURL": "..."
       }
 ```
 
-### Building ContainerD
+### Hyper-v support
+This feature in AKS-Engine is for testing the in-development versions of ContainerD and Kubernetes, and is not for production use. Be sure to review [open issues](https://github.com/azure/aks-engine/issues?q=containerd+label%3Awindows+is%3Aopen) if you want to test or contribute to this effort.
 
-As of March 3, 2020, the ContainerD and network plugin repos don't have public builds available. This repo has a script that will build them from source and create two ZIP files: [build-windows-containerd.sh](../../scripts/build-windows-containerd.sh)
+The current default for a Hyper-V enabled containerD sets process isolated containers as default.  It is required to explicity set the [Build Numbers of the OS](https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-containers/#handling-multiple-windows-versions-in-the-same-cluster) in the api models to add Hyper-V options to containerD.  For example, with the default settings, if your VM OS version is Windows Server 2004 (10.0.19041) and you apply a pod spec with no RuntimeClass setting, you will get a 2004 container running as a process isolated container.  
 
-Upload these ZIP files to a location that your cluster will be able to reach, then put those URLs in `windowsContainerdURL` and `windowsSdnPluginURL` in the AKS-Engine API model shown above.
+To Configure other OS as hyper-v containers in the containerD set the following on the WindowsProfile:
+
+```
+"windowsProfile": {
+      ...
+      "windowsPublisher": "MicrosoftWindowsServer",
+      "windowsOffer": "WindowsServer",
+      "windowsSku": "Datacenter-Core-2004-with-Containers-smalldisk",
+      "imageVersion": "latest",
+      "windowsRuntimes": {
+        "default": "process",
+        "hypervRuntimes": [
+          {"buildNumber": "17763"},
+          {"buildNumber": "19041"}
+        ]
+      }
+    },
+```
+
+Supported Hyperv OS build Id's are: 
+
+- 17763 - Windows Server 2019 (1809)
+- 18362 - Windows Server SAC 1903
+- 18363 - Windows Server SAC 1909
+- 19041 - Windows Server SAC 2004
+
+If you wish to use an OS version for a container below your current Host OS version or explicitly run in a Hyper-v conatiners, you will need to create a RuntimeClass object and map the pod to the RuntimeClass.  Note that Hyper-V support is currently backwards compatible.  You have to have a Host OS that is the same version or newer than the version of the container you wish to run.  Multi-arch container images are not supported; You must have a single arch image if Hyper-V is enabled in containerd.
+
+For example, assuming a Windows Host OS of 2004 (10.0.19041), you can apply the following `RuntimeClass` 
+
+```yaml
+apiVersion: node.k8s.io/v1beta1
+kind: RuntimeClass
+metadata:
+  name: windows-2019
+handler: 'runhcs-wcow-hypervisor-17763'
+scheduling:
+  nodeSelector:
+    kubernetes.io/os: 'windows'
+    kubernetes.io/arch: 'amd64'
+    node.kubernetes.io/windows-build: '10.0.19041'
+  tolerations:
+  - effect: NoSchedule
+    key: os
+    operator: Equal
+    value: "windows"
+```
+
+And then you would be able to run a 2019/1809 (10.0.17763) container by setting the `runtimeClassName` to the `windows-2019` RuntimeClass on the container template:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: iis-ltsc2019
+  labels:
+    app: iis-ltsc2019
+spec:
+  replicas: 1
+  template:
+    metadata:
+      name: iis-ltsc2019
+      labels:
+        app: iis-ltsc2019
+    spec:
+      runtimeClassName: windows-2019
+      containers:
+      - name: iis
+        image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
+        resources:
+          limits:
+            cpu: 1
+            memory: 800m
+          requests:
+            cpu: .1
+            memory: 300m
+        ports:
+          - containerPort: 80
+      nodeSelector:
+        "kubernetes.io/os": windows
+  selector:
+    matchLabels:
+      app: iis-ltsc2019
+
+```
+
+The `handler` names for `RuntimeClass` will be dependent on the `hypervRuntimes` you enabled in the api model and will be in the format of `runhcs-wcow-hypervisor-$buildNumber`.  The possible values (depending on configuration) are:
+
+- runhcs-wcow-process (defaults process isolated for current host OS build number)
+- runhcs-wcow-hypervisor-17763
+- runhcs-wcow-hypervisor-18362
+- runhcs-wcow-hypervisor-18363
+- runhcs-wcow-hypervisor-19041
+
+Current limitations:
+
+- Currently the Runtime handlers are not configurable.  
+- If you specify a handler that does not map the fields in [../../parts/k8s/containerdtemplate.toml](parts/k8s/containerdtemplate.toml), then the container will not start. 
+- If you map to a container version that is higher than your current OS image your container will not start.
+- Multi-arch container images are not supported
+
+You can learn more about RuntimeClasses and the future of the Windows support:
+
+- https://kubernetes.io/docs/concepts/containers/runtime-class/
+- https://github.com/kubernetes/enhancements/blob/master/keps/sig-windows/windows-runtimeclass-support.md
+
+### Building ContainerD with Hyper-V
+
+As of Aug 10, 2020, the ContainerD Hyper-V support doesn't have public builds available. This repo has a script that will build it from source and create a ZIP file: [build-windows-containerd.sh](../../scripts/build-windows-containerd.sh)
+
+Upload these ZIP files to a location that your cluster will be able to reach, then put those URLs in `windowsContainerdURL` in the AKS-Engine API model shown above.
diff --git a/examples/windows/README.md b/examples/windows/README.md
@@ -15,9 +15,9 @@ These cluster definition examples demonstrate how to create customized Docker En
 
 - kubernetes.json - this is the simplest case for a 2-node Windows Kubernetes cluster
 - kubernetes-custom-image.json - example using an existing Azure image for Windows nodes.
-- kubernetes-shared-image.json - exmple using an Azure image from a shared image gallery for Windows nodes.
-- kubernetes-custom-vhd.json - exmaple using a custom VHD (uploaded to an Azure storage account or other accessible location) for Windows nodes.
+- kubernetes-shared-image.json - example using an Azure image from a shared image gallery for Windows nodes.
+- kubernetes-custom-vhd.json - example using a custom VHD (uploaded to an Azure storage account or other accessible location) for Windows nodes.
 - kubernetes-hybrid.json - example with both Windows & Linux nodes in the same cluster
-- kubernetes-hyperv.json - example with 2 Windows nodes with the [alpha Hyper-V isolation support](https://kubernetes.io/docs/getting-started-guides/windows/#hyper-v-containers) enabled
+- kubernetes-hyperv.json - example with 2 Windows nodes with the [experimental Hyper-V isolation support](../../docs/topics/features.md) enabled. Learn more about about [RuntimeClasses for hyper visor selection](https://kubernetes.io/docs/concepts/containers/runtime-class/).
 - kubernetes-wincni.json - example using kubenet plugin on Linux nodes and WinCNI on Windows
 - kubernetes-windows-version.json - example of how to build a cluster with a specific Windows patch version
diff --git a/examples/windows/kubernetes-hybrid.azure-containerd.json b/examples/windows/kubernetes-hybrid.azure-containerd.json
@@ -7,8 +7,7 @@
       "kubernetesConfig": {
         "networkPlugin": "azure",
         "containerRuntime": "containerd",
-        "windowsContainerdURL": "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cri-containerd.zip",
-        "windowsSdnPluginURL": "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cni-containerd.zip"
+        "windowsContainerdURL": "https://github.com/containerd/containerd/releases/download/v1.4.0-rc.0/containerd-1.4.0-rc.0-windows-amd64.tar.gz"
       }
     },
     "masterProfile": {

diff --git a/examples/windows/kubernetes-hybrid.kubenet-containerd.json b/examples/windows/kubernetes-hybrid.kubenet-containerd.json
@@ -7,7 +7,7 @@
       "kubernetesConfig": {
         "networkPlugin": "kubenet",
         "containerRuntime": "containerd",
-        "windowsContainerdURL": "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cri-containerd.zip",
+        "windowsContainerdURL": "https://github.com/containerd/containerd/releases/download/v1.4.0-rc.0/containerd-1.4.0-rc.0-windows-amd64.tar.gz",
         "windowsSdnPluginURL": "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cni-containerd.zip"
       }
     },

diff --git a/examples/windows/kubernetes-hyperv.json b/examples/windows/kubernetes-hyperv.json
@@ -5,37 +5,42 @@
       "orchestratorType": "Kubernetes",
       "orchestratorRelease": "1.15",
       "kubernetesConfig": {
-               "apiServerConfig" : {
-                 "--feature-gates": "HyperVContainer=true"
-               },
-               "kubeletConfig" : {
-                    "--feature-gates": "HyperVContainer=true"
-                }
-          }
+        "networkPlugin": "azure",
+        "containerRuntime": "containerd",
+        "windowsContainerdURL": "https://k8swin.blob.core.windows.net/k8s-windows/containerd/containerplat-aks-test-0.0.8.zip"
+      }
     },
     "masterProfile": {
       "count": 1,
       "dnsPrefix": "hypervtest",
       "vmSize": "Standard_D2_v3"
     },
     "agentPoolProfiles": [
-	    {
-		"name": "windowspool",
-		"count": 2,
-		"vmSize": "Standard_D2_v3",
-		"availabilityProfile": "AvailabilitySet",
-		"osType": "Windows",
-		"osDiskSizeGB": 128,
-		"extensions": [
-		    {
-                        "name": "winrm"
-		    }
-		]
-	    }
+      {
+        "name": "windowspool",
+        "count": 2,
+        "vmSize": "Standard_D4s_v3",
+        "availabilityProfile": "AvailabilitySet",
+        "osType": "Windows",
+        "osDiskSizeGB": 128
+      }
     ],
     "windowsProfile": {
       "adminUsername": "azureuser",
-      "adminPassword": "replacepassword1234$"
+      "adminPassword": "replacepassword1234$",
+      "enableAutomaticUpdates": false,
+      "sshEnabled": true,
+      "windowsPublisher": "MicrosoftWindowsServer",
+      "windowsOffer": "WindowsServer",
+      "windowsSku": "Datacenter-Core-2004-with-Containers-smalldisk",
+      "imageVersion": "latest",
+      "windowsRuntimes": {
+        "default": "process",
+        "hypervRuntimes": [
+          {"buildNumber": "17763"},
+          {"buildNumber": "19041"}
+        ]
+      }
     },
     "linuxProfile": {
       "adminUsername": "azureuser",
@@ -50,12 +55,6 @@
     "servicePrincipalProfile": {
       "clientId": "",
       "secret": ""
-    },
-    "extensionProfiles": [
-      {
-        "name": "winrm",
-        "version": "v1"
-      }
-    ]
+    }
   }
-}
+}
diff --git a/parts/k8s/containerdtemplate.toml b/parts/k8s/containerdtemplate.toml
@@ -0,0 +1,67 @@
+root = "C:\\ProgramData\\containerd\\root"
+state = "C:\\ProgramData\\containerd\\state"
+
+[grpc]
+  address = "\\\\.\\pipe\\containerd-containerd"
+  max_recv_message_size = 16777216
+  max_send_message_size = 16777216
+
+[ttrpc]
+  address = ""
+
+[debug]
+  address = ""
+  level = "debug"
+
+[metrics]
+  address = ""
+  grpc_histogram = false
+
+[cgroup]
+  path = ""
+
+[plugins]
+  [plugins.cri]
+    stream_server_address = "127.0.0.1"
+    stream_server_port = "0"
+    enable_selinux = false
+    sandbox_image = "{{pauseImage}}-windows-{{currentversion}}-amd64"
+    stats_collect_period = 10
+    systemd_cgroup = false
+    enable_tls_streaming = false
+    max_container_log_line_size = 16384
+    [plugins.cri.containerd]
+      snapshotter = "windows"
+      no_pivot = false
+      [plugins.cri.containerd.default_runtime]
+        runtime_type = "io.containerd.runhcs.v1"
+        [plugins.cri.containerd.default_runtime.options]
+          Debug = true
+          DebugType = 2
+          SandboxImage = "{{pauseImage}}-windows-{{currentversion}}-amd64"
+          SandboxPlatform = "windows/amd64"
+          SandboxIsolation = {{sandboxIsolation}}
+      [plugins.cri.containerd.runtimes]
+        [plugins.cri.containerd.runtimes.runhcs-wcow-process]
+          runtime_type = "io.containerd.runhcs.v1"
+          [plugins.cri.containerd.runtimes.runhcs-wcow-process.options]
+            Debug = true
+            DebugType = 2
+            SandboxImage = "{{pauseImage}}-windows-{{currentversion}}-amd64"
+            SandboxPlatform = "windows/amd64"
+{{hypervisors}}
+    [plugins.cri.cni]
+      bin_dir = "{{cnibin}}"
+      conf_dir = "{{cniconf}}"
+    [plugins.cri.registry]
+      [plugins.cri.registry.mirrors]
+        [plugins.cri.registry.mirrors."docker.io"]
+          endpoint = ["https://registry-1.docker.io"]
+  [plugins.diff-service]
+    default = ["windows"]
+  [plugins.scheduler]
+    pause_threshold = 0.02
+    deletion_threshold = 0
+    mutation_threshold = 100
+    schedule_delay = "0s"
+    startup_delay = "100ms"
diff --git a/parts/k8s/kuberneteswindowssetup.ps1 b/parts/k8s/kuberneteswindowssetup.ps1
@@ -76,6 +76,8 @@ $global:DockerVersion = "{{WrapAsParameter "windowsDockerVersion"}}"
 
 ## ContainerD Usage
 $global:ContainerRuntime = "{{WrapAsParameter "containerRuntime"}}"
+$global:DefaultContainerdRuntimeHandler = "{{WrapAsParameter "defaultContainerdRuntimeHandler"}}"
+$global:HypervRuntimeHandlers = "{{WrapAsParameter "hypervRuntimeHandlers"}}"
 
 ## VM configuration passed by Azure
 $global:WindowsTelemetryGUID = "{{WrapAsParameter "windowsTelemetryGUID"}}"
@@ -180,6 +182,7 @@ try
     # to the windows machine, and run the script manually to watch
     # the output.
     if ($true) {
+        Write-Log ".\CustomDataSetupScript.ps1 -MasterIP $MasterIP -KubeDnsServiceIp $KubeDnsServiceIp -MasterFQDNPrefix $MasterFQDNPrefix -Location $Location -AgentKey $AgentKey -AADClientId $AADClientId -AADClientSecret $AADClientSecret -NetworkAPIVersion $NetworkAPIVersion -TargetEnvironment $TargetEnvironment"
         Write-Log "Provisioning $global:DockerServiceName... with IP $MasterIP"
 
         $global:globalTimer = [System.Diagnostics.Stopwatch]::StartNew()