diff --git a/docs/topics/features.md b/docs/topics/features.md index 127cba5c4c..90af6ffed0 100644 --- a/docs/topics/features.md +++ b/docs/topics/features.md @@ -514,7 +514,7 @@ We are investigating possible risks & mitigations for when VMs are deprovisioned Kubernetes 1.18 introduces alpha support for the ContainerD runtime on Windows Server 2019. This is still a work-in-progress tracked in [kubernetes/enhancements#1001](https://github.com/kubernetes/enhancements/issues/1001). This feature in AKS-Engine is for testing the in-development versions of ContainerD and Kubernetes, and is not for production use. Be sure to review [open issues](https://github.com/azure/aks-engine/issues?q=containerd+label%3Awindows+is%3Aopen) if you want to test or contribute to this effort. -Currently it requires URLs to custom ContainerD and CNI plugin builds. +Containerd now has supported builds at https://github.com/containerd/containerd/releases/tag/v1.4.0-rc.0. You can find nightly builds of Containerd at https://github.com/marosset/windows-cri-containerd/releases/download/nightly/windows-cri-containerd.zip. ### Deploying multi-OS clusters with ContainerD @@ -525,15 +525,125 @@ These parameters are all required. ```json "kubernetesConfig": { - "networkPlugin": "kubenet", + "networkPlugin": "azure", "containerRuntime": "containerd", - "windowsContainerdURL": "...", - "windowsSdnPluginURL": "..." + "windowsContainerdURL": "..." } ``` -### Building ContainerD +### Hyper-v support +This feature in AKS-Engine is for testing the in-development versions of ContainerD and Kubernetes, and is not for production use. Be sure to review [open issues](https://github.com/azure/aks-engine/issues?q=containerd+label%3Awindows+is%3Aopen) if you want to test or contribute to this effort. -As of March 3, 2020, the ContainerD and network plugin repos don't have public builds available. This repo has a script that will build them from source and create two ZIP files: [build-windows-containerd.sh](../../scripts/build-windows-containerd.sh) +The current default for a Hyper-V enabled containerD sets process isolated containers as default. It is required to explicity set the [Build Numbers of the OS](https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-containers/#handling-multiple-windows-versions-in-the-same-cluster) in the api models to add Hyper-V options to containerD. For example, with the default settings, if your VM OS version is Windows Server 2004 (10.0.19041) and you apply a pod spec with no RuntimeClass setting, you will get a 2004 container running as a process isolated container. -Upload these ZIP files to a location that your cluster will be able to reach, then put those URLs in `windowsContainerdURL` and `windowsSdnPluginURL` in the AKS-Engine API model shown above. +To Configure other OS as hyper-v containers in the containerD set the following on the WindowsProfile: + +``` +"windowsProfile": { + ... + "windowsPublisher": "MicrosoftWindowsServer", + "windowsOffer": "WindowsServer", + "windowsSku": "Datacenter-Core-2004-with-Containers-smalldisk", + "imageVersion": "latest", + "windowsRuntimes": { + "default": "process", + "hypervRuntimes": [ + {"buildNumber": "17763"}, + {"buildNumber": "19041"} + ] + } + }, +``` + +Supported Hyperv OS build Id's are: + +- 17763 - Windows Server 2019 (1809) +- 18362 - Windows Server SAC 1903 +- 18363 - Windows Server SAC 1909 +- 19041 - Windows Server SAC 2004 + +If you wish to use an OS version for a container below your current Host OS version or explicitly run in a Hyper-v conatiners, you will need to create a RuntimeClass object and map the pod to the RuntimeClass. Note that Hyper-V support is currently backwards compatible. You have to have a Host OS that is the same version or newer than the version of the container you wish to run. Multi-arch container images are not supported; You must have a single arch image if Hyper-V is enabled in containerd. + +For example, assuming a Windows Host OS of 2004 (10.0.19041), you can apply the following `RuntimeClass` + +```yaml +apiVersion: node.k8s.io/v1beta1 +kind: RuntimeClass +metadata: + name: windows-2019 +handler: 'runhcs-wcow-hypervisor-17763' +scheduling: + nodeSelector: + kubernetes.io/os: 'windows' + kubernetes.io/arch: 'amd64' + node.kubernetes.io/windows-build: '10.0.19041' + tolerations: + - effect: NoSchedule + key: os + operator: Equal + value: "windows" +``` + +And then you would be able to run a 2019/1809 (10.0.17763) container by setting the `runtimeClassName` to the `windows-2019` RuntimeClass on the container template: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: iis-ltsc2019 + labels: + app: iis-ltsc2019 +spec: + replicas: 1 + template: + metadata: + name: iis-ltsc2019 + labels: + app: iis-ltsc2019 + spec: + runtimeClassName: windows-2019 + containers: + - name: iis + image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019 + resources: + limits: + cpu: 1 + memory: 800m + requests: + cpu: .1 + memory: 300m + ports: + - containerPort: 80 + nodeSelector: + "kubernetes.io/os": windows + selector: + matchLabels: + app: iis-ltsc2019 + +``` + +The `handler` names for `RuntimeClass` will be dependent on the `hypervRuntimes` you enabled in the api model and will be in the format of `runhcs-wcow-hypervisor-$buildNumber`. The possible values (depending on configuration) are: + +- runhcs-wcow-process (defaults process isolated for current host OS build number) +- runhcs-wcow-hypervisor-17763 +- runhcs-wcow-hypervisor-18362 +- runhcs-wcow-hypervisor-18363 +- runhcs-wcow-hypervisor-19041 + +Current limitations: + +- Currently the Runtime handlers are not configurable. +- If you specify a handler that does not map the fields in [../../parts/k8s/containerdtemplate.toml](parts/k8s/containerdtemplate.toml), then the container will not start. +- If you map to a container version that is higher than your current OS image your container will not start. +- Multi-arch container images are not supported + +You can learn more about RuntimeClasses and the future of the Windows support: + +- https://kubernetes.io/docs/concepts/containers/runtime-class/ +- https://github.com/kubernetes/enhancements/blob/master/keps/sig-windows/windows-runtimeclass-support.md + +### Building ContainerD with Hyper-V + +As of Aug 10, 2020, the ContainerD Hyper-V support doesn't have public builds available. This repo has a script that will build it from source and create a ZIP file: [build-windows-containerd.sh](../../scripts/build-windows-containerd.sh) + +Upload these ZIP files to a location that your cluster will be able to reach, then put those URLs in `windowsContainerdURL` in the AKS-Engine API model shown above. diff --git a/examples/windows/README.md b/examples/windows/README.md index 63b829e805..8fd459a74e 100644 --- a/examples/windows/README.md +++ b/examples/windows/README.md @@ -15,9 +15,9 @@ These cluster definition examples demonstrate how to create customized Docker En - kubernetes.json - this is the simplest case for a 2-node Windows Kubernetes cluster - kubernetes-custom-image.json - example using an existing Azure image for Windows nodes. -- kubernetes-shared-image.json - exmple using an Azure image from a shared image gallery for Windows nodes. -- kubernetes-custom-vhd.json - exmaple using a custom VHD (uploaded to an Azure storage account or other accessible location) for Windows nodes. +- kubernetes-shared-image.json - example using an Azure image from a shared image gallery for Windows nodes. +- kubernetes-custom-vhd.json - example using a custom VHD (uploaded to an Azure storage account or other accessible location) for Windows nodes. - kubernetes-hybrid.json - example with both Windows & Linux nodes in the same cluster -- kubernetes-hyperv.json - example with 2 Windows nodes with the [alpha Hyper-V isolation support](https://kubernetes.io/docs/getting-started-guides/windows/#hyper-v-containers) enabled +- kubernetes-hyperv.json - example with 2 Windows nodes with the [experimental Hyper-V isolation support](../../docs/topics/features.md) enabled. Learn more about about [RuntimeClasses for hyper visor selection](https://kubernetes.io/docs/concepts/containers/runtime-class/). - kubernetes-wincni.json - example using kubenet plugin on Linux nodes and WinCNI on Windows - kubernetes-windows-version.json - example of how to build a cluster with a specific Windows patch version diff --git a/examples/windows/kubernetes-hybrid.azure-containerd.json b/examples/windows/kubernetes-hybrid.azure-containerd.json index 46107da68d..c4d6b66ed3 100644 --- a/examples/windows/kubernetes-hybrid.azure-containerd.json +++ b/examples/windows/kubernetes-hybrid.azure-containerd.json @@ -7,8 +7,7 @@ "kubernetesConfig": { "networkPlugin": "azure", "containerRuntime": "containerd", - "windowsContainerdURL": "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cri-containerd.zip", - "windowsSdnPluginURL": "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cni-containerd.zip" + "windowsContainerdURL": "https://github.com/containerd/containerd/releases/download/v1.4.0-rc.0/containerd-1.4.0-rc.0-windows-amd64.tar.gz" } }, "masterProfile": { diff --git a/examples/windows/kubernetes-hybrid.kubenet-containerd.json b/examples/windows/kubernetes-hybrid.kubenet-containerd.json index 6597e30797..af1f9c3343 100644 --- a/examples/windows/kubernetes-hybrid.kubenet-containerd.json +++ b/examples/windows/kubernetes-hybrid.kubenet-containerd.json @@ -7,7 +7,7 @@ "kubernetesConfig": { "networkPlugin": "kubenet", "containerRuntime": "containerd", - "windowsContainerdURL": "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cri-containerd.zip", + "windowsContainerdURL": "https://github.com/containerd/containerd/releases/download/v1.4.0-rc.0/containerd-1.4.0-rc.0-windows-amd64.tar.gz", "windowsSdnPluginURL": "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cni-containerd.zip" } }, diff --git a/examples/windows/kubernetes-hyperv.json b/examples/windows/kubernetes-hyperv.json index 2877a1dd5c..249c5798d7 100644 --- a/examples/windows/kubernetes-hyperv.json +++ b/examples/windows/kubernetes-hyperv.json @@ -5,13 +5,10 @@ "orchestratorType": "Kubernetes", "orchestratorRelease": "1.15", "kubernetesConfig": { - "apiServerConfig" : { - "--feature-gates": "HyperVContainer=true" - }, - "kubeletConfig" : { - "--feature-gates": "HyperVContainer=true" - } - } + "networkPlugin": "azure", + "containerRuntime": "containerd", + "windowsContainerdURL": "https://k8swin.blob.core.windows.net/k8s-windows/containerd/containerplat-aks-test-0.0.8.zip" + } }, "masterProfile": { "count": 1, @@ -19,23 +16,31 @@ "vmSize": "Standard_D2_v3" }, "agentPoolProfiles": [ - { - "name": "windowspool", - "count": 2, - "vmSize": "Standard_D2_v3", - "availabilityProfile": "AvailabilitySet", - "osType": "Windows", - "osDiskSizeGB": 128, - "extensions": [ - { - "name": "winrm" - } - ] - } + { + "name": "windowspool", + "count": 2, + "vmSize": "Standard_D4s_v3", + "availabilityProfile": "AvailabilitySet", + "osType": "Windows", + "osDiskSizeGB": 128 + } ], "windowsProfile": { "adminUsername": "azureuser", - "adminPassword": "replacepassword1234$" + "adminPassword": "replacepassword1234$", + "enableAutomaticUpdates": false, + "sshEnabled": true, + "windowsPublisher": "MicrosoftWindowsServer", + "windowsOffer": "WindowsServer", + "windowsSku": "Datacenter-Core-2004-with-Containers-smalldisk", + "imageVersion": "latest", + "windowsRuntimes": { + "default": "process", + "hypervRuntimes": [ + {"buildNumber": "17763"}, + {"buildNumber": "19041"} + ] + } }, "linuxProfile": { "adminUsername": "azureuser", @@ -50,12 +55,6 @@ "servicePrincipalProfile": { "clientId": "", "secret": "" - }, - "extensionProfiles": [ - { - "name": "winrm", - "version": "v1" - } - ] + } } -} +} \ No newline at end of file diff --git a/parts/k8s/containerdtemplate.toml b/parts/k8s/containerdtemplate.toml new file mode 100644 index 0000000000..c9c0e11a5b --- /dev/null +++ b/parts/k8s/containerdtemplate.toml @@ -0,0 +1,67 @@ +root = "C:\\ProgramData\\containerd\\root" +state = "C:\\ProgramData\\containerd\\state" + +[grpc] + address = "\\\\.\\pipe\\containerd-containerd" + max_recv_message_size = 16777216 + max_send_message_size = 16777216 + +[ttrpc] + address = "" + +[debug] + address = "" + level = "debug" + +[metrics] + address = "" + grpc_histogram = false + +[cgroup] + path = "" + +[plugins] + [plugins.cri] + stream_server_address = "127.0.0.1" + stream_server_port = "0" + enable_selinux = false + sandbox_image = "{{pauseImage}}-windows-{{currentversion}}-amd64" + stats_collect_period = 10 + systemd_cgroup = false + enable_tls_streaming = false + max_container_log_line_size = 16384 + [plugins.cri.containerd] + snapshotter = "windows" + no_pivot = false + [plugins.cri.containerd.default_runtime] + runtime_type = "io.containerd.runhcs.v1" + [plugins.cri.containerd.default_runtime.options] + Debug = true + DebugType = 2 + SandboxImage = "{{pauseImage}}-windows-{{currentversion}}-amd64" + SandboxPlatform = "windows/amd64" + SandboxIsolation = {{sandboxIsolation}} + [plugins.cri.containerd.runtimes] + [plugins.cri.containerd.runtimes.runhcs-wcow-process] + runtime_type = "io.containerd.runhcs.v1" + [plugins.cri.containerd.runtimes.runhcs-wcow-process.options] + Debug = true + DebugType = 2 + SandboxImage = "{{pauseImage}}-windows-{{currentversion}}-amd64" + SandboxPlatform = "windows/amd64" +{{hypervisors}} + [plugins.cri.cni] + bin_dir = "{{cnibin}}" + conf_dir = "{{cniconf}}" + [plugins.cri.registry] + [plugins.cri.registry.mirrors] + [plugins.cri.registry.mirrors."docker.io"] + endpoint = ["https://registry-1.docker.io"] + [plugins.diff-service] + default = ["windows"] + [plugins.scheduler] + pause_threshold = 0.02 + deletion_threshold = 0 + mutation_threshold = 100 + schedule_delay = "0s" + startup_delay = "100ms" \ No newline at end of file diff --git a/parts/k8s/kuberneteswindowssetup.ps1 b/parts/k8s/kuberneteswindowssetup.ps1 index e435ea126e..a40109d94e 100644 --- a/parts/k8s/kuberneteswindowssetup.ps1 +++ b/parts/k8s/kuberneteswindowssetup.ps1 @@ -76,6 +76,8 @@ $global:DockerVersion = "{{WrapAsParameter "windowsDockerVersion"}}" ## ContainerD Usage $global:ContainerRuntime = "{{WrapAsParameter "containerRuntime"}}" +$global:DefaultContainerdRuntimeHandler = "{{WrapAsParameter "defaultContainerdRuntimeHandler"}}" +$global:HypervRuntimeHandlers = "{{WrapAsParameter "hypervRuntimeHandlers"}}" ## VM configuration passed by Azure $global:WindowsTelemetryGUID = "{{WrapAsParameter "windowsTelemetryGUID"}}" @@ -180,6 +182,7 @@ try # to the windows machine, and run the script manually to watch # the output. if ($true) { + Write-Log ".\CustomDataSetupScript.ps1 -MasterIP $MasterIP -KubeDnsServiceIp $KubeDnsServiceIp -MasterFQDNPrefix $MasterFQDNPrefix -Location $Location -AgentKey $AgentKey -AADClientId $AADClientId -AADClientSecret $AADClientSecret -NetworkAPIVersion $NetworkAPIVersion -TargetEnvironment $TargetEnvironment" Write-Log "Provisioning $global:DockerServiceName... with IP $MasterIP" $global:globalTimer = [System.Diagnostics.Stopwatch]::StartNew() diff --git a/parts/k8s/windowscontainerdfunc.ps1 b/parts/k8s/windowscontainerdfunc.ps1 index 5039ddbdc4..ec8d9a3f5c 100644 --- a/parts/k8s/windowscontainerdfunc.ps1 +++ b/parts/k8s/windowscontainerdfunc.ps1 @@ -21,6 +21,67 @@ function RegisterContainerDService { } } +function CreateHypervisorRuntime { + Param( + [Parameter(Mandatory = $true)][string] + $image, + [Parameter(Mandatory = $true)][string] + $version, + [Parameter(Mandatory = $true)][string] + $buildNumber + ) + + return @" + [plugins.cri.containerd.runtimes.runhcs-wcow-hypervisor-$buildnumber] + runtime_type = "io.containerd.runhcs.v1" + [plugins.cri.containerd.runtimes.runhcs-wcow-hypervisor-$buildnumber.options] + Debug = true + DebugType = 2 + SandboxImage = "$image-windows-$version-amd64" + SandboxPlatform = "windows/amd64" + SandboxIsolation = 1 +"@ +} + +function CreateHypervisorRuntimes { + Param( + [Parameter(Mandatory = $true)][string[]] + $builds, + [Parameter(Mandatory = $true)][string] + $image + ) + + Write-Host "Adding hyperv runtimes $builds" + $hypervRuntimes = "" + ForEach ($buildNumber in $builds) { + $windowsVersion = Select-Windows-Version -buildNumber $buildNumber + $runtime = createHypervisorRuntime -image $pauseImage -version $windowsVersion -buildNumber $buildNumber + if ($hypervRuntimes -eq "") { + $hypervRuntimes = $runtime + } + else { + $hypervRuntimes = $hypervRuntimes + "`r`n" + $runtime + } + } + + return $hypervRuntimes +} + +function Select-Windows-Version { + param ( + [Parameter()] + [string] + $buildNumber + ) + + switch ($buildNumber) { + "17763" { return "1809" } + "18362" { return "1903" } + "18363" { return "1909" } + "19041" { return "2004" } + Default { return "" } + } +} function Install-Containerd { Param( @@ -39,130 +100,60 @@ function Install-Containerd { } # TODO: check if containerd is already installed and is the same version before this. - $zipfile = [Io.path]::Combine($ENV:TEMP, "containerd.zip") - DownloadFileOverHttp -Url $ContainerdUrl -DestinationPath $zipfile - Expand-Archive -path $zipfile -DestinationPath $global:ContainerdInstallLocation -Force - del $zipfile + + # Extract the package + if ($ContainerdUrl.endswith(".zip")) { + $zipfile = [Io.path]::Combine($ENV:TEMP, "containerd.zip") + DownloadFileOverHttp -Url $ContainerdUrl -DestinationPath $zipfile + Expand-Archive -path $zipfile -DestinationPath $global:ContainerdInstallLocation -Force + del $zipfile + } + elseif ($ContainerdUrl.endswith(".tar.gz")) { + # upstream containerd package is a tar + $tarfile = [Io.path]::Combine($ENV:TEMP, "containerd.tar.gz") + DownloadFileOverHttp -Url $ContainerdUrl -DestinationPath $tarfile + mkdir -Force "C:\Program Files\containerd" + tar -xzf $tarfile -C $global:ContainerdInstallLocation + mv $global:ContainerdInstallLocation\bin\* $global:ContainerdInstallLocation\ + del $tarfile + del -Recurse -Force $global:ContainerdInstallLocation\bin + } + # get configuration options Add-SystemPathEntry $global:ContainerdInstallLocation - - # TODO: remove if the node comes up without this code - # $configDir = [Io.Path]::Combine($ENV:ProgramData, "containerd") - # if (-Not (Test-Path $configDir)) { - # mkdir $configDir - # } - - # TODO: call containerd.exe dump config, then modify instead of starting with hardcoded + $cdbinary = Join-Path $global:ContainerdInstallLocation containerd.exe $configFile = [Io.Path]::Combine($global:ContainerdInstallLocation, "config.toml") - $clusterConfig = ConvertFrom-Json ((Get-Content $global:KubeClusterConfigPath -ErrorAction Stop) | Out-String) $pauseImage = $clusterConfig.Cri.Images.Pause + $formatedbin = $(($CNIBinDir).Replace("\", "/")) + $formatedconf = $(($CNIConfDir).Replace("\", "/")) + $sandboxIsolation = 0 + $windowsVersion = (Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion").ReleaseId + $hypervRuntimes = "" + $hypervHandlers = $global:HypervRuntimeHandlers.split(",", [System.StringSplitOptions]::RemoveEmptyEntries) + + # configure + if ($global:DefaultContainerdRuntimeHandler -eq "hyperv") { + Write-Host "default runtime for containerd set to hyperv" + $sandboxIsolation = 1 + } - @" -version = 2 -root = "C:\\ProgramData\\containerd\\root" -state = "C:\\ProgramData\\containerd\\state" -plugin_dir = "" -disabled_plugins = [] -required_plugins = [] -oom_score = 0 - -[grpc] - address = "\\\\.\\pipe\\containerd-containerd" - tcp_address = "" - tcp_tls_cert = "" - tcp_tls_key = "" - uid = 0 - gid = 0 - max_recv_message_size = 16777216 - max_send_message_size = 16777216 - -[ttrpc] - address = "" - uid = 0 - gid = 0 - -[debug] - address = "" - uid = 0 - gid = 0 - level = "" - -[metrics] - address = "" - grpc_histogram = false - -[cgroup] - path = "" - -[timeouts] - "io.containerd.timeout.shim.cleanup" = "5s" - "io.containerd.timeout.shim.load" = "5s" - "io.containerd.timeout.shim.shutdown" = "3s" - "io.containerd.timeout.task.state" = "2s" - -[plugins] - [plugins."io.containerd.gc.v1.scheduler"] - pause_threshold = 0.02 - deletion_threshold = 0 - mutation_threshold = 100 - schedule_delay = "0s" - startup_delay = "100ms" - [plugins."io.containerd.grpc.v1.cri"] - disable_tcp_service = true - stream_server_address = "127.0.0.1" - stream_server_port = "0" - stream_idle_timeout = "4h0m0s" - enable_selinux = false - sandbox_image = "$pauseImage" - stats_collect_period = 10 - systemd_cgroup = false - enable_tls_streaming = false - max_container_log_line_size = 16384 - disable_cgroup = false - disable_apparmor = false - restrict_oom_score_adj = false - max_concurrent_downloads = 3 - disable_proc_mount = false - [plugins."io.containerd.grpc.v1.cri".containerd] - snapshotter = "windows" - default_runtime_name = "runhcs-wcow-process" - no_pivot = false - [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime] - runtime_type = "" - runtime_engine = "" - runtime_root = "" - privileged_without_host_devices = false - [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime] - runtime_type = "" - runtime_engine = "" - runtime_root = "" - privileged_without_host_devices = false - [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] - [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-process] - runtime_type = "io.containerd.runhcs.v1" - runtime_engine = "" - runtime_root = "" - privileged_without_host_devices = false - [plugins."io.containerd.grpc.v1.cri".cni] - bin_dir = "$(($CNIBinDir).Replace("\","//"))" - conf_dir = "$(($CNIConfDir).Replace("\","//"))" - max_conf_num = 1 - conf_template = "" - [plugins."io.containerd.grpc.v1.cri".registry] - [plugins."io.containerd.grpc.v1.cri".registry.mirrors] - [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] - endpoint = ["https://registry-1.docker.io"] - [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming] - tls_cert_file = "" - tls_key_file = "" - [plugins."io.containerd.metadata.v1.bolt"] - content_sharing_policy = "shared" - [plugins."io.containerd.runtime.v2.task"] - platforms = ["windows/amd64", "linux/amd64"] - [plugins."io.containerd.service.v1.diff-service"] - default = ["windows", "windows-lcow"] -"@ | Out-File -Encoding ascii $configFile + $template = Get-Content -Path "c:\AzureData\k8s\containerdtemplate.toml" + if ($sandboxIsolation -eq 0 -And $hypervHandlers.Count -eq 0) { + # remove the value hypervisor place holder + $template = $template | Select-String -Pattern 'hypervisors' -NotMatch | Out-String + } + else { + $hypervRuntimes = CreateHypervisorRuntimes -builds @($hypervHandlers) -image $pauseImage + } + + $template.Replace('{{sandboxIsolation}}', $sandboxIsolation). + Replace('{{pauseImage}}', $pauseImage). + Replace('{{hypervisors}}', $hypervRuntimes). + Replace('{{cnibin}}', $formatedbin). + Replace('{{cniconf}}', $formatedconf). + Replace('{{currentversion}}', $windowsVersion) | ` + Out-File -FilePath "$configFile" -Encoding ascii RegisterContainerDService -} \ No newline at end of file +} diff --git a/parts/windowsparams.t b/parts/windowsparams.t index 258c8c5aad..831618759c 100644 --- a/parts/windowsparams.t +++ b/parts/windowsparams.t @@ -109,4 +109,18 @@ "description": "The version of Docker to be installed on Windows Nodes" }, "type": "string" + }, + "defaultContainerdRuntimeHandler": { + "defaultValue": "process", + "metadata": { + "description": "The containerd handler type (process isolated or hyperv)" + }, + "type": "string" + }, + "hypervRuntimeHandlers": { + "defaultValue": "", + "metadata": { + "description": "comma separated list of hyperv values" + }, + "type": "string" } diff --git a/pkg/api/const.go b/pkg/api/const.go index 773c031c96..391838139f 100644 --- a/pkg/api/const.go +++ b/pkg/api/const.go @@ -51,6 +51,8 @@ const ( KubernetesWindowsDockerVersion = "19.03.11" // KubernetesDefaultWindowsSku is the default SKU for Windows VMs in kubernetes KubernetesDefaultWindowsSku = "Datacenter-Core-1809-with-Containers-smalldisk" + // KubernetesDefaultWindowsRuntimeHandler is the default containerd handler for windows pods + KubernetesDefaultWindowsRuntimeHandler = "process" ) // validation values diff --git a/pkg/api/converterfromapi.go b/pkg/api/converterfromapi.go index a5c958db1e..00048fbf8e 100644 --- a/pkg/api/converterfromapi.go +++ b/pkg/api/converterfromapi.go @@ -217,6 +217,16 @@ func convertWindowsProfileToVLabs(api *WindowsProfile, vlabsProfile *vlabs.Windo vlabsProfile.EnableAHUB = to.BoolPtr(true) } vlabsProfile.AlwaysPullWindowsPauseImage = api.AlwaysPullWindowsPauseImage + if api.WindowsRuntimes != nil { + vlabsProfile.WindowsRuntimes = &vlabs.WindowsRuntimes{} + vlabsProfile.WindowsRuntimes.Default = api.WindowsRuntimes.Default + vlabsProfile.WindowsRuntimes.HypervRuntimes = []vlabs.RuntimeHandlers{} + for _, h := range api.WindowsRuntimes.HypervRuntimes { + handler := vlabs.RuntimeHandlers{} + handler.BuildNumber = h.BuildNumber + vlabsProfile.WindowsRuntimes.HypervRuntimes = append(vlabsProfile.WindowsRuntimes.HypervRuntimes, handler) + } + } } func convertOrchestratorProfileToVLabs(api *OrchestratorProfile, o *vlabs.OrchestratorProfile) { diff --git a/pkg/api/converterfromapi_test.go b/pkg/api/converterfromapi_test.go index 1ceb34fca7..a4222994b0 100644 --- a/pkg/api/converterfromapi_test.go +++ b/pkg/api/converterfromapi_test.go @@ -783,6 +783,12 @@ func TestConvertWindowsProfileToVlabs(t *testing.T) { WindowsSku: "2019-Datacenter-Core-smalldisk", WindowsDockerVersion: "18.09", EnableAHUB: to.BoolPtr(true), + WindowsRuntimes: &WindowsRuntimes{ + Default: "process", + HypervRuntimes: []RuntimeHandlers{ + {BuildNumber: "17763"}, + }, + }, }, expected: vlabs.WindowsProfile{ AdminUsername: "user", @@ -796,6 +802,12 @@ func TestConvertWindowsProfileToVlabs(t *testing.T) { WindowsDockerVersion: "18.09", Secrets: []vlabs.KeyVaultSecrets{}, EnableAHUB: to.BoolPtr(true), + WindowsRuntimes: &vlabs.WindowsRuntimes{ + Default: "process", + HypervRuntimes: []vlabs.RuntimeHandlers{ + {BuildNumber: "17763"}, + }, + }, }, }, { diff --git a/pkg/api/convertertoapi.go b/pkg/api/convertertoapi.go index 7c8989508a..b991d929bb 100644 --- a/pkg/api/convertertoapi.go +++ b/pkg/api/convertertoapi.go @@ -211,6 +211,16 @@ func convertVLabsWindowsProfile(vlabs *vlabs.WindowsProfile, api *WindowsProfile api.EnableAHUB = to.BoolPtr(true) } api.AlwaysPullWindowsPauseImage = vlabs.AlwaysPullWindowsPauseImage + if vlabs.WindowsRuntimes != nil { + api.WindowsRuntimes = &WindowsRuntimes{} + api.WindowsRuntimes.Default = vlabs.WindowsRuntimes.Default + api.WindowsRuntimes.HypervRuntimes = []RuntimeHandlers{} + for _, h := range vlabs.WindowsRuntimes.HypervRuntimes { + handler := RuntimeHandlers{} + handler.BuildNumber = h.BuildNumber + api.WindowsRuntimes.HypervRuntimes = append(api.WindowsRuntimes.HypervRuntimes, handler) + } + } } func convertVLabsOrchestratorProfile(vp *vlabs.Properties, api *OrchestratorProfile, isUpdate bool) error { diff --git a/pkg/api/convertertoapi_test.go b/pkg/api/convertertoapi_test.go index 28b7efc0f8..035ddc7112 100644 --- a/pkg/api/convertertoapi_test.go +++ b/pkg/api/convertertoapi_test.go @@ -921,6 +921,12 @@ func TestConvertVLabsWindowsProfile(t *testing.T) { WindowsSku: "2019-Datacenter-Core-smalldisk", WindowsDockerVersion: "18.09", EnableAHUB: to.BoolPtr(true), + WindowsRuntimes: &vlabs.WindowsRuntimes{ + Default: "process", + HypervRuntimes: []vlabs.RuntimeHandlers{ + {BuildNumber: "17763"}, + }, + }, }, expected: WindowsProfile{ AdminUsername: "user", @@ -934,6 +940,12 @@ func TestConvertVLabsWindowsProfile(t *testing.T) { WindowsDockerVersion: "18.09", Secrets: []KeyVaultSecrets{}, EnableAHUB: to.BoolPtr(true), + WindowsRuntimes: &WindowsRuntimes{ + Default: "process", + HypervRuntimes: []RuntimeHandlers{ + {BuildNumber: "17763"}, + }, + }, }, }, { diff --git a/pkg/api/types.go b/pkg/api/types.go index 4b18e00cd7..020857ce82 100644 --- a/pkg/api/types.go +++ b/pkg/api/types.go @@ -233,6 +233,18 @@ type WindowsProfile struct { EnableAHUB *bool `json:"enableAHUB,omitempty"` WindowsPauseImageURL string `json:"windowsPauseImageURL"` AlwaysPullWindowsPauseImage *bool `json:"alwaysPullWindowsPauseImage,omitempty"` + WindowsRuntimes *WindowsRuntimes `json:"windowsRuntimes,omitempty"` +} + +// WindowsRuntimes configures containerd runtimes that are available on the windows nodes +type WindowsRuntimes struct { + Default string `json:"default,omitempty"` + HypervRuntimes []RuntimeHandlers `json:"hypervRuntimes,omitempty"` +} + +// RuntimeHandlers configures the runtime settings in containerd +type RuntimeHandlers struct { + BuildNumber string `json:"buildNumber,omitempty"` } // ProvisioningState represents the current state of container service resource. @@ -1801,6 +1813,28 @@ func (w *WindowsProfile) GetWindowsDockerVersion() string { return KubernetesWindowsDockerVersion } +// GetWindowsDefaultRuntimeHandler get the default containerd runtime handler or return default value +func (w *WindowsProfile) GetWindowsDefaultRuntimeHandler() string { + if w.WindowsRuntimes != nil && w.WindowsRuntimes.Default != "" { + return w.WindowsRuntimes.Default + } + + return KubernetesDefaultWindowsRuntimeHandler +} + +// GetWindowsHypervRuntimeHandlers gets comma separated list of runtimehandler names +func (w *WindowsProfile) GetWindowsHypervRuntimeHandlers() string { + if w.WindowsRuntimes != nil && len(w.WindowsRuntimes.HypervRuntimes) > 0 { + handlernames := []string{} + for _, h := range w.WindowsRuntimes.HypervRuntimes { + handlernames = append(handlernames, h.BuildNumber) + } + return strings.Join(handlernames, ",") + } + + return "" +} + // GetWindowsSku gets the marketplace sku specified (such as Datacenter-Core-1809-with-Containers-smalldisk) or returns default value func (w *WindowsProfile) GetWindowsSku() string { if w.WindowsSku != "" { diff --git a/pkg/api/types_test.go b/pkg/api/types_test.go index dc48a34667..0a8dfe36e0 100644 --- a/pkg/api/types_test.go +++ b/pkg/api/types_test.go @@ -3587,6 +3587,16 @@ func TestWindowsProfile(t *testing.T) { t.Fatalf("Expected GetWindowsDockerVersion() to equal default KubernetesWindowsDockerVersion, got %s", dv) } + dh := w.GetWindowsDefaultRuntimeHandler() + if dh != KubernetesDefaultWindowsRuntimeHandler { + t.Fatalf("Expected GetWindowsDefaultRuntimeHandler() to equal default KubernetesWindowsDockerVersion, got %s", dh) + } + + rth := w.GetWindowsHypervRuntimeHandlers() + if rth != "" { + t.Fatalf("Expected GetWindowsHypervRuntimeHandlers() to equal default empty, got %s", rth) + } + windowsSku := w.GetWindowsSku() if windowsSku != KubernetesDefaultWindowsSku { t.Fatalf("Expected GetWindowsSku() to equal default KubernetesDefaultWindowsSku, got %s", windowsSku) @@ -3644,6 +3654,13 @@ func TestWindowsProfile(t *testing.T) { SSHEnabled: &trueVar, IsCredentialAutoGenerated: to.BoolPtr(false), EnableAHUB: to.BoolPtr(false), + WindowsRuntimes: &WindowsRuntimes{ + Default: "hyperv", + HypervRuntimes: []RuntimeHandlers{ + {BuildNumber: "17763"}, + {BuildNumber: "18362"}, + }, + }, } dv = w.GetWindowsDockerVersion() @@ -3670,6 +3687,16 @@ func TestWindowsProfile(t *testing.T) { if enableAHUB { t.Fatalf("Expected GetEnableAHUB() to equal default 'false', got %t", enableAHUB) } + + hypervruntimeHandler := w.GetWindowsDefaultRuntimeHandler() + if hypervruntimeHandler != "hyperv" { + t.Fatalf("Expected GetWindowsDefaultRuntimeHandler() to equal hyperv, got %s", hypervruntimeHandler) + } + + hypervruntimeHandlers := w.GetWindowsHypervRuntimeHandlers() + if hypervruntimeHandlers != "17763,18362" { + t.Fatalf("Expected GetWindowsDefaultRuntimeHandler() to equal '17763,18362', got %s", hypervruntimeHandler) + } } func TestWindowsProfileCustomOS(t *testing.T) { diff --git a/pkg/api/vlabs/types.go b/pkg/api/vlabs/types.go index 8b863b87e1..bda0f866de 100644 --- a/pkg/api/vlabs/types.go +++ b/pkg/api/vlabs/types.go @@ -187,6 +187,18 @@ type WindowsProfile struct { EnableAHUB *bool `json:"enableAHUB,omitempty"` WindowsPauseImageURL string `json:"windowsPauseImageURL"` AlwaysPullWindowsPauseImage *bool `json:"alwaysPullWindowsPauseImage,omitempty"` + WindowsRuntimes *WindowsRuntimes `json:"windowsRuntimes,omitempty"` +} + +// WindowsRuntimes configures containerd runtimes that are available on the windows nodes +type WindowsRuntimes struct { + Default string `json:"default,omitempty"` + HypervRuntimes []RuntimeHandlers `json:"hypervRuntimes,omitempty"` +} + +// RuntimeHandlers configures the runtime settings in containerd +type RuntimeHandlers struct { + BuildNumber string `json:"buildNumber,omitempty"` } // ProvisioningState represents the current state of container service resource. diff --git a/pkg/api/vlabs/validate.go b/pkg/api/vlabs/validate.go index e3b30545d6..abb335319f 100644 --- a/pkg/api/vlabs/validate.go +++ b/pkg/api/vlabs/validate.go @@ -1232,6 +1232,9 @@ func (a *Properties) validateWindowsProfile(isUpdate bool) error { if e := validateCsiProxyWindowsProperties(w, version); e != nil { return e } + if e := validateWindowsRuntimes(w.WindowsRuntimes); e != nil { + return e + } return nil } @@ -1243,6 +1246,33 @@ func validateCsiProxyWindowsProperties(w *WindowsProfile, k8sVersion string) err return nil } +func validateWindowsRuntimes(r *WindowsRuntimes) error { + if r == nil { + // can be blank defaults will be applied + return nil + } + + if r.Default != "process" && r.Default != "hyperv" { + return errors.New("Default runtime types are process or hyperv") + } + + if r.HypervRuntimes != nil { + handlersMap := make(map[string]bool) + for _, h := range r.HypervRuntimes { + if h.BuildNumber != "17763" && h.BuildNumber != "18362" && h.BuildNumber != "18363" && h.BuildNumber != "19041" { + return errors.New("Current hyper-v build id values supported are 17763, 18362, 18363, 19041") + } + + if _, ok := handlersMap[h.BuildNumber]; ok { + return errors.Errorf("Hyper-v RuntimeHandlers have duplicate runtime with build number '%s', Windows Runtimes must be unique", h.BuildNumber) + } + handlersMap[h.BuildNumber] = true + } + } + + return nil +} + func (a *AgentPoolProfile) validateOrchestratorSpecificProperties(orchestratorType string) error { // for Kubernetes, we don't support AgentPoolProfile.DNSPrefix @@ -1338,25 +1368,6 @@ func validateKeyVaultSecrets(secrets []KeyVaultSecrets, requireCertificateStore return nil } -// Validate ensures that the WindowsProfile is valid -func (w *WindowsProfile) Validate(orchestratorType string) error { - if w.WindowsImageSourceURL != "" { - if orchestratorType != DCOS && orchestratorType != Kubernetes { - return errors.New("Windows Custom Images are only supported if the Orchestrator Type is DCOS or Kubernetes") - } - } - if e := validate.Var(w.AdminUsername, "required"); e != nil { - return errors.New("WindowsProfile.AdminUsername is required, when agent pool specifies windows") - } - if e := validate.Var(w.AdminPassword, "required"); e != nil { - return errors.New("WindowsProfile.AdminPassword is required, when agent pool specifies windows") - } - if !validatePasswordComplexity(w.AdminUsername, w.AdminPassword) { - return errors.New("WindowsProfile.AdminPassword complexity not met. Windows password should contain 3 of the following categories - uppercase letters(A-Z), lowercase(a-z) letters, digits(0-9), special characters (~!@#$%^&*_-+=`|\\(){}[]:;<>,.?/')") - } - return validateKeyVaultSecrets(w.Secrets, true) -} - func validatePasswordComplexity(name string, password string) (out bool) { if strings.EqualFold(name, password) { diff --git a/pkg/api/vlabs/validate_test.go b/pkg/api/vlabs/validate_test.go index 07bd92e448..3bee6c41a5 100644 --- a/pkg/api/vlabs/validate_test.go +++ b/pkg/api/vlabs/validate_test.go @@ -1204,6 +1204,104 @@ func TestProperties_ValidateWindowsProfile(t *testing.T) { isUpdate: true, expectedError: nil, }, + { + name: "wrong runtime", + wp: &WindowsProfile{ + AdminUsername: "azure", + AdminPassword: "replacePassword1234$", + WindowsRuntimes: &WindowsRuntimes{ + Default: "something", + }, + }, + expectedError: errors.New("Default runtime types are process or hyperv"), + }, + { + name: "process runtime", + wp: &WindowsProfile{ + AdminUsername: "azure", + AdminPassword: "replacePassword1234$", + WindowsRuntimes: &WindowsRuntimes{ + Default: "process", + }, + }, + expectedError: nil, + }, + { + name: "hyperv runtime", + wp: &WindowsProfile{ + AdminUsername: "azure", + AdminPassword: "replacePassword1234$", + WindowsRuntimes: &WindowsRuntimes{ + Default: "process", + }, + }, + expectedError: nil, + }, + { + name: "invalid runtime handler name", + wp: &WindowsProfile{ + AdminUsername: "azure", + AdminPassword: "replacePassword1234$", + WindowsRuntimes: &WindowsRuntimes{ + Default: "process", + HypervRuntimes: []RuntimeHandlers{ + {BuildNumber: "something"}, + }, + }, + }, + expectedError: errors.New("Current hyper-v build id values supported are 17763, 18362, 18363, 19041"), + }, + { + name: "valid handler names", + wp: &WindowsProfile{ + AdminUsername: "azure", + AdminPassword: "replacePassword1234$", + WindowsRuntimes: &WindowsRuntimes{ + Default: "process", + HypervRuntimes: []RuntimeHandlers{ + {BuildNumber: "17763"}, + {BuildNumber: "18362"}, + {BuildNumber: "18363"}, + {BuildNumber: "19041"}, + }, + }, + }, + expectedError: nil, + }, + { + name: "some valid handlers some not", + wp: &WindowsProfile{ + AdminUsername: "azure", + AdminPassword: "replacePassword1234$", + WindowsRuntimes: &WindowsRuntimes{ + Default: "process", + HypervRuntimes: []RuntimeHandlers{ + {BuildNumber: "17763"}, + {BuildNumber: "18362"}, + {BuildNumber: "invalid"}, + {BuildNumber: "19041"}, + }, + }, + }, + expectedError: errors.New("Current hyper-v build id values supported are 17763, 18362, 18363, 19041"), + }, + { + name: "valid handlers must be unique", + wp: &WindowsProfile{ + AdminUsername: "azure", + AdminPassword: "replacePassword1234$", + WindowsRuntimes: &WindowsRuntimes{ + Default: "process", + HypervRuntimes: []RuntimeHandlers{ + {BuildNumber: "17763"}, + {BuildNumber: "18362"}, + {BuildNumber: "18363"}, + {BuildNumber: "17763"}, + }, + }, + }, + expectedError: errors.New("Hyper-v RuntimeHandlers have duplicate runtime with build number '17763', Windows Runtimes must be unique"), + }, } for _, test := range tests { @@ -3888,55 +3986,6 @@ func TestProperties_ValidateVNET(t *testing.T) { } } -func TestWindowsProfile_Validate(t *testing.T) { - tests := []struct { - name string - orchestratorType string - w *WindowsProfile - expectedMsg string - }{ - { - name: "unsupported orchestrator", - orchestratorType: "Mesos", - w: &WindowsProfile{ - WindowsImageSourceURL: "http://fakeWindowsImageSourceURL", - }, - expectedMsg: "Windows Custom Images are only supported if the Orchestrator Type is DCOS or Kubernetes", - }, - { - name: "empty adminUsername", - orchestratorType: "Kubernetes", - w: &WindowsProfile{ - WindowsImageSourceURL: "http://fakeWindowsImageSourceURL", - AdminUsername: "", - AdminPassword: "password", - }, - expectedMsg: "WindowsProfile.AdminUsername is required, when agent pool specifies windows", - }, - { - name: "empty password", - orchestratorType: "DCOS", - w: &WindowsProfile{ - WindowsImageSourceURL: "http://fakeWindowsImageSourceURL", - AdminUsername: "azure", - AdminPassword: "", - }, - expectedMsg: "WindowsProfile.AdminPassword is required, when agent pool specifies windows", - }, - } - - for _, test := range tests { - test := test - t.Run(test.name, func(t *testing.T) { - t.Parallel() - err := test.w.Validate(test.orchestratorType) - if err.Error() != test.expectedMsg { - t.Errorf("should error on unsupported orchType with msg : %s, but got : %s", test.expectedMsg, err.Error()) - } - }) - } -} - func TestValidate_VaultKeySecrets(t *testing.T) { tests := []struct { diff --git a/pkg/engine/const.go b/pkg/engine/const.go index 5cc6249f20..e305996d1e 100644 --- a/pkg/engine/const.go +++ b/pkg/engine/const.go @@ -87,6 +87,7 @@ const ( kubernetesWindowsAzureCniFunctionsPS1 = "k8s/windowsazurecnifunc.ps1" kubernetesWindowsHostsConfigAgentFunctionsPS1 = "k8s/windowshostsconfigagentfunc.ps1" kubernetesWindowsOpenSSHFunctionPS1 = "k8s/windowsinstallopensshfunc.ps1" + kubernetesWindowsHypervtemplatetoml = "k8s/containerdtemplate.toml" ) // cloud-init (i.e. ARM customData) source file references diff --git a/pkg/engine/params.go b/pkg/engine/params.go index 0e7178ac5a..d6d7227c8b 100644 --- a/pkg/engine/params.go +++ b/pkg/engine/params.go @@ -250,6 +250,9 @@ func getParameters(cs *api.ContainerService, generatorCode string, aksEngineVers addValue(parametersMap, fmt.Sprintf("windowsKeyVaultID%dCertificateStore%d", i, j), c.CertificateStore) } } + + addValue(parametersMap, "defaultContainerdRuntimeHandler", properties.WindowsProfile.GetWindowsDefaultRuntimeHandler()) + addValue(parametersMap, "hypervRuntimeHandlers", properties.WindowsProfile.GetWindowsHypervRuntimeHandlers()) } for _, extension := range properties.ExtensionProfiles { diff --git a/pkg/engine/template_generator.go b/pkg/engine/template_generator.go index 8ef270e12a..30414142d6 100644 --- a/pkg/engine/template_generator.go +++ b/pkg/engine/template_generator.go @@ -513,6 +513,7 @@ func getContainerServiceFuncMap(cs *api.ContainerService) template.FuncMap { kubernetesWindowsAzureCniFunctionsPS1, kubernetesWindowsHostsConfigAgentFunctionsPS1, kubernetesWindowsOpenSSHFunctionPS1, + kubernetesWindowsHypervtemplatetoml, } // Create a buffer, new zip diff --git a/pkg/engine/templates_generated.go b/pkg/engine/templates_generated.go index cd83981832..1aa3b2b1c4 100644 --- a/pkg/engine/templates_generated.go +++ b/pkg/engine/templates_generated.go @@ -110,6 +110,7 @@ // ../../parts/k8s/cloud-init/jumpboxcustomdata.yml // ../../parts/k8s/cloud-init/masternodecustomdata.yml // ../../parts/k8s/cloud-init/nodecustomdata.yml +// ../../parts/k8s/containerdtemplate.toml // ../../parts/k8s/kubeconfig.json // ../../parts/k8s/kubernetesparams.t // ../../parts/k8s/kuberneteswindowsfunctions.ps1 @@ -22386,6 +22387,89 @@ func k8sCloudInitNodecustomdataYml() (*asset, error) { return a, nil } +var _k8sContainerdtemplateToml = []byte(`root = "C:\\ProgramData\\containerd\\root" +state = "C:\\ProgramData\\containerd\\state" + +[grpc] + address = "\\\\.\\pipe\\containerd-containerd" + max_recv_message_size = 16777216 + max_send_message_size = 16777216 + +[ttrpc] + address = "" + +[debug] + address = "" + level = "debug" + +[metrics] + address = "" + grpc_histogram = false + +[cgroup] + path = "" + +[plugins] + [plugins.cri] + stream_server_address = "127.0.0.1" + stream_server_port = "0" + enable_selinux = false + sandbox_image = "{{pauseImage}}-windows-{{currentversion}}-amd64" + stats_collect_period = 10 + systemd_cgroup = false + enable_tls_streaming = false + max_container_log_line_size = 16384 + [plugins.cri.containerd] + snapshotter = "windows" + no_pivot = false + [plugins.cri.containerd.default_runtime] + runtime_type = "io.containerd.runhcs.v1" + [plugins.cri.containerd.default_runtime.options] + Debug = true + DebugType = 2 + SandboxImage = "{{pauseImage}}-windows-{{currentversion}}-amd64" + SandboxPlatform = "windows/amd64" + SandboxIsolation = {{sandboxIsolation}} + [plugins.cri.containerd.runtimes] + [plugins.cri.containerd.runtimes.runhcs-wcow-process] + runtime_type = "io.containerd.runhcs.v1" + [plugins.cri.containerd.runtimes.runhcs-wcow-process.options] + Debug = true + DebugType = 2 + SandboxImage = "{{pauseImage}}-windows-{{currentversion}}-amd64" + SandboxPlatform = "windows/amd64" +{{hypervisors}} + [plugins.cri.cni] + bin_dir = "{{cnibin}}" + conf_dir = "{{cniconf}}" + [plugins.cri.registry] + [plugins.cri.registry.mirrors] + [plugins.cri.registry.mirrors."docker.io"] + endpoint = ["https://registry-1.docker.io"] + [plugins.diff-service] + default = ["windows"] + [plugins.scheduler] + pause_threshold = 0.02 + deletion_threshold = 0 + mutation_threshold = 100 + schedule_delay = "0s" + startup_delay = "100ms"`) + +func k8sContainerdtemplateTomlBytes() ([]byte, error) { + return _k8sContainerdtemplateToml, nil +} + +func k8sContainerdtemplateToml() (*asset, error) { + bytes, err := k8sContainerdtemplateTomlBytes() + if err != nil { + return nil, err + } + + info := bindataFileInfo{name: "k8s/containerdtemplate.toml", size: 0, mode: os.FileMode(0), modTime: time.Unix(0, 0)} + a := &asset{bytes: bytes, info: info} + return a, nil +} + var _k8sKubeconfigJson = []byte(` { "apiVersion": "v1", "clusters": [ @@ -23357,6 +23441,8 @@ $global:DockerVersion = "{{WrapAsParameter "windowsDockerVersion"}}" ## ContainerD Usage $global:ContainerRuntime = "{{WrapAsParameter "containerRuntime"}}" +$global:DefaultContainerdRuntimeHandler = "{{WrapAsParameter "defaultContainerdRuntimeHandler"}}" +$global:HypervRuntimeHandlers = "{{WrapAsParameter "hypervRuntimeHandlers"}}" ## VM configuration passed by Azure $global:WindowsTelemetryGUID = "{{WrapAsParameter "windowsTelemetryGUID"}}" @@ -23461,6 +23547,7 @@ try # to the windows machine, and run the script manually to watch # the output. if ($true) { + Write-Log ".\CustomDataSetupScript.ps1 -MasterIP $MasterIP -KubeDnsServiceIp $KubeDnsServiceIp -MasterFQDNPrefix $MasterFQDNPrefix -Location $Location -AgentKey $AgentKey -AADClientId $AADClientId -AADClientSecret $AADClientSecret -NetworkAPIVersion $NetworkAPIVersion -TargetEnvironment $TargetEnvironment" Write-Log "Provisioning $global:DockerServiceName... with IP $MasterIP" $global:globalTimer = [System.Diagnostics.Stopwatch]::StartNew() @@ -24728,6 +24815,67 @@ function RegisterContainerDService { } } +function CreateHypervisorRuntime { + Param( + [Parameter(Mandatory = $true)][string] + $image, + [Parameter(Mandatory = $true)][string] + $version, + [Parameter(Mandatory = $true)][string] + $buildNumber + ) + + return @" + [plugins.cri.containerd.runtimes.runhcs-wcow-hypervisor-$buildnumber] + runtime_type = "io.containerd.runhcs.v1" + [plugins.cri.containerd.runtimes.runhcs-wcow-hypervisor-$buildnumber.options] + Debug = true + DebugType = 2 + SandboxImage = "$image-windows-$version-amd64" + SandboxPlatform = "windows/amd64" + SandboxIsolation = 1 +"@ +} + +function CreateHypervisorRuntimes { + Param( + [Parameter(Mandatory = $true)][string[]] + $builds, + [Parameter(Mandatory = $true)][string] + $image + ) + + Write-Host "Adding hyperv runtimes $builds" + $hypervRuntimes = "" + ForEach ($buildNumber in $builds) { + $windowsVersion = Select-Windows-Version -buildNumber $buildNumber + $runtime = createHypervisorRuntime -image $pauseImage -version $windowsVersion -buildNumber $buildNumber + if ($hypervRuntimes -eq "") { + $hypervRuntimes = $runtime + } + else { + $hypervRuntimes = $hypervRuntimes + "` + "`" + `r` + "`" + `n" + $runtime + } + } + + return $hypervRuntimes +} + +function Select-Windows-Version { + param ( + [Parameter()] + [string] + $buildNumber + ) + + switch ($buildNumber) { + "17763" { return "1809" } + "18362" { return "1903" } + "18363" { return "1909" } + "19041" { return "2004" } + Default { return "" } + } +} function Install-Containerd { Param( @@ -24746,133 +24894,64 @@ function Install-Containerd { } # TODO: check if containerd is already installed and is the same version before this. - $zipfile = [Io.path]::Combine($ENV:TEMP, "containerd.zip") - DownloadFileOverHttp -Url $ContainerdUrl -DestinationPath $zipfile - Expand-Archive -path $zipfile -DestinationPath $global:ContainerdInstallLocation -Force - del $zipfile + + # Extract the package + if ($ContainerdUrl.endswith(".zip")) { + $zipfile = [Io.path]::Combine($ENV:TEMP, "containerd.zip") + DownloadFileOverHttp -Url $ContainerdUrl -DestinationPath $zipfile + Expand-Archive -path $zipfile -DestinationPath $global:ContainerdInstallLocation -Force + del $zipfile + } + elseif ($ContainerdUrl.endswith(".tar.gz")) { + # upstream containerd package is a tar + $tarfile = [Io.path]::Combine($ENV:TEMP, "containerd.tar.gz") + DownloadFileOverHttp -Url $ContainerdUrl -DestinationPath $tarfile + mkdir -Force "C:\Program Files\containerd" + tar -xzf $tarfile -C $global:ContainerdInstallLocation + mv $global:ContainerdInstallLocation\bin\* $global:ContainerdInstallLocation\ + del $tarfile + del -Recurse -Force $global:ContainerdInstallLocation\bin + } + # get configuration options Add-SystemPathEntry $global:ContainerdInstallLocation - - # TODO: remove if the node comes up without this code - # $configDir = [Io.Path]::Combine($ENV:ProgramData, "containerd") - # if (-Not (Test-Path $configDir)) { - # mkdir $configDir - # } - - # TODO: call containerd.exe dump config, then modify instead of starting with hardcoded + $cdbinary = Join-Path $global:ContainerdInstallLocation containerd.exe $configFile = [Io.Path]::Combine($global:ContainerdInstallLocation, "config.toml") - $clusterConfig = ConvertFrom-Json ((Get-Content $global:KubeClusterConfigPath -ErrorAction Stop) | Out-String) $pauseImage = $clusterConfig.Cri.Images.Pause + $formatedbin = $(($CNIBinDir).Replace("\", "/")) + $formatedconf = $(($CNIConfDir).Replace("\", "/")) + $sandboxIsolation = 0 + $windowsVersion = (Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion").ReleaseId + $hypervRuntimes = "" + $hypervHandlers = $global:HypervRuntimeHandlers.split(",", [System.StringSplitOptions]::RemoveEmptyEntries) + + # configure + if ($global:DefaultContainerdRuntimeHandler -eq "hyperv") { + Write-Host "default runtime for containerd set to hyperv" + $sandboxIsolation = 1 + } - @" -version = 2 -root = "C:\\ProgramData\\containerd\\root" -state = "C:\\ProgramData\\containerd\\state" -plugin_dir = "" -disabled_plugins = [] -required_plugins = [] -oom_score = 0 - -[grpc] - address = "\\\\.\\pipe\\containerd-containerd" - tcp_address = "" - tcp_tls_cert = "" - tcp_tls_key = "" - uid = 0 - gid = 0 - max_recv_message_size = 16777216 - max_send_message_size = 16777216 - -[ttrpc] - address = "" - uid = 0 - gid = 0 - -[debug] - address = "" - uid = 0 - gid = 0 - level = "" - -[metrics] - address = "" - grpc_histogram = false - -[cgroup] - path = "" - -[timeouts] - "io.containerd.timeout.shim.cleanup" = "5s" - "io.containerd.timeout.shim.load" = "5s" - "io.containerd.timeout.shim.shutdown" = "3s" - "io.containerd.timeout.task.state" = "2s" + $template = Get-Content -Path "c:\AzureData\k8s\containerdtemplate.toml" + if ($sandboxIsolation -eq 0 -And $hypervHandlers.Count -eq 0) { + # remove the value hypervisor place holder + $template = $template | Select-String -Pattern 'hypervisors' -NotMatch | Out-String + } + else { + $hypervRuntimes = CreateHypervisorRuntimes -builds @($hypervHandlers) -image $pauseImage + } -[plugins] - [plugins."io.containerd.gc.v1.scheduler"] - pause_threshold = 0.02 - deletion_threshold = 0 - mutation_threshold = 100 - schedule_delay = "0s" - startup_delay = "100ms" - [plugins."io.containerd.grpc.v1.cri"] - disable_tcp_service = true - stream_server_address = "127.0.0.1" - stream_server_port = "0" - stream_idle_timeout = "4h0m0s" - enable_selinux = false - sandbox_image = "$pauseImage" - stats_collect_period = 10 - systemd_cgroup = false - enable_tls_streaming = false - max_container_log_line_size = 16384 - disable_cgroup = false - disable_apparmor = false - restrict_oom_score_adj = false - max_concurrent_downloads = 3 - disable_proc_mount = false - [plugins."io.containerd.grpc.v1.cri".containerd] - snapshotter = "windows" - default_runtime_name = "runhcs-wcow-process" - no_pivot = false - [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime] - runtime_type = "" - runtime_engine = "" - runtime_root = "" - privileged_without_host_devices = false - [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime] - runtime_type = "" - runtime_engine = "" - runtime_root = "" - privileged_without_host_devices = false - [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] - [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-process] - runtime_type = "io.containerd.runhcs.v1" - runtime_engine = "" - runtime_root = "" - privileged_without_host_devices = false - [plugins."io.containerd.grpc.v1.cri".cni] - bin_dir = "$(($CNIBinDir).Replace("\","//"))" - conf_dir = "$(($CNIConfDir).Replace("\","//"))" - max_conf_num = 1 - conf_template = "" - [plugins."io.containerd.grpc.v1.cri".registry] - [plugins."io.containerd.grpc.v1.cri".registry.mirrors] - [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] - endpoint = ["https://registry-1.docker.io"] - [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming] - tls_cert_file = "" - tls_key_file = "" - [plugins."io.containerd.metadata.v1.bolt"] - content_sharing_policy = "shared" - [plugins."io.containerd.runtime.v2.task"] - platforms = ["windows/amd64", "linux/amd64"] - [plugins."io.containerd.service.v1.diff-service"] - default = ["windows", "windows-lcow"] -"@ | Out-File -Encoding ascii $configFile + $template.Replace('{{sandboxIsolation}}', $sandboxIsolation). + Replace('{{pauseImage}}', $pauseImage). + Replace('{{hypervisors}}', $hypervRuntimes). + Replace('{{cnibin}}', $formatedbin). + Replace('{{cniconf}}', $formatedconf). + Replace('{{currentversion}}', $windowsVersion) | ` + "`" + ` + Out-File -FilePath "$configFile" -Encoding ascii RegisterContainerDService -}`) +} +`) func k8sWindowscontainerdfuncPs1Bytes() ([]byte, error) { return _k8sWindowscontainerdfuncPs1, nil @@ -28802,6 +28881,20 @@ var _windowsparamsT = []byte(` {{if IsKubernetes}} "description": "The version of Docker to be installed on Windows Nodes" }, "type": "string" + }, + "defaultContainerdRuntimeHandler": { + "defaultValue": "process", + "metadata": { + "description": "The containerd handler type (process isolated or hyperv)" + }, + "type": "string" + }, + "hypervRuntimeHandlers": { + "defaultValue": "", + "metadata": { + "description": "comma separated list of hyperv values" + }, + "type": "string" } `) @@ -28982,6 +29075,7 @@ var _bindata = map[string]func() (*asset, error){ "k8s/cloud-init/jumpboxcustomdata.yml": k8sCloudInitJumpboxcustomdataYml, "k8s/cloud-init/masternodecustomdata.yml": k8sCloudInitMasternodecustomdataYml, "k8s/cloud-init/nodecustomdata.yml": k8sCloudInitNodecustomdataYml, + "k8s/containerdtemplate.toml": k8sContainerdtemplateToml, "k8s/kubeconfig.json": k8sKubeconfigJson, "k8s/kubernetesparams.t": k8sKubernetesparamsT, "k8s/kuberneteswindowsfunctions.ps1": k8sKuberneteswindowsfunctionsPs1, @@ -29182,6 +29276,7 @@ var _bintree = &bintree{nil, map[string]*bintree{ "masternodecustomdata.yml": {k8sCloudInitMasternodecustomdataYml, map[string]*bintree{}}, "nodecustomdata.yml": {k8sCloudInitNodecustomdataYml, map[string]*bintree{}}, }}, + "containerdtemplate.toml": {k8sContainerdtemplateToml, map[string]*bintree{}}, "kubeconfig.json": {k8sKubeconfigJson, map[string]*bintree{}}, "kubernetesparams.t": {k8sKubernetesparamsT, map[string]*bintree{}}, "kuberneteswindowsfunctions.ps1": {k8sKuberneteswindowsfunctionsPs1, map[string]*bintree{}}, diff --git a/scripts/build-windows-containerd.sh b/scripts/build-windows-containerd.sh index c4d08fd6bd..18f38c60b5 100755 --- a/scripts/build-windows-containerd.sh +++ b/scripts/build-windows-containerd.sh @@ -31,41 +31,19 @@ cp bin/ctr.exe /output #cp bin/containerd.exe /output # missing CRI plugin, so build from containerd/cri cd \$GOPATH cd src/github.com/containerd -git clone https://github.com/containerd/cri.git +git clone https://github.com/jterry75/cri.git cd cri +git checkout windows_port git rev-parse HEAD > /output/cri-revision.txt make containerd cp _output/containerd.exe /output apt update apt install -y zip cd /output -zip windows-cri-containerd.zip *.exe *.txt +zip windows-cri-containerd.zip *.exe *.txt *.toml rm -f /output/*.exe rm -f /output/*.txt EOF chmod +x $OUTDIR/buildcri.sh -cat < $OUTDIR/buildcni.sh -set -e -x -o pipefail -export GOOS=windows -export GOARCH=amd64 -mkdir -p src/github.com/Microsoft -cd src/github.com/Microsoft -git clone https://github.com/Microsoft/windows-container-networking.git -cd windows-container-networking -git rev-parse HEAD > /output/cni-revision.txt -make all -mv out/*.exe /output -apt update -apt install -y zip -cd /output -zip windows-cni-containerd.zip *.exe *.txt -rm -f /output/*.exe -rm -f /output/*.txt -EOF -chmod +x $OUTDIR/buildcni.sh - - - docker run $DOCKERARGS -v $OUTDIR:/output golang:$GOTAG /bin/bash -c /output/buildcri.sh -docker run $DOCKERARGS -v $OUTDIR:/output golang:$GOTAG /bin/bash -c /output/buildcni.sh diff --git a/test/e2e/engine/template.go b/test/e2e/engine/template.go index 6c872bd960..80bc9dfcb8 100644 --- a/test/e2e/engine/template.go +++ b/test/e2e/engine/template.go @@ -230,10 +230,8 @@ func Build(cfg *config.Config, masterSubnetID string, agentSubnetIDs []string, i } if config.ContainerRuntime == "containerd" && - prop.OrchestratorProfile.KubernetesConfig.WindowsContainerdURL == "" && - prop.OrchestratorProfile.KubernetesConfig.WindowsSdnPluginURL == "" { - prop.OrchestratorProfile.KubernetesConfig.WindowsContainerdURL = "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cri-containerd.zip" - prop.OrchestratorProfile.KubernetesConfig.WindowsSdnPluginURL = "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cni-containerd.zip" + prop.OrchestratorProfile.KubernetesConfig.WindowsContainerdURL == "" { + prop.OrchestratorProfile.KubernetesConfig.WindowsContainerdURL = "https://github.com/containerd/containerd/releases/download/v1.4.0-rc.0/containerd-1.4.0-rc.0-windows-amd64.tar.gz" } if config.ContainerRuntime != "" { diff --git a/test/e2e/test_cluster_configs/containerd.json b/test/e2e/test_cluster_configs/containerd.json index 5491211c8a..85593b7cbb 100644 --- a/test/e2e/test_cluster_configs/containerd.json +++ b/test/e2e/test_cluster_configs/containerd.json @@ -8,7 +8,7 @@ "kubernetesConfig": { "networkPlugin": "azure", "containerRuntime": "containerd", - "windowsContainerdURL": "https://github.com/marosset/windows-cri-containerd/releases/download/nightly/windows-cri-containerd.zip" + "windowsContainerdURL": "https://github.com/containerd/containerd/releases/download/v1.4.0-rc.0/containerd-1.4.0-rc.0-windows-amd64.tar.gz" } }, "masterProfile": {