Fix kubeletConfig for Windows agent nodes #3753
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: PatrickLang If they are not already assigned, you can assign the PR to them by writing The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ghost
assigned 
|
Hmm, looks like I broke |
Codecov Report
@@ Coverage Diff @@
## master #3753 +/- ##
=========================================
+ Coverage 55.48% 55.6% +0.11%
=========================================
Files 108 108
Lines 16140 16160 +20
=========================================
+ Hits 8955 8985 +30
+ Misses 6420 6408 -12
- Partials 765 767 +2 |
|
@CecileRobertMichon or @jackfrancis - can you review tomorrow? |
|
Resulting command line before: c:\k\kubelet.exe
--allow-privileged=true
--azure-container-registry-config=c:\k\azure.json
--cgroups-per-qos=false
--cloud-config=c:\k\azure.json
--cloud-provider=azure
--cluster-dns=$global:KubeDnsServiceIp
--cluster-domain=cluster.local
--cni-bin-dir=c:\k\azurecni\bin
--cni-conf-dir=c:\k\azurecni\netconf
--enable-debugging-handlers
--enforce-node-allocatable=""
--hairpin-mode=promiscuous-bridge
--hostname-override=$env:computername
--image-pull-progress-deadline=20m
--kubeconfig=c:\k\config
--network-plugin=cni
--pod-infra-container-image=kubletwin/pause
--resolv-conf=""
--runtime-request-timeout=10m
--v=2
--volume-plugin-dir=$global:VolumePluginDir After (with extra kubelet params passed): c:\k\kubelet.exe
--address=0.0.0.0
--allow-privileged=true
--anonymous-auth=false
--authorization-mode=Webhook
--azure-container-registry-config=c:\k\azure.json
--cadvisor-port=0
--cgroups-per-qos=false
--client-ca-file=c:\k\ca.crt
--cloud-config=c:\k\azure.json
--cloud-provider=azure
--cluster-dns=$global:KubeDnsServiceIp
--cluster-dns=10.0.0.10
--cluster-domain=cluster.local
--cni-bin-dir=c:\k\azurecni\bin
--cni-conf-dir=c:\k\azurecni\netconf
--enforce-node-allocatable=""
--event-qps=0
--eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%
--feature-gates=HyperVContainer=true,PodPriority=true
--hairpin-mode=promiscuous-bridge
--hostname-override=$global:AzureHostname
--image-gc-high-threshold=85
--image-gc-low-threshold=80
--image-pull-progress-deadline=20m
--keep-terminated-pod-volumes=false
--kubeconfig=c:\k\config
--max-pods=30
--network-plugin=cni
--network-plugin=cni
--node-labels=$global:KubeletNodeLabels
--node-status-update-frequency=10s
--non-masquerade-cidr=10.240.0.0/12
--pod-infra-container-image=kubletwin/pause
--pod-max-pids=100
--resolv-conf=""
--volume-plugin-dir=$global:VolumePluginDir |
| staticWindowsKubeletConfig["--azure-container-registry-config"] = "c:\\k\\azure.json" | ||
| staticWindowsKubeletConfig["--pod-infra-container-image"] = "kubletwin/pause" | ||
| staticWindowsKubeletConfig["--kubeconfig"] = "c:\\k\\config" | ||
| staticWindowsKubeletConfig["--cloud-config"] = "c:\\k\\azure.json" | ||
| staticWindowsKubeletConfig["--cgroups-per-qos"] = "false" | ||
| staticWindowsKubeletConfig["--enforce-node-allocatable"] = "\"\"" | ||
| staticWindowsKubeletConfig["--enforce-node-allocatable"] = "\"\"\"\"" |
There was a problem hiding this comment.
It's like an escape room. You escape one just to find you must escape another.
| @@ -136,6 +144,11 @@ func setKubeletConfig(cs *api.ContainerService) { | |||
| } | |||
| setMissingKubeletValues(profile.KubernetesConfig, o.KubernetesConfig.KubeletConfig) | |||
|
|
|||
| if profile.OSType == "Windows" { | |||
| // Remove Linux-specific values | |||
| delete(profile.KubernetesConfig.KubeletConfig, "--pod-manifest-path") | |||
There was a problem hiding this comment.
Ha. I wonder if we should do this for Linux as well. Instead of this recent change that @tariq1890 made:
|
rekicked unit test golang /lgtm |
|
glad I did that last check. I should fix that one because that will break private vnet support later |
|
/hold |
|
/hold cancel |
What this PR does / why we need it
Windows kubelet configs were fixed, and could not be overridden in the acs-engine apimodel. This meant that optional feature gates couldn't be enabled for Windows nodes.
Which issue this PR fixes: fixes #2627, #3266
Special notes for your reviewer:
Example apimodel.json:
{ "apiVersion": "vlabs", "properties": { "orchestratorProfile": { "orchestratorType": "Kubernetes", "orchestratorVersion": "1.10.6", "kubernetesConfig": { "apiServerConfig" : { "--feature-gates": "HyperVContainer=true" }, "kubeletConfig" : { "--feature-gates": "HyperVContainer=true" } } }, "masterProfile": { "count": 1, "dnsPrefix": "plang0607a", "vmSize": "Standard_D2_v3" }, "agentPoolProfiles": [ { "name": "windowspool", "count": 2, "vmSize": "Standard_D2_v3", "availabilityProfile": "AvailabilitySet", "osType": "Windows", "osDiskSizeGB": 127, "extensions": [ { "name": "winrm" } ] }, { "name": "linuxpool", "count": 2, "vmSize": "Standard_D2_v2", "availabilityProfile": "AvailabilitySet", "osType": "Linux" } ], "windowsProfile": { "adminUsername": "", "adminPassword": "", "windowsPublisher": "MicrosoftWindowsServer", "windowsOffer": "WindowsServerSemiAnnual", "windowsSku": "Datacenter-Core-1803-with-Containers-smalldisk" }, "extensionProfiles": [ { "name": "winrm", "version": "v1" } ], "linuxProfile": { "adminUsername": "", "ssh": { "publicKeys": [ { "keyData": "" } ] } }, "servicePrincipalProfile": { "clientId": "", "secret": "" } } }Once the cluster is deployed - I can run Windows Server version 1709 and 1803 pods successfully. 2016 is crashing for an unrelated issue that's a new bug.
I deployed these samples
Release note: