Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

Enabling Azure CNI for Windows #2174

Merged
merged 12 commits into from
Feb 7, 2018
Merged

Enabling Azure CNI for Windows #2174

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
91a5256
enabling azure cni
Jan 29, 2018
2e7134a
delete overwrite
Jan 31, 2018
4bf1f69
address comments
Feb 1, 2018
87f591e
address comments
Feb 1, 2018
dc0b3f5
fix kubeStartStr
Feb 2, 2018
a7a3e81
fix kubeStartStr
Feb 2, 2018
bbcdae0
remove misc files
Feb 2, 2018
5f0a631
squash commits for kubeStartStr
saiyan86 Feb 4, 2018
17ca28f
passed final test
Feb 5, 2018
30e4c63
rebase cleanup
jackfrancis Feb 6, 2018
6c739ad
setting Azure CNI for vlabs only
jackfrancis Feb 6, 2018
db18611
default back to kubenet
jackfrancis Feb 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added debug
View file
Open in desktop
Binary file not shown.
126 changes: 104 additions & 22 deletions parts/k8s/kuberneteswindowssetup.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,20 @@ $global:UseInstanceMetadata = "{{WrapAsVariable "useInstanceMetadata"}}"
$global:CNIPath = [Io.path]::Combine("$global:KubeDir", "cni")
$global:NetworkMode = "L2Bridge"
$global:CNIConfig = [Io.path]::Combine($global:CNIPath, "config", "`$global:NetworkMode.conf")
$global:CNIConfigPath = [Io.path]::Combine("$global:CNIPath", "config")
$global:WindowsCNIKubeletOptions = " --network-plugin=cni --cni-bin-dir=$global:CNIPath --cni-conf-dir=$global:CNIConfigPath"
$global:HNSModule = [Io.path]::Combine("$global:KubeDir", "hns.psm1")

$global:VolumePluginDir = [Io.path]::Combine("$global:KubeDir", "volumeplugins")
#azure cni
$global:NetworkPolicy = "{{WrapAsVariable "networkPolicy"}}"
$global:VNetCNIPluginsURL = "{{WrapAsVariable "vnetCniWindowsPluginsURL"}}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is vnetCniWindowsPluginsURL defined?

Copy link
Contributor Author

@saiyan86 saiyan86 Jan 31, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In parts/k8s/kubernetesparams.t line 535


$global:AzureCNIDir = [Io.path]::Combine("$global:KubeDir", "azurecni")
$global:AzureCNIBinDir = [Io.path]::Combine("$global:AzureCNIDir", "bin")
$global:AzureCNIConfDir = [Io.path]::Combine("$global:AzureCNIDir", "netconf")
$global:AzureCNIKubeletOptions = " --network-plugin=cni --cni-bin-dir=$global:AzureCNIBinDir --cni-conf-dir=$global:AzureCNIConfDir"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nice. Can you do me a favor to add a WindowsCNIKubeletOptions for windows cni option to make it more readable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

$global:AzureCNIEnabled = $false

filter Timestamp {"$(Get-Date -Format o): $_"}

Expand Down Expand Up @@ -168,6 +179,56 @@ New-InfraContainer()
docker build -t kubletwin/pause .
}

function
Set-VnetPluginMode($mode)
{
# Sets Azure VNET CNI plugin operational mode.
$fileName = [Io.path]::Combine("$global:AzureCNIConfDir", "10-azure.conflist")
(Get-Content $fileName) | %{$_ -replace "`"mode`":.*", "`"mode`": `"$mode`","} | Out-File -encoding ASCII -filepath $fileName
}

function
Install-VnetPlugins()
{
# Create CNI directories.
mkdir $global:AzureCNIBinDir
mkdir $global:AzureCNIConfDir

# Download Azure VNET CNI plugins.
# Mirror from https://github.com/Azure/azure-container-networking/releases
$zipfile = [Io.path]::Combine("$global:AzureCNIDir", "azure-vnet.zip")
Invoke-WebRequest -Uri $global:VNetCNIPluginsURL -OutFile $zipfile
Expand-Archive -path $zipfile -DestinationPath $global:AzureCNIBinDir
del $zipfile

# Windows does not need a separate CNI loopback plugin because the Windows
# kernel automatically creates a loopback interface for each network namespace.
# Copy CNI network config file and set bridge mode.
move $global:AzureCNIBinDir/*.conflist $global:AzureCNIConfDir

# Enable CNI in kubelet.
$global:AzureCNIEnabled = $true
}

function
Set-AzureNetworkPolicy()
{
# Azure VNET network policy requires tunnel (hairpin) mode because policy is enforced in the host.
Set-VnetPluginMode "tunnel"
}

function
Set-NetworkConfig
{
Write-Log "Configuring networking with NetworkPolicy:$global:NetworkPolicy"

# Configure network policy.
if ($global:NetworkPolicy -eq "azure") {
Install-VnetPlugins
Set-AzureNetworkPolicy
}
}

function
Write-KubernetesStartFiles($podCIDR)
{
Expand All @@ -184,12 +245,15 @@ c:\k\kubelet.exe --hostname-override=`$global:AzureHostname --pod-infra-containe
$KubeletCommandLine += " --api-servers=https://`${global:MasterIP}:443"
}

# network plugin config
$KubeletCommandLine += " --network-plugin=cni --cni-bin-dir=`$global:CNIPath --cni-conf-dir `$global:CNIPath\config"

# more time is needed to pull windows server images
$KubeletCommandLine += " --image-pull-progress-deadline=20m --cgroups-per-qos=false --enforce-node-allocatable=`"`""
$KubeletCommandLine += " --volume-plugin-dir=`$global:VolumePluginDir"
# Configure kubelet to use CNI plugins if enabled.
if ($global:AzureCNIEnabled) {
$KubeletCommandLine += $global:AzureCNIKubeletOptions
} else {
$KubeletCommandLine += $global:WindowsCNIKubeletOptions
}

$KubeletArgListStr = "`"" + ($KubeletArgList -join "`",`"") + "`""

Expand All @@ -208,7 +272,19 @@ c:\k\kubelet.exe --hostname-override=`$global:AzureHostname --pod-infra-containe
`$global:CNIConfig = "$global:CNIConfig"
`$global:HNSModule = "$global:HNSModule"
`$global:VolumePluginDir = "$global:VolumePluginDir"
`$global:NetworkPolicy="$global:NetworkPolicy"

"@

if ($global:NetworkPolicy -eq "azure") {
$kubeStartStr += @"
Write-Host "NetworkPolicy azure, starting kubelet."
$KubeletCommandLine

"@
} else {
$kubeStartStr += @"

function
Get-DefaultGateway(`$CIDR)
{
Expand All @@ -233,29 +309,29 @@ Update-CNIConfig(`$podCIDR, `$masterSubnetGW)
{
`$jsonSampleConfig =
"{
""cniVersion"": ""0.2.0"",
""name"": ""<NetworkMode>"",
""type"": ""wincni.exe"",
""master"": ""Ethernet"",
""capabilities"": { ""portMappings"": true },
""ipam"": {
""environment"": ""azure"",
""subnet"":""<PODCIDR>"",
""routes"": [{
""cniVersion"": ""0.2.0"",
""name"": ""<NetworkMode>"",
""type"": ""wincni.exe"",
""master"": ""Ethernet"",
""capabilities"": { ""portMappings"": true },
""ipam"": {
""environment"": ""azure"",
""subnet"":""<PODCIDR>"",
""routes"": [{
""GW"":""<PODGW>""
}]
},
""dns"" : {
}]
},
""dns"" : {
""Nameservers"" : [ ""<NameServers>"" ]
},
""AdditionalArgs"" : [
},
""AdditionalArgs"" : [
{
""Name"" : ""EndpointPolicy"", ""Value"" : { ""Type"" : ""OutBoundNAT"", ""ExceptionList"": [ ""<ClusterCIDR>"", ""<MgmtSubnet>"" ] }
""Name"" : ""EndpointPolicy"", ""Value"" : { ""Type"" : ""OutBoundNAT"", ""ExceptionList"": [ ""<ClusterCIDR>"", ""<MgmtSubnet>"" ] }
},
{
""Name"" : ""EndpointPolicy"", ""Value"" : { ""Type"" : ""ROUTE"", ""DestinationPrefix"": ""<ServiceCIDR>"", ""NeedEncap"" : true }
""Name"" : ""EndpointPolicy"", ""Value"" : { ""Type"" : ""ROUTE"", ""DestinationPrefix"": ""<ServiceCIDR>"", ""NeedEncap"" : true }
}
]
]
}"

`$configJson = ConvertFrom-Json `$jsonSampleConfig
Expand Down Expand Up @@ -329,8 +405,11 @@ try
catch
{
Write-Error `$_
}
}

"@
}

$kubeStartStr | Out-File -encoding ASCII -filepath $global:KubeletStartFile

$kubeProxyStartStr = @"
Expand Down Expand Up @@ -430,6 +509,9 @@ try
Write-Log "Create the Pause Container kubletwin/pause"
New-InfraContainer

Write-Log "Configure networking"
Set-NetworkConfig

Write-Log "write kubelet startfile with pod CIDR of $podCIDR"
Write-KubernetesStartFiles $podCIDR

Expand All @@ -450,4 +532,4 @@ try
catch
{
Write-Error $_
}
}
2 changes: 2 additions & 0 deletions pkg/acsengine/const.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ const (
DefaultInternalLbStaticIPOffset = 10
// NetworkPolicyNone is the string expression for no network policy
NetworkPolicyNone = "none"
// NetworkPolicyAzure is the string expression for Azure CNI network policy
NetworkPolicyAzure = "azure"
// NetworkPluginKubenet is the string expression for kubenet network plugin
NetworkPluginKubenet = "kubenet"
// DefaultNetworkPolicy defines the network policy to use by default
Expand Down
2 changes: 0 additions & 2 deletions pkg/acsengine/defaults-kubelet.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,6 @@ func setKubeletConfig(cs *api.ContainerService) {
for key, val := range staticLinuxKubeletConfig {
staticWindowsKubeletConfig[key] = val
}
// Windows kubelet config overrides
staticWindowsKubeletConfig["--network-plugin"] = NetworkPluginKubenet

// Default Kubelet config
defaultKubeletConfig := map[string]string{
Expand Down
6 changes: 4 additions & 2 deletions pkg/acsengine/defaults.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ const (
// AzureCniPluginVer specifies version of Azure CNI plugin, which has been mirrored from
// https://github.com/Azure/azure-container-networking/releases/download/${AZURE_PLUGIN_VER}/azure-vnet-cni-linux-amd64-${AZURE_PLUGIN_VER}.tgz
// to https://acs-mirror.azureedge.net/cni/
AzureCniPluginVer = "v1.0.1"
AzureCniPluginVer = "v1.0.2"
)

var (
Expand Down Expand Up @@ -313,7 +313,9 @@ func setOrchestratorDefaults(cs *api.ContainerService) {
o.KubernetesConfig.EtcdVersion = DefaultEtcdVersion
}
if a.HasWindows() {
o.KubernetesConfig.NetworkPolicy = DefaultNetworkPolicyWindows
if o.KubernetesConfig.NetworkPolicy == "" {
o.KubernetesConfig.NetworkPolicy = DefaultNetworkPolicyWindows
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does Windows need to default to "none"? Originally it was default to "none" because Windows doesn't support azure cni.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in const.go by setting DefaultNetworkPolicyWindows to "azure".

}
} else {
if o.KubernetesConfig.NetworkPolicy == "" {
o.KubernetesConfig.NetworkPolicy = DefaultNetworkPolicy
Expand Down
2 changes: 1 addition & 1 deletion pkg/api/vlabs/validate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -699,7 +699,7 @@ func (a *Properties) validateNetworkPolicy() error {
}

// Temporary safety check, to be removed when Windows support is added.
if (networkPolicy == "calico" || networkPolicy == "azure") && a.HasWindows() {
if (networkPolicy == "calico") && a.HasWindows() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: no need to have parenthesis here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is parenthesis around (networkPolicy == "calico") removed?

return fmt.Errorf("networkPolicy '%s' is not supporting windows agents", networkPolicy)
}

Expand Down