AKS : Advanced Networking / Calico Network Policy

[thatInfrastructureGuy]

This PR aims to add Calico Network policy to AKS as described in MS Docs.

Added network_policy to kubernetes_cluster resource.

PR Inspiration: #1479

[katbyte]

Hi @thatInfrastructureGuy,

Thank you for this PR! i've left some mostly minor comments inline but overall this is looking pretty good. Once those few issues are addressed and tests pass I think this should be good to merge 🙂

[katbyte]

Hi @thatInfrastructureGuy,

Thank you for the changes, this LGTM now. However the tests are failing:

------- Stdout: -------
=== RUN   TestAccAzureRMKubernetesCluster_advancedNetworkingAzureCalicoPolicy
=== PAUSE TestAccAzureRMKubernetesCluster_advancedNetworkingAzureCalicoPolicy
=== CONT  TestAccAzureRMKubernetesCluster_advancedNetworkingAzureCalicoPolicy
--- FAIL: TestAccAzureRMKubernetesCluster_advancedNetworkingAzureCalicoPolicy (218.11s)
    testing.go:538: Step 0 error: Error applying: 1 error occurred:
        	* azurerm_kubernetes_cluster.test: 1 error occurred:
        	* azurerm_kubernetes_cluster.test: Error creating/updating Managed Kubernetes Cluster "acctestaks190305190900741436" (Resource Group "acctestRG-190305190900741436"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="BadRequest" Message="Parameter networkProfile.networkPolicy is not allowed."

Are there any steps we need to enable networkPolicy on our subscription?

Also getting many of these:

* azurerm_virtual_network.test: 1 error occurred:
        	* azurerm_virtual_network.test: Error waiting for completion of Virtual Network "acctestvirtnet190305190900434025" (Resource Group "acctestRG-190305190900434025"): Code="InternalServerError" Message="An error occurred." Details=[]

[thatInfrastructureGuy]

@katbyte Thanks for reverting back. Yes. To enable network policy in your subscription please follow:

# To create an AKS with network policy, first enable a feature flag on your subscription. To register the EnableNetworkPolicy feature flag, use the az feature register command as shown in the following example:
az feature register --name EnableNetworkPolicy --namespace Microsoft.ContainerService

# It takes a few minutes for the status to show Registered. You can check on the registration status using the az feature list command:
az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/EnableNetworkPolicy')].{Name:name,State:properties.state}"

# When ready, refresh the registration of the Microsoft.ContainerService resource provider using the az provider register command:
az provider register --namespace Microsoft.ContainerService

Source : https://docs.microsoft.com/en-us/azure/aks/use-network-policies#before-you-begin

[thatInfrastructureGuy]

Also getting many of these:

* azurerm_virtual_network.test: 1 error occurred:
        	* azurerm_virtual_network.test: Error waiting for completion of Virtual Network "acctestvirtnet190305190900434025" (Resource Group "acctestRG-190305190900434025"): Code="InternalServerError" Message="An error occurred." Details=[]

I believe this might be related to above comment as VNET creation is not getting completed because of "unknown parameter" networkProfile.networkPolicy.

[thatInfrastructureGuy]

@katbyte Is there anything I can assist with? Can you please guide me for the documentation of labels. Eg: What does waiting-response stand for?

Thanks for helping me out!

[jfcoz]

@katbyte , is there something more to change ?

[RichardFowles89]

Hi guys, do you have any idea when this will be available for general release? We want to implement it at work.

Thanks!

[dannydombrowski]

Hey @katbyte it has been 15 days since your last activity on this issue. Could you please let us know if there is anything else left before merging this issue?

[katbyte]

Sorry for the delay here @dannydombrowski, @RichardFowles89, @jfcoz & @thatInfrastructureGuy our attention was elsewhere.

I have verified that the tests now pass for us and this will get into v1.24 🙂

[RichardFowles89]

That's great - thank you! I will keep my eye out.

[ghost]

This has been released in version 1.24.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
	version = "~> 1.24.0"
}
# ... other configuration ...

[RichardFowles89]

Great! Thank you for letting me know

[soggychipsnz]

Has anyone had success with provisioning a network-policy enabled AKS cluster with the new provider?

When attempting to provision an AKS cluster with this enabled, my deployment eventually fails with:
azurerm_kubernetes_cluster.aks_cluster: Error waiting for completion of Managed Kubernetes Cluster "REMOVED" (Resource Group REMOVED"): Code="NodesNotReady" Message=""

The same state deploys fine without the network_policy enabled.
RBAC and the Azure network plugin are set, also using 1.24.0 of the azurerm provider.
The network_profile network policy is being set to the value of "calico"
If I diff the exported ARM template of the failed deployment with one that has been successfully provisioned via ARM template, nothing stands out as incorrect.

Any assistance on this one or confirmation that it is working for others would be appreciated.

[thatInfrastructureGuy]

@soggychipsnz I am running into same issue. I redeployed (not restarted) the VMs and did a terraform apply again to fix it.

[soggychipsnz]

Thanks for confirming, I was wondering if I was missing something obvious.
How does one redeploy individual nodes from a managed container service? (assuming the nodes are the VMs you are referring to)

I have opened a case with MS to check out the failed instance as well just incase they have anything to add.

[thatInfrastructureGuy]

Just navigate to MC_${RESOURCE_GROUP}_${CLUSTER_NAME} resource group. On each VM; click the redeploy button available in the portal.

[soggychipsnz]

Cheers for the feedback. I'll give that a go after I get closure from my case I have open with MS as they might be able to provide more understanding as to why this issue is happening.

[thatInfrastructureGuy]

@soggychipsnz MicrosoftDocs/azure-docs#28567

Seems like azure backend issue which is currently being fixed. No ETA.

[feiskyer]

Yep, this is bug for v1.12.x. versions below that are still working

[thatInfrastructureGuy]

@feiskyer Thanks for the update. Is there a recommended path forward for users? Eg:

1. Using AKS 1.12 with azure-npm // OR
2. AKS <= 1.11 with Calico.

Which network plugin is likely to graduate to GA?

[feiskyer]

@thatInfrastructureGuy yep, both of those are working. it's up to users to decide which one is preferred.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!