Support for Custom CA Certificates

[TimJongerius]

Is there an existing issue for this?

• I have searched the existing issues

Description

Add an option to upload additional ca certificates during cluster creation like it is already possible using the Cli (https://learn.microsoft.com/en-us/azure/aks/custom-certificate-authority)

New or Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Potential Terraform Configuration

No response

References

No response

[zioproto]

Hello,
this is a preview feature, currently tracked with Azure/AKS#2259

As soon as the feature is GA and the Terraform Provider supports the feature, we can start the implementation in the module.

[TimJongerius]

@zioproto Okay thanks for the update. I can see that it's possible to activate the custom ca daemonset for additional nodepools. Is there a reason why I can't specify it for the default_node_pool?

[TimJongerius]

@zioproto it seems to be available now
https://registry.terraform.io/providers/hashicorp/azurerm/3.64.0/docs/resources/kubernetes_cluster#custom_ca_trust_certificates_base64

When would it be possible to update the module?

[lonegunmanb]

Hi @TimJongerius, according to this post, the feature hasn't an ETA for GA yet, are you sure that this feature is GA already?

[TimJongerius]

Hi @lonegunmanb,

according to this link
https://learn.microsoft.com/en-us/azure/aks/custom-certificate-authority
it seems to be still in preview.

Also for the azure cli it's only available after enabling aks-preview.

However the terraform provider started to support it by adding the
custom_ca_trust_certificates_base64 property from 3.63 upwards.
https://github.com/hashicorp/terraform-provider-azurerm/blob/v3.63.0/CHANGELOG.md

Before, to work around this limitation without the need to deployment a very complex daemonset I used a terraform provisioner to upload the certificate with the cli + aks preview after the aks deployment. Because the custom_ca_trust_certificates_base64 property wasn't known to the terraform provider it didn't change that property when I redeployed the module, hence the nodepools didn't get drained.

With 3.63 this behavior changed since the provider is know removing this property and I have no way to supply it with the aks module. The only way to avoid this is to fixate the provider on a version < 3.63.0.

Why do we have to wait for GA if the azurerm provider has already started to support it?

[lonegunmanb]

Thanks for asking @TimJongerius, a preview feature might be changed or even removed totally at any time, so when the provider introduces a preview feature it also introduces the corresponding risk, it happened before and it would happen again. This Aks module is one of our "verified" modules. We'd like to keep these verified modules as stable as possible, so we decide that we should release the major version upgrade which contains breaking changes every six months.

I fully understand the reason you want this feature in this module, and thanks for using our modules. We don't have a best practice on balance between stability and capability, do you have any suggestions?

[HouseDamage]

Any idea when this feature will go Globally Available?

Have been tracking this for a long time but unable to find out when it's planned for GA release.

Thanks!

[zioproto]

Any idea when this feature will go Globally Available?

Have been tracking this for a long time but unable to find out when it's planned for GA release.

Thanks!

The correct place to ask this question is Azure/AKS#2259

[HouseDamage]

@zioproto - I know, but the commenting is closed! :(

[asifkd012020]

When is GA planned for this feature?

[syepes]

Looks like this is getting deprecated, anyone know what will be the replacement solution?

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/4.0-upgrade-guide#aks-migration-to-stable-api

[TimJongerius]

The deprecation of this feature is unacceptable. Many people rely on connecting AKS to on-premises registries. In the recent past, it worked—perhaps not perfectly, but it worked. If this feature is deprecated, please explain how we are supposed to connect private, on-premise registries (e.g. Nexus), which almost always use private CAs, to AKS. We need to pull images from these sources for compliance and other critical reasons.

We adopted this feature with the expectation that it would eventually become generally available. If it is removed, it will severely disrupt the delivery pipelines of many of my clients. An alternative solution must be provided. Simply stating that it was a preview feature with no support is not sufficient. This functionality is essential.

Why even is this getting deprecated? It doesn't seem to be that hard of a thing to implement.