Annotations in the helm_release not working (nginx-ingress in-specific)

[sunilnagavelli]

Terraform, Provider, Kubernetes and Helm Versions

Terraform version: 1.4.0
Provider version:  ~>2.9.0
Kubernetes version: 1.26.3

Affected Resource(s)

• helm_release nginx-ingress-controller
• helm_repository https://charts.bitnam.com/bitnami

Terraform Configuration Files

#Create Nginx Ingress controller
resource "helm_release" "nginx_ingress" {
  name       = "nginx-ingress"
  namespace  = "nginx-ingress"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "nginx-ingress-controller"

  set {
    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal"
    value = true
  }
  
  set {
    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal-subnet"
    value = var.snet_k8s_in_dg_aha
  }

  set {
    name  = "controller.service.loadBalancerIP"
    value = azurerm_lb.nginx_ingress_lb.frontend_ip_configuration[0].private_ip_address
  }

  # values = [
  #   file("./manifests/ingress-controller/ingress-nginx-values.yaml")
  # ]

  create_namespace = true
}

resource "azurerm_lb" "nginx_ingress_lb" {
  name                = "nginx-ingress-lb"
  location            = "eastus2"
  resource_group_name = "rsg-dev-eus2"

  frontend_ip_configuration {
    name                          = "PrivateIPAddress"
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = var.snet_k8s_in_dg_aha
  }
}

Debug Output

  # helm_release.nginx_ingress will be created
  + resource "helm_release" "nginx_ingress" {
      + atomic                     = false
      + chart                      = "nginx-ingress-controller"
      + cleanup_on_fail            = false
      + create_namespace           = true
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "nginx-ingress"
      + namespace                  = "ingress-controller"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://charts.bitnami.com/bitnami"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + values                     = [
          + <<-EOT
                # nginix ingress controller annotations
                
                controller:
                  nodeSelector:
                    agentpool: app
                  service:
                    loadBalancerIP: 10.250.26.17
                    annotations:
                      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
                      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
                      service.beta.kubernetes.io/azure-load-balancer-internal-subnet: 10.250.26.16/28
                defaultBackend:
                  nodeSelector:
                    agentpool: app
            EOT,
        ]
      + verify                     = false
      + version                    = "9.7.3"
      + wait                       = true
      + wait_for_jobs              = false
    }

NOTE: In addition to Terraform debugging, please set HELM_DEBUG=1 to enable debugging info from helm.

Panic Output

Steps to Reproduce

1. terraform apply

Expected Behavior

Should create nginx-ingress controller in the namespace nginx-ingress with the internal loadbalancer ip provided 10.250.26.17 and in the subnet range 10.250.26.16/28

I have tried with both set and also values block, both blocks did not work unfortunately.

Actual Behavior

Creates a nginx ingress controller with a public IP, which is not at all expected.

Important Factoids

References

• 🌱 Bump k8s.io/client-go from 0.27.1 to 0.28.1 #1234

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jrhouston]

Are you sure controller.service is the right path to set? I don't see a controller: block in the values.yaml for nginx-ingress. Can you link me to the chart you're using?

[nihr43]

For future googlers.. as hinted above, watch out for whether you're using charts.bitnami.com or kubernetes.github.io. The packages are different. This can probably be closed.
The bitnami package doesn't seem to like the bool passed to it from 'set', but 'values' works.
These both work and are equivalent:

bitnami:

resource "helm_release" "nginx-internal-dev-bitnami" {
  depends_on = [kubernetes_namespace.dev]
  name       = "internal-dev-bitnami"
  namespace  = "dev"

  repository = "https://charts.bitnami.com/bitnami"
  chart      = "nginx-ingress-controller"

  set {
    name  = "service.type"
    value = "LoadBalancer"
  }
  set {
    name  = "ingressClassResource.name"
    value = "internal-dev-bitnami"
  }
  values = [
    <<EOF
service:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
EOF
  ]
}

kubernetes.github.io:

resource "helm_release" "nginx-internal-dev" {
  depends_on = [kubernetes_namespace.dev]
  name       = "internal-dev"
  namespace  = "dev"

  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"

  set {
    name  = "controller.ingressClassResource.name"
    value = "internal-dev"
  }
  set {
    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal"
    value = "true"
  }
}

Charts:

ingress-nginx-4.9.1
nginx-ingress-controller-10.3.5