Possible bug? IP Whitelist not working in one environment but works in another #6658

javierguzman · 2024-10-15T09:20:36Z

javierguzman
Oct 15, 2024

Hello all,

I have two environments "testing" and "qa". I set nginx-ingress in my cluster using Helm with Terraform:

resource "helm_release" "ingress" {
  name      = "${var.environment}-ingress"
  namespace = kubernetes_namespace_v1.ingress_namespace.metadata[0].name
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart     = "ingress-nginx"
  version   = "4.11.0"
  create_namespace = false
  values = [templatefile("${path.module}/customValues.tftpl", { lb_name = local.lb_name })]
}

Then my customValues file is as follows:

controller:
  allowSnippetAnnotations: true
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: external
      service.beta.kubernetes.io/aws-load-balancer-name: ${lb_name}
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "10254"
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/healthz"
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "30"
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "3"
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
ingressShim:
  enabled: true

And then one of my ingresses is as follows:

resource "kubernetes_ingress_v1" "ingress_business_portal" {
 // we use external dns only if we don't have the CDN
 wait_for_load_balancer = true

 metadata {
   name      = "business-portal"
   namespace = local.business_portal_namespace
   annotations = {
     "nginx.ingress.kubernetes.io/ssl-redirect": true
     "nginx.ingress.kubernetes.io/from-to-www-redirect": true
     "cert-manager.io/cluster-issuer": "${var.environment}-letsencrypt-cluster-issuer"
     "cert-manager.io/renew-before": "360h" // 15d
     "cert-manager.io/duration": "2160h" // 90d
     "cert-manager.io/private-key-algorithm": "RSA"
     "cert-manager.io/private-key-size": 2048
     "cert-manager.io/private-key-encoding": "PKCS1"
     "cert-manager.io/private-key-rotation-policy": "Always"
     "cert-manager.io/common-name": local.domain_environment,
     "nginx.ingress.kubernetes.io/whitelist-source-range": local.full_ip_whitelist
     "external-dns.alpha.kubernetes.io/hostname": var.is_cdn_enabled ? "" : "${local.domain_environment},www.${local.domain_environment}"
     "external-dns.alpha.kubernetes.io/ignore": var.is_cdn_enabled ? "true": "false"
   }

   labels = {
     app = "business-portal"
   }
 }
 spec {
   ingress_class_name = "nginx"

   tls {
     hosts = [local.domain_environment]
     secret_name = local.cert_secret_name
   }

   rule {
     host = local.domain_environment
     http {
       path {
         path      = "/"
         path_type = "Prefix"
         backend {
           service {
             name = kubernetes_service_v1.business_portal_service.metadata[0].name
             port {
               number = 3000
             }
           }
         }
       }
     }
   }
 }
 timeouts {
   create = "5m"
   delete = "5m"
 }
}

However, IP Whitelist works on QA but no in testing, even though when I retrieve the configmaps and describe both ingresses they seem equal. Has anyone faced something similar? The only thing that I can think of is that in the past I tried to enabled proxy protocol in testing but ended up giving up. I have tried "kubectl rollout restart deployment testing-ingress-ingress-nginx-controller -n testing-ingress" as well.

Thank you in advance and regards