Operator reconciler error: failed calling webhook "mtargetgroupbinding.elbv2.k8s.aws"

[Slijkhuis]

I'm getting the following error:

Failed deploy model due to Internal error occurred: failed calling webhook "mtargetgroupbinding.elbv2.k8s.aws": Post "https://aws-load-balancer-webhook-service.kube-system.svc:443/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "aws-load-balancer-controller-ca")

My setup is as follows:

• Namespace A
  • Ingress A in group 'ingress-group-1'
  • Service 1
  • Service 2
• Namespace B
  • Ingress B in group 'ingress-group-1'
  • Service 3

The Ingress resources have the same annotations. With the exception of group.order.

Uncollapse for the full YAMLs.

ingress-a.yml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-a
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: ingress-group-1
    alb.ingress.kubernetes.io/group.order: 10
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect:
      '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode":"HTTP_301"}}'
    alb.ingress.kubernetes.io/security-groups: '...'
spec:
  rules:
    - host: service1.example.com
      http:
        paths:
          - backend:
              serviceName: service1-nodeport
              servicePort: 80
    - host: service2.example.com
      http:
        paths:
          - backend:
              serviceName: service2-nodeport
              servicePort: 80

ingress-b.yml in another namespace:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-b
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: ingress-group-1
    alb.ingress.kubernetes.io/group.order: 20
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect:
      '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode":"HTTP_301"}}'
    alb.ingress.kubernetes.io/security-groups: '...'
spec:
  rules:
    - host: service3.example.com
      http:
        paths:
          - backend:
              serviceName: service3-nodeport
              servicePort: 80

My expectations (✅ = met, ❌ = not met):

• ✅ A single ALB is created.
• ✅ The routing rules of both ingresses to be merged into a single list of rules on the 443 listener of the ALB.
• ✅ A *.example.com ACM certificate is automatically discovered and used for port 443.
• ❌ All traffic on port 80 is redirected to port 443. (✅ see update 1)
• ❌ The operator has no errors.

My expectations seem to be met relatively well, but the operator is reporting the above error nonetheless. Also, it seems the HTTP to HTTPS redirect is not working. Instead of redirecting, it actually makes the same routing rules on the ALB listener for port 80, connecting to the same target groups as the 443 listener.

Why am I getting the error?

Do my my expectations match the actual expected behaviour?

---

Update 1: the reason that the SSL redirect is not working is not related to the error it seems. I misconfigured the annotations. I'm missing the following rule in the ingress:

  - http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
        path: /*

Adding that rule actually resolves the redirect issue.

So, the only issue left is the error that the operator logs.

[M00nF1sh]

@Slijkhuis
How did you installed the controller? i assume you are using helm charts.
If using helm chart(and without using cert-manager), the chart should automatically generates a new cert on every install/upgrades.
The cert is stored as secret named aws-load-balancer-tls, you can view it by kubectl -n kube-system get secrets/aws-load-balancer-tls -o yaml

The cert's CA is configured in our two webhooks, which you can view by

kubectl get mutatingwebhookconfigurations/aws-load-balancer-webhook -o yaml
kubectl get validatingwebhookconfigurations/aws-load-balancer-webhook -o yaml

Would you help check whether the clientConfig.caBundle(which appears multiple times in each webhookConfiguration) mismatches with ca.crt in aws-load-balancer-tls secret?
If so, please update the clientConfig.caBundle to match ca.crt.
Not sure how this occurs, maybe during helm install, some resources didn't get updated correctly due to network issues

[Slijkhuis]

Thanks for pointing me in the right direction. I use the Helm chart indeed and I use ArgoCD to sync my configuration with the actual state on the cluster. It turns out that we configured ArgoCD to globally ignore some paths for checking sync status, which includes /webhooks/0/clientConfig/caBundle on admissionregistration.k8s.io/MutatingWebhookConfiguration and admissionregistration.k8s.io/ValidatingWebhookConfiguration. Clearly, this was causing the issue. After destroying aws-load-balancer-controller and recreating it everything worked smooth.

In short, it's a misconfiguration on my end. Thanks for helping me figure it out!