GKE Ingress controller ignoring ingress.class annotation

[sfriedel]

I'm running multiple ingress controllers on GKE, ingress-gce via the "HTTP load balancing" addon and ingress-nginx installed via the helm chart.

This setup was working nicely for roughly half a year but since the latest upgrade to 1.10.7-gke.1 the gce ingress controller is creating cloud load balancers for Ingress resources that it should not act on because they have an kubernetes.io/ingress.class: nginx annotation.

This is one if the Ingress resources that the GCE ingress controller created a load balancer for:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/backends: '{"k8s-be-30389--b3e8e09a299650c8":"Unknown","k8s-be-30967--b3e8e09a299650c8":"Unknown","k8s-be-31662--b3e8e09a299650c8":"Unknown","k8s-be-32269--b3e8e09a299650c8":"Unknown","k8s-be-32270--b3e8e09a299650c8":"HEALTHY","k8s-be-32586--b3e8e09a299650c8":"Unknown","k8s-be-32744--b3e8e09a299650c8":"Unknown"}'
    ingress.kubernetes.io/forwarding-rule: k8s-fw-review-review-feature-wi-fqtki7-civist--b3e8e09a299650c0
    ingress.kubernetes.io/target-proxy: k8s-tp-review-review-feature-wi-fqtki7-civist--b3e8e09a299650c0
    ingress.kubernetes.io/url-map: k8s-um-review-review-feature-wi-fqtki7-civist--b3e8e09a299650c0
    kubernetes.io/ingress.class: nginx
  creationTimestamp: 2018-09-05T13:29:12Z
  generation: 1
  labels:
    app: review-feature-wi-fqtki7-civist
    chart: civist-0.1.0
    heritage: Tiller
    release: review-feature-wi-fqtki7
  name: review-feature-wi-fqtki7-civist
  namespace: review
  resourceVersion: "68953312"
  selfLink: /apis/extensions/v1beta1/namespaces/review/ingresses/review-feature-wi-fqtki7-civist
  uid: *** REMOVED ***
spec:
  rules:
  - host: *** REMOVED ***
    http:
      paths:
      - backend:
          serviceName: svc1
          servicePort: http
  - host: *** REMOVED ***
    http:
      paths:
      - backend:
          serviceName: svc2
          servicePort: http
  - host: *** REMOVED ***
    http:
      paths:
      - backend:
          serviceName: svc3
          servicePort: http
  - host: *** REMOVED ***
    http:
      paths:
      - backend:
          serviceName: svc4
          servicePort: http
  - host: *** REMOVED ***
    http:
      paths:
      - backend:
          serviceName: svc5
          servicePort: http
  - host: *** REMOVED ***
    http:
      paths:
      - backend:
          serviceName: svc6
          servicePort: svc-http
status:
  loadBalancer:
    ingress:
    - ip: *** REMOVED ***

Any ideas why this is happening?

[freehan]

@sfriedel
Can you confirm the GCE LB resources still exists? You can go to the load balancer UI and check if relevant LB exists.

The annotations (e.g. ingress.kubernetes.io/backends) for the GCE resources may be misleading. The ingress-gce controller does not remove those annotations when ingress class changed. If the ingress once was processed by ingress-gce, then annotations will exist.

[sfriedel]

Can you confirm the GCE LB resources still exists? You can go to the load balancer UI and check if relevant LB exists.

@freehan I just checked and all the GCE LB resources still exist.

Interestingly the UI shows all backends as healthy while in the ingress.kubernetes.io/backends annotation they are still labeled "Unknown".

The annotations (e.g. ingress.kubernetes.io/backends) for the GCE resources may be misleading. The ingress-gce controller does not remove those annotations when ingress class changed. If the ingress once was processed by ingress-gce, then annotations will exist.

I never changed the ingress class annotation for the affected Ingress resources. They were created with the kubernetes.io/ingress.class already set to nginx so in my understanding the gce ingress controller should have ignored them.

[freehan]

@sfriedel
I believe the reason why ingress-gce controller has not GC the LB resource is due to this: #481.
Can you confirm if you have any other GCE ingress?

I am not sure why the LB was created in the first place.

Interestingly the UI shows all backends as healthy while in the ingress.kubernetes.io/backends annotation they are still labeled "Unknown".

This is because ingress-gce controller no longer process this ingress. And it sets it unknown usually when LB was just created https://github.com/kubernetes/ingress-gce/blob/master/pkg/backends/backends.go#L171

Is it possible that some of your scripts adds the ingress class annotation after the ingress was created? For instance,kubectl apply or kubectl annotate

[sfriedel]

@freehan Thank you for taking the time to look into this

I believe the reason why ingress-gce controller has not GC the LB resource is due to this: #481.
Can you confirm if you have any other GCE ingress?

I do have other GCE ingresses running in the same GKE cluster. They are living in a separate kubernetes namespace though if that makes a difference.

~$ kubectl get ing --all-namespaces -o jsonpath='{.items[*].metadata.annotations.kubernetes\.io/ingress\.class}
gce gce gce nginx nginx

Is it possible that some of your scripts adds the ingress class annotation after the ingress was created? For instance,kubectl apply or kubectl annotate

The ingresses were created via helm, but as far as I'm aware helm only patches resources when it upgrades an existing release. For the affected ingress the helm release is at revision 1 so it was never updated after the initial helm install. So unless helm does something weird the kubernetes.io/ingress.class: nginx was already present when the ingress was created.

[freehan]

@sfriedel
Could you send me your project name, cluster name and cluster location (zonal or regional) to my email?
I can take a closer look.

[sfriedel]

@freehan done, thank you!

[sfriedel]

@freehan I did not see this behavior again and also since the update to v1.10.7-gke.2 the Ingress sync seems to behave as expected.

/close

[k8s-ci-robot]

@sfriedel: Closing this issue.

In response to this:

@freehan I did not see this behavior again and also since the update to v1.10.7-gke.2 the Ingress sync seems to behave as expected.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[thomasjungblut]

I have a cluster with 1.11.6-gke.3 and I see this behaviour again, I've set the annotation kubernetes.io/ingress.class: "nginx-ingress-internal" on the ingress object but it still creates a loadbalancer for me.

quick describe on the ingress itself:

Name:             api-service-ingress
Namespace:        javap
Address:          35.201.87.176
Default backend:  default-http-backend:80 (10.20.1.9:8080)
Rules:
  Host         Path  Backends
  ----         ----  --------
  example.com
               /a/*   api-service-a:8080 (<none>)
               /b/*   api-service-b:8080 (<none>)
Annotations:
  kubernetes.io/ingress.class:            nginx-ingress-internal
  ingress.kubernetes.io/backends:         {"k8s-be-30447--5c15d52dd0dea308":"Unknown","k8s-be-31564--5c15d52dd0dea308":"HEALTHY","k8s-be-32076--5c15d52dd0dea308":"HEALTHY"}
  ingress.kubernetes.io/forwarding-rule:  k8s-fw-javap-api-service-ingress--5c15d52dd0dea308
  ingress.kubernetes.io/target-proxy:     k8s-tp-javap-api-service-ingress--5c15d52dd0dea308
  ingress.kubernetes.io/url-map:          k8s-um-javap-api-service-ingress--5c15d52dd0dea308
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  CREATE  8m25s                 nginx-ingress-controller  Ingress javap/api-service-ingress
  Normal  CREATE  8m25s                 nginx-ingress-controller  Ingress javap/api-service-ingress
  Normal  CREATE  6m26s                 loadbalancer-controller   ip: 35.201.XX.XXX
  Normal  UPDATE  118s (x3 over 6m26s)  nginx-ingress-controller  Ingress javap/api-service-ingress
  Normal  UPDATE  118s (x3 over 6m26s)  nginx-ingress-controller  Ingress javap/api-service-ingress

[rramkumar1]

@thomasjungblut Looks like the fix for this has not been released on GKE yet.

I'm curious how you got into this state? Did you first create the Ingress and then a couple minutes later add the annotation?

[thomasjungblut]

Did you first create the Ingress and then a couple minutes later add the annotation?

I'm using helm and I purged and installed it from scratch, so no- this should've been one transaction ideally.

It's some really weird interaction and I'm also not entirely sure if Helm isn't to blame (v2.12.3). I'm co-deploying quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0 as listening on the nginx-ingress-internal class. Turned out that I created an issue with its leader election (configmap rbac rule missing) which rendered it "inactive".

After I fixed the leader election issue, I receive the following in my logs, which show that it was removing the external loadbalancer from the status field:

I0129 22:38:46.906502       6 status.go:388] updating Ingress javap/api-service-ingress status from [{35.201.XXX.XXX }] to []
I0129 22:38:46.914129       6 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"javap", Name:"api-service-ingress", UID:"94782b99-2415-11e9-a11c-42010a9a008e", APIVersion:"extensions/v1beta1", ResourceVersion:"295897", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress javap/api-service-ingress
I0129 22:38:51.254664       6 store.go:393] ignoring ingress ingress-nginx-gce-ingress based on annotation kubernetes.io/ingress.class

Which is an even funnier state, the load balancer continues to exist but the status in the ingress resource doesn't reflect it anymore (GCE annotations are still there though).

If I deploy the ingress cleanly (via kubectl delete and helm upgrade, so the nginx ingress controller is already running) I don't see the loadbalancer-controller interfering.

So looks like this is some race between deploying the nginx ingress controller and the ingress that is picked up by the loadbalancer-controller. Is my assumption correct that the loadbalancer-controller will just create a load balancer if there is no competing "ingress.class" annotation and thus ignore the annotation completely?

If so, then I would simply need to setup a hook in Helm to have the ingress applied after everything else. Not sure if I would expect this behaviour from the loadbalancer-controller though without the nginx ingress controller deployed, should be fairly easy to test in isolation.

[rramkumar1]

@thomasjungblut Sorry for the delay in response.

Is my assumption correct that the loadbalancer-controller will just create a load balancer if there is no competing "ingress.class" annotation and thus ignore the annotation completely

If there is no ingress.class annotation present, ingress-gce will go ahead and create the LB. If the annotation is present but not set to "gce", it will not create the LB. Hope that clarifies.

Is there anything else I can do to help on this issue or should it be closed?

[thomasjungblut]

Makes sense, let's close. Thanks for the clarification.

[stinger-forever]

Recently , I have same issue on both clusters....
I have some GCE ingresses and have several nginx.class ingresses on the same cluster.
Now when I create new nginx class controller, service implementing successfully, get proper IP address, but the nginx class annotaion is ignored during deploy and new L7 balancer is creating.