The reliance on deprecated annotation causes Kubernetes validation issues

[tim-sendible]

What happened:

Kubernetes has in-built validation to prevent users from adding the annotation kubernetes.io/ingress.class alongside the newer spec.ingressClassName.

Although external-dns was recently updated for K8s 1.22 support, it still relies on this annotation. As a result, it is not possible to create a new ingress in one go. One must apply the ingress yaml and then apply a patch to include the annotation. This has a knock-on effect to breaking declarative 'gitOps' tools such as argoCD.

More information on the validation here: https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/pkg/apis/networking/validation/validation.go?L237#tab=references

What you expected to happen:
External-dns needs to rely on the newer spec.ingressClassName and remove the reliance on the deprecated kubernetes.io/ingress.class annotation.

How to reproduce it (as minimally and precisely as possible):
Run the following:

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  ingressClassName: nginx
  rules:
  - host: "my.example.domain"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myexample-service
            port:
              number: 80
EOF

Resulting error:
The Ingress "example" is invalid: annotations.kubernetes.io/ingress.class: Invalid value: "nginx": can not be set when the class field is also set

Anything else we need to know?:

Environment:

• External-DNS version (use external-dns --version): k8s.gcr.io/external-dns/external-dns:v0.10.0
• DNS provider: Route53
• Others:

[calexandre]

Totally agree with this.
Please note that you can still use the --annotation-filter to filter other annotations defined by yourself on your ingress objects.

In my case I run two nginx instances and two external-dns instances, one for public network and the other one for private network.

For example, to workaround this issue I annotated my ingress resources with something like kubernetes.io/external-dns-class: "nginx-public" and in my "public" external-dns instance I configured the following with --annotation-filter=kubernetes.io/external-dns-class in (nginx-public). I then used the same approach for the "private" instances.

You can use any annotation you like, just be sure to match the filter in the external-dns configuration.

By using this method you workaround the warning/error and your gitops pipeline will work without issues.

Hope it helps!

[tim-sendible]

Ah that's extremely helpful thanks @calexandre.

[alexanderkjeldaas]

This also breaks gitlab auto-dev-ops.

I'm a bit surprised to find this issue open. The issue is almost a month old and external-dns is broken on every GCP cluster running the most recent release. Shouldn't this be an extremely high priority issue for this software?

I mean, this must affect a significant percentage of all existing k8s clusters.

I assumed that packages under K8S SIGs had a maintenance team, but I can't actually find anything.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tim-sendible]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tim-sendible]

/remove-lifecycle stale

[Nuru]

Just ran into this myself while prepping for Kubernetes 1.22 upgrade.

The backward compatible workaround would be a hack reading the ingressClass field from the Ingress, generating a synthetic kubernetes.io/ingress.class: nginx annotation, and then filtering on that, but not only is that ugly, I am not sure it is even realistically feasible.

Given that the upgrade path requires non-automated changes to the Ingress resource itself, I think it is fine to require a non-automated change to external-dns, too. I propose a new filter --ingress-class-filter which filters on the Ingress.spec.ingressClassName value.

Not a substitute, but it would also be good if the project publish an official annotation for this, something like

external-dns.alpha.kubernetes.io/ingress-class

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Nuru]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[jsalatiel]

/remove-stale

[Nuru]

/remove-lifecycle stale

[tchupp]

This is still relevant 👀

[dylanhitt]

It's been awhile since I've implemented this and I was surprised to find the best way to filter is --annotation-filter. I'd expect to find a --ingressClass filter filtering with spec.ingressClassName then annotation-filter filtering the previous results. If ingressClass isn't set it maintains the current behavior.

I don't know. However, this project has provided me a lot of value. I wouldn't mind taking a crack at it, but some guidance on the desired approach would be helpful.

[gw0]

I believe the solution is implemented in this unapproved PR #2054.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Nuru]

/remove-lifecycle stale

[gw0]

I believe this is fixed with #3589 included in the latest release v0.13.5.

[szuecs]

Please tell if it wasn't fixed and we will reopen it
/close

[k8s-ci-robot]

@szuecs: Closing this issue.

In response to this:

Please tell if it wasn't fixed and we will reopen it
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.