Gateway API: add an annotation to control the 15021 status port in the generated Service

[vpedosyuk]

This PR adds a new Gateway annotation - networking.istio.io/service-expose-status-port. This is related to istio/istio#54453. This is a dependency for istio/istio#54525

[istio-policy-bot]

😊 Welcome @vpedosyuk! This is either your first contribution to the Istio api repo, or it's been
a while since you've been here.

You can learn more about the Istio working groups, Code of Conduct, and contribution guidelines
by referring to Contributing to Istio.

Thanks for contributing!

Courtesy of your friendly welcome wagon.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: vpedosyuk / name: Vlad Pedosyuk (15c295d, dafd0ee)

[istio-testing]

Hi @vpedosyuk. Thanks for your PR.

I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[costinm]

Alternative: install option for Istiod ( I assume someone using status port won't want it only for a subset of gateways).

If there is really a common use cases for users to fine tune each gateway with a different port - some design doc or survey on what other gateway implementations (in K8S) are using for status and health checking, before defining a new Istio only API would be great, so at least we know what we reinvent.

AFAIK - we already have a way to fine tune the service, I don't remember if we documented the naming scheme or is internal only - but like alpha/experimental it would be hard to change, and users are already familiar with configuring Service.

[linsun]

/ok-to-test

[linsun]

Do you need this configuration per gateway? I tend to agree install option seems better

[vpedosyuk]

Hi!

An install option is indeed an alternative. I actually started with this approach, but then it seemed to me that having a dedicated annotation might be a slightly more intuitive approach. It would nicely fit with the existing networking.istio.io/service-type annotation.

I suppose there's one pretty common use case when an Istio Gateway acts as a second-layer gateway and has some other gateway in front of it, such as an Application Load Balancer managed by a cloud provider. In this case, the health check endpoint is configurable and can be targeted at the 15021 port, e.g.:

1. GKE: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-configuration#direct_health
2. EKS: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.11/guide/ingress/annotations/#health-check

Such an internal Gateway would have the following annotations:

annotations:
  networking.istio.io/service-type: "ClusterIP"
  networking.istio.io/service-expose-status-port: "true"

I think it's a perfectly possible situation when a user would apply the above scheme for all incoming HTTP traffic (e.g. to integrate with GCP Cloud Armor seamlessly). But then for TCP traffic like Kafka or some special HTTP traffic, they'd expose another Gateway without the status port:

annotations:
  networking.istio.io/service-type: "LoadBalancer"
  networking.istio.io/service-expose-status-port: "false"

I don't think the install option would be sufficient for this use case.

[istio-testing]

@vpedosyuk: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
release-notes_api	15c295d	link	false	/test release-notes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[vpedosyuk]

@costinm I've checked multiple Gateway API implementations and it seems like only the Envoy Gateway implements deep Gateway customizations:
https://gateway.envoyproxy.io/docs/tasks/operations/customize-envoyproxy/
https://gateway.envoyproxy.io/docs/api/extension_types/#envoyproxy

So when it comes to the Gateway resource customizing approach, you're right about reinventing, there's an API defined for that:
https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.GatewayInfrastructure
https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.LocalParametersReference

I suppose implementation of that API in Istio is being discussed in #46594. I don't know if it'd take into account this particular issue with the 15021 port. Though, it'd be great to avoid any service spec yaml patches through parametersRef just to remove this conflicting port from my GKE loadbalancers 😅.

[costinm]

I understand the concern about your gateway, but we would have to maintain this API for a very long time, expose the status port as API ( which I think is more implementation detail and doesn't have a standard or documented behavior), and add more complexity on whatever is reading the status port.

Perhaps a clean approach would be to define a new Listener - because it is an actual listener - we could use the name in the code to identify the intent to be an status port. No API change, and it would work for the other 'magic ports' we have (capture, etc).

A benefit is that other policies could be attached, maybe even routes and security.

The question would be how to indicate internal handlers - maybe using reserved names for the backendRef, like x-status or x-prometheus.

The much simpler approach is to customize the injection template or add a mesh config option....