Address port range and "all ports" use cases

[jpeach]

In the API review the topic of "all ports" services was raised.

#780 (comment)

And there is a KEP around this for Services:

kubernetes/enhancements#2611

This issue is for capturing use cases for gateways that need large numbers of ports, possibly all of them.

[jpeach]

@beriberikix In the "all ports" KEP, SIP/RTP/gaming servers are mentioned. Do you know of any use cases that would need lots of Gateway ports?

[beriberikix]

In my experience, they tend to have just a handful or simply one port per protocol. But paging the expert on all things WebRTC @Sean-Der!

[robscott]

/cc @prameshj

[prameshj]

A survey was sent out as part of the services KEP to better understand the usecases. The results are in tiny.cc/allportsslides which details some usecases.

[youngnick]

Feels like I could summarise most of those use cases as "protocols designed before everyone wanted to fit everything through NAT/PAT".

I think the other use case to keep in mind for "all ports" is also "no port", for Routes that don't fit into the L4 bucket. I think that with a minimum of effort, the Gateway construct could be applied to things like IP routes, having the Gateway be a more general router. A Gateway object with a set of IPRoutes that indicated CIDR and next-hop could be a reasonable standard for a basic BGP config, for example.

More usefully, I think that a Gateway could also be considered as a VPN endpoint, allowing an IPRoute to specify what subnets are available behind the Listener.

In both of those cases, the "port" is not meaningful, so we could either:

• leave it unspecified
• ignore it
• have a way to say "all ports"

which are all roughly equivalent in terms of the actual use case.

[beriberikix]

Feels like I could summarise most of those use cases as "protocols designed before everyone wanted to fit everything through NAT/PAT".

I think that's accurate for the use cases mentioned (which is plenty!) but there are modern protocols that don't or can't rely on NAT, ex. telco and IoT. I won't get started on L3/L2 or even non-IP protocols that are part of 5G :) That said, besides what's covered in the survey, covering "ports are not meaningful" would suffice.

[ccfishk]

Feels like I could summarise most of those use cases as "protocols designed before everyone wanted to fit everything through NAT/PAT".

Pretty all-inclusive all use cases ! 👍
And we might need consider IPV6 though considerable difference is alpha-numeric address and length, NDP discovery, but features like IPSec is built into IPV6 for key infrastructure.
I dont think is too early to consider extensible to 5G protocol.

[youngnick]

Since we have decided we will leave this until after v1alpha2 is out, I'll put this in the freshly-created v1beta1 milestone.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[bowei]

/remove-lifecycle rotten

[robscott]

We discussed this issue at our community meeting yesterday, specifically trying to understand if we'd need to make any changes prior to beta graduation if we wanted to add this feature in the future. I've written a doc that explores that in more detail. My conclusion at this point is that this functionality can be added in a backwards compatible way.

Around the same time, @markmc wrote an excellent doc proposing support for port ranges on Gateway listeners. This doc also contains some great historical context for conversations related to this concept.

If others agree with my conclusion that this can be a compatible change added post-v1beta1, I think we can safely remove this issue from the v1beta1 milestone.

/cc @bowei @hbagdi @jpeach @youngnick

[youngnick]

After reading both those docs, I agree that Rob's plan makes sense. I think there are some things we'll need to be careful with (particularly the status, since creating an extra Listener status per port in the range could rapidly run up against object size limits), but since making a required port optional does not require an API revision (although it is a breaking change for implementations), we can add using the endPort idea. I like that using port and endPort also allows handling the "please dynamically allocate me a port" case as well.

[markmc]

The endPort proposal seems like a neat way of supporting ranges, and aligns with NetworkPolicy endPort

I left some comments in Rob's doc - I think we need to work out the semantics of port ranges a bit more

For the SIP/RTP use case, I understand the request is to forward traffic from a large range of ports through to a single backend with no port mapping

In my TCP gateway sharing doc, I show how individual TCPRoutes could be auto-assigned a port from within the range

If we wanted to support both "an attached route gets all the ports from the range" and "an attached route get a single port from the range", where would that be described? In the listener or the route?

[howardjohn]

If we have auto-allocated ports, how do clients known what to use? Is the expectation that a producer creates the route, reads the status, then tells clients to use that port? That seems a bit concerning since its not particularly stable. If I have multiple clusters, for example, we get a different port here.

Is there prior art in K8s ingress products to auto assign ports?

[doonhammer]

From my naive POV the a part of the core issue is that the networking model for the Gateway API is heavily dependent of a proxy model hence the listener construct for ports etc. For the use cases I am thinking of a more transparent networking model is required, and hence the need to be more flexible in IP address/port assignment/filtering. I would like to be able to implement a similar feature as the AWS Gateway Load Balancer in Kubernetes using the Gateway API. I wrote up a more detailed use case as issue 1052 , to help clarify my view point.

[markmc]

If we have auto-allocated ports, how do clients known what to use? Is the expectation that a producer creates the route, reads the status, then tells clients to use that port?

Right

That seems a bit concerning since its not particularly stable. If I have multiple clusters, for example, we get a different port here.

Love the question!

Naive clarification - is the assumption it's fine for the address to differ across clusters, or is it that you'd use DNS to ensure the address stable for clients too?

My thinking is that there are plenty of cases where it would be routine for the client to be supplied a port and address, e.g. URI-like DB connection string

Is there prior art in K8s ingress products to auto assign ports?

I can't speak usefully to that, but was more thinking towards pushing the boundaries of the API in terms of the combination of gateway sharing and TCP routing

ITSM that gateway sharing and TCP routing are important independent steps forward this API has taken, but the two don't really seem to mix very well currently?

[robscott]

Some great discussion in this thread, thanks! After going through some of the comments on my related doc, I realized that we're really talking about 2 different kinds of automatic port allocation.

1. Assign a port (or range) to a Gateway listener that does not specify one
2. Assign a port to a Route from a range on a parent Gateway listener

I think the second one is actually something we can tackle separately, and in the context of the port field on Route parent refs. I'd be interested in the use cases for that, but it may be best to move that discussion to another thread since we already have so much going on here.

I've been trying to narrow my focus to any issues that could block v1beta1. I think the discussion here has shown that this issue does not need to block v1beta1 for the following reasons:

1. We have a clear path to add this functionality in the future without breaking API changes
2. The lack of this functionality is not preventing the majority of potential uses from using the API

Will discuss this at community meeting to confirm, but if others agree, I'll remove this issue from the v1beta1 milestone.

[robscott]

We discussed this at the community meeting today. The high level summary is that we're not planning on making any port related changes in the upcoming v1beta1 release.

Although we discussed the possibility of preemptively making port optional prior to beta graduation, that would open a rather complex topic of how an empty/unspecified port should be interpreted. Instead we'll leave this decision for a later release.

As part of the upcoming release, we'll update our implementation and versioning guidelines to emphasize that required fields may become optional in future releases of the API. (This is already stated in https://gateway-api.sigs.k8s.io/concepts/versioning/#api-authors, but may not be highlighted clearly enough).

[markmc]

I've filed #1060 and #1061 in an effort to help untangle this discussion a bit.

This leaves this issue to focus on just the port range concept, and "all ports" as a special case of that.

[markmc]

I've also filed #1062 which is the "auto-assign a listener port to a route" idea which confusingly overlaps with other port range use cases.

[youngnick]

Thanks @markmc for clarifying here. I think those separate issues really help.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[shaneutt]

While grooming we saw that this one was open for a long period of time without anyone with a strong use case to champion it. We're going to close this as we don't expect anyone's ready to drive it forward, but if you still want this feature and have a strong use case we will be happy to reconsider this and re-open.