Proxy can act as a Kubernetes Ingress Controller

[galvesribeiro]

Hello folks!

First of all, great work! Congratz for the first preview release! It is indeed something .Net Community was lagging behind and had for too long to deal with clumsy things like nginx.

So, I'm coming to a point where I'd like to make a POC with YARP for a controlled small set of microservices in Kubernetes and that comes to a need to make it an ingestion controller.

With that in mind, I would like to ask if the Kube integration is something you guys plan to have on this repo, on some other under MSFT umbrella, or if it is something that the community will have to implemente for their own needs.

If it is something MSFT will (or already is) develop, ok, all we have todo is wait. If that is something that community will have to build, is that something that would be acceptable on this repo as a PR? If we should go to this PR, I'de like to use this issue to discuss the approach we should take to contribute it.

Thanks! Keep it going!

[cwoolum]

I was just thinking it would be pretty cool to have an operator for this. I don't think it would make sense in this repo though. Also, I don't see there even being a docker image based on this repo since this is only a library.

The operator would probably want to provide guidance on building an image.. maybe a Dockerfile?

[davidfowl]

I think it should be in this repository as another project.

[galvesribeiro]

Yeah, I think that too. The other project can contain the Dockerfile and whatever is required for the controller to run.

[samsp-msft]

Acting as a K8s ingestion controller is something we have been thinking about and is on the backlog, but we don't yet know where it would fit in the timeline.

The first step is what are your requirements?

[galvesribeiro]

@samsp-msft Well, the current proxy functionality that the framework has in the current preview, has everything we need for now.

What would be generally required for people using Kubernetes is an ingress controller that can be configured using kubernetes config objects, like YAML, and that can spin up/down pods that will handle the traffic and at same time, provide configuration to those pods in a transparent way so whenever you change the config of the ingress deployment, it would "replicate" the config to the running proxy pods.

I'll dig into the specifics until this weekend and will post back here some sort of proposal.

Will keep in touch.

[davidfowl]

We have a prototype of Kestrel as an ingress written by @jkotalik so we have a pretty good idea of what's required to integrate YARP.

[jkotalik]

Yeah, I was tempted to port that over soon as a starting piece that can be improved upon. Code is here: https://github.com/jkotalik/kestrel-ingress

[galvesribeiro]

Hey @jkotalik that is awesome :) That follow more or less the design that I had in mind.

Implement it using YARP would be easier. The only thing I'm concerned is the Kubernetes API client... I've being researching multiple clients to use back in the day when I was writing Orleans.Clustering.Kubernetes and all of them was really a mess by either not being maintained or lacking auth/credentials support, or any other problem that made me just create one with the APIs that I need to use.

Unless we plan on having a Microsoft.Extensions.KubernetesClient to be reused and maintained under .Net Foundation (just a joke - we do have a client for pure Docker with Docker.DotNet btw), I think we should do the same, use HttpClientFactory nicities and invoke only the necessary kube APIs for this.

So, regardless of the way we go with it, is this something you wanna tackle and bring the initial stuff in? Or is it something we should discuss more, build a plan and reimplement it leveraging what you have?

[davidfowl]

No we’re not writing a k8s client. The C# client is maintained and we’ll contribute to it as needed. I’m sure @brendandburns would appreciate it. Turns out it would also be continent to have informers implemented for this PR (kubernetes-client/csharp#394) would be helpful @macsux

[galvesribeiro]

Okey, no problem. I dropped that client back in the day I don't remember why but I hope it got better. Glad to know it is having informers support.

[rwkarg]

If the Ingress Controller allows specifying additional clusters (and credentials) in addition to the cluster it's running in, then that would allow for multi-cluster ingress. This would bring parity with the Google Anthos multi-cluster Ingress.

There could either be a single selected cluster that hosts the YARP instances (this matches the Anthos model) or YARP could be hosted outside of k8s and just have a list of clusters to monitor (a Controller doesn't have to run from within a k8s cluster, it just needs credentials/permissions).

Further, allowing a cluster discovery mechanism (here's credentials, monitor for aks/eks/gke clusters in this account) would allow clusters to be created/destroyed and dynamically get picked up without config changes.

[irperez]

I'm excited to see that ingress for YARP is being considered. To get real adoption in my opinion, you need to handle HTTPS within the ingress itself which I don't see in the kestrel ingress prototype. In particular leverage Cert-Manager infrastructure. Which makes generating those certs super easy.

[rahamohebbi]

It is very exciting to see this feature being worked on for 1.0 release. I am planning to replace my custom made load balancer (that is based on ProxyKit) with YARP.

Is there a way to use the Kubernetes Ingress feature before 1.0 release. I can help with testing it as well.

[Tratcher]

@rahamohebbi there will be public previews before the 1.0 release. Stay tuned on this issue for progress on the feature.

[jkotalik]

We have liftoff folks! Initial implementation for the k8s ingress controller in draft.

[Lukejkw]

Please excuse my noob question but if you were wanting to leverage other functionality in your gateway such as authn and authz, how are those settings defined in the ingress?

Would you not just be able to expose your gateway (running YARP) using a Load Balancer service and then configure it as per normal? That way you still only have one public IP and can route to backend services as necessary?

I suspect I am missing something here