Support YARN for resource management

[linhr]

It should be possible to generate a Rust client for the Hadoop RPC protocol and communicate with YARN for resource management.

[zuston]

There is no need to implement this by the hadoop protocol, you can simply use the java to start/stop the sail process from my point.

[linhr]

There is no need to implement this by the hadoop protocol, you can simply use the java to start/stop the sail process from my point.

Thanks for the suggestion! This is a viable approach. We'll evaluate the path forward when we start to work on this.

[zuston]

There is no need to implement this by the hadoop protocol, you can simply use the java to start/stop the sail process from my point.

Thanks for the suggestion! This is a viable approach. We'll evaluate the path forward when we start to work on this.

Yes, I think I have some experience on this. If you want to do this, I'm glad to join this design discussion

[linhr]

@zuston Thanks for your interest in this! We'd love to have you in the design discussion.

What do you think about the option that uses the Hadoop RPC protocol in Rust? Admittedly, this is a non-trivial option that I haven't seen many people doing. The benefit is that the JVM won't be a dependency of Sail, so the user experience remains straightforward.

[zuston]

What do you think about the option that uses the Hadoop RPC protocol in Rust? Admittedly, this is a non-trivial option that I haven't seen many people doing. The benefit is that the JVM won't be a dependency of Sail, so the user experience remains straightforward.

Yes, the design with Hadoop RPC protocol is cool but not easy to make all things aligned with the vanilla yarn client, which means the development may be harder than the JVM based impl.

BTW, the resource manager is not performance sensitive and Yarn is a legacy system. If you want to spend more time to develop Kubernetes operator for sail, I will it will be more valuable