[Question/Feature] Can you set up ray jobs to run on pre-existing clusters?

[verbiiyo]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

We have a kubernetes cluster set up on GCP and it already is configured to run argo jobs, but I'd like to deploy my ray code to run on this cluster without messing anything up on their end. Is it possible to run jobs directly? is there additional setup? or do I need to create my own cluster?

Use case

This would help people run ray and argo jobs on the same cluster.

Related issues

I'm not sure.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

• Preferred method: You can use ray job CLI to submit your job directly (https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html).

• (Not recommended) clusterSelector (See the section "Manual tests" in [Feature] [RayJobs] Use finalizers to implement stopping a job upon cluster deletion #735 for a detailed example)

  • clusterSelector is an anti-pattern for the Kubernetes operator design pattern, and thus some bugs are hard to address. For example, when you create a RayJob custom resource with clusterSelector, two jobs will be created in RayCluster (See [Bug] 409 conflicts cause two jobs creations #756 for more details.). We did not promise to support the stability of clusterSelector.

cc @architkulkarni for more insights about ray job.