[Managed Jobs] Support Rsync for managed jobs

[hemildesai]

Currently, managed jobs automatically translate local files and workdir to cloud bucket mounts. This means that managed jobs cannot be launched if a user does not have access to cloud storage.

I am wondering if its possible to support rsync for workdir and local files in managed jobs where the jobs controller will essentially act as a jump host for the rsync. So the flow would be to rsync to jobs controller first, then rsync from the jobs controller to the jobs which are launched by the controller.

Ideally, this should be bidirectional - i.e. later on, it should also be possible to rsync back any changes to the workdir from the controller to the local machine.

[romilbhardwaj]

+1 - this is important for the Kubernetes-only case where the user does not have access to any cloud object store.

[hemildesai]

@romilbhardwaj In the k8s only use case, managed jobs would also need to support the jobs controller being launched in k8s. Right now, autostop is set on the jobs controller which means it will fail to launch in k8s IIRC

[romilbhardwaj]

Running controllers on Kubernetes is supported now :) We updated the downstream code to skip autostop for Kubernetes in #3377:

skypilot/sky/backends/cloud_vm_ray_backend.py

Lines 2014 to 2021 in 7692eaa

	# Skip stop feature for Kubernetes controllers.
	if (isinstance(to_provision.cloud, clouds.Kubernetes) and
	controller_utils.Controllers.from_name(cluster_name)
	is not None):
	assert (clouds.CloudImplementationFeatures.STOP
	in requested_features), requested_features
	requested_features.remove(
	clouds.CloudImplementationFeatures.STOP)

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This issue was closed because it has been stalled for 10 days with no activity.