[UX] Fail early from provisioning when rsync is uninstalled

[landscapepainter]

This PR closes #2113

When rsync is not installed in the user's machine, ray_up() is reattempted for _MAX_RAY_UP_RETRY number of times without failing. As this is time consuming, this PR fails early in the provisioning step when rsync is not installed and gently hints the user to install rsync.

Considered failing from need_ray_up() as well, but at this point, state.db is already updated with INIT status of the cluster attempted to provisioned which isn't the most optimal scenario. Also, it takes time for single failure before reaching need_ray_up().

Tested (run the relevant ones):

• Code formatting: bash format.sh
• manally tested with running sky launch after deleting rsync with and w/o --dryrun option

[Michaelvll]

Thank you for the fix @landscapepainter! We may need to move the check into the upper level?

[landscapepainter]

@Michaelvll Ready for another look. Moved the check to _provision and doing the check only when it's not a dryrun. Tested with and without --dryrun flag as well after uninstalling rsync.

[Michaelvll]

@Michaelvll Ready for another look. Moved the check to _provision and doing the check only when it's not a dryrun. Tested with and without --dryrun flag as well after uninstalling rsync.

Thanks for the update @landscapepainter! Why do we only check it when it's not a dryrun? I feel like we can have it for the dryrun case as well. Aslo, can we have a separate function for the checking and call that function in the _provision? Reason: in the future, we may want to have more backend-specific prerequisite tests.

[landscapepainter]

@Michaelvll I guess it makes sense if the user can check while dryrun as well. Ready for another look!

[Michaelvll]

Thanks for the fix @landscapepainter! LGTM. Left two comments.