[Core] Auto mapping for cluster name

[Michaelvll]

Closes #1853 and Closes #963

The backward compatibility is guaranteed by the restore of ray yaml in the backend_utils.write_cluster_config

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • sky launch -c i-am-a-very-long-cluster-name-with-more-than-35-characters --cloud gcp --cpus 2
  • sky status -r
  • sky exec i-am-a-very-long-cluster-name-with-more-than-35-characters echo hi
  • sky autostop -i 1 i-am-a-very-long-cluster-name-with-more-than-35-characters
  • sky status -r
• All smoke tests: pytest tests/test_smoke.py
• All smoke tests: pytest tests/test_smoke.py --aws
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[Michaelvll]

Logging/UX: Tried sky spot launch --cpus 2+ -n spot-maskgit-minerl-maskgit-interp --cloud gcp 'for i in $(seq 10000); do echo $i; sleep 1; done' which is a long name a user told us before. In controller logs

Hey @concretevitamin, thanks for pointing this out! We intentionally truncate the cluster name earlier to 30 chars to avoid the cluster name being too long after appending the job id, which will cause another truncation in the underlying sky.launch, hiding the job_id in the cluster name. I think it should be fine, as a normal user is not expected to use sky status in the spot controller, and the cluster name in the controller log is not critical as there will be a single cluster name for each spot job. If we check the sky spot queue, the spot job should be the same as the user specified. Wdyt?

[Michaelvll]

Thanks for the detailed review @concretevitamin @cblmemo! I fixed most of the comments.

I just realized that we should not tag clusters with original cluster names, as for most of the clouds, tags have the same length limit as cluster names, so I removed the tags.

TODO:

• Add a readme for the design of cluster name mapping
• Test again.
  • manual test
  • pytest tests/test_smoke.py
  • pytest tests/test_smoke.py --aws

[cblmemo]

Thanks for the detailed review @concretevitamin @cblmemo! I fixed most of the comments.

I just realized that we should not tag clusters with original cluster names, as for most of the clouds, tags have the same length limit as cluster names, so I removed the tags.

TODO:

• Add a readme for design of cluster name mapping
• Test again.

Just a quick thought - then maybe we could have a column for cluster_name_on_cloud for sky status -a..?

[concretevitamin]

LGTM @Michaelvll - thanks for this major improvement!