[Core] Add HOST_CONTROLLERS check for clouds

[romilbhardwaj]

Adds CloudImplementationFeatures.HOST_CONTROLLERS to check if a cloud supports hosting controllers. cc #3377 #3363

Example output with spot.controller.resources.cloud: Kubernetes in config.yaml:

(base) ➜  sky-experiments git:(core_host_controllers) ✗ sky spot launch --cloud gcp -- echo hi
Task from command: echo hi
Managed spot job 'sky-cmd' will be launched on (estimated):
I 04-02 09:18:00 optimizer.py:690] == Optimizer ==
I 04-02 09:18:00 optimizer.py:701] Target: minimizing cost
I 04-02 09:18:00 optimizer.py:713] Estimated cost: $0.1 / hour
I 04-02 09:18:00 optimizer.py:713] 
I 04-02 09:18:00 optimizer.py:836] Considered resources (1 node):
I 04-02 09:18:00 optimizer.py:906] --------------------------------------------------------------------------------------------------
I 04-02 09:18:00 optimizer.py:906]  CLOUD   INSTANCE              vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE   COST ($)   CHOSEN   
I 04-02 09:18:00 optimizer.py:906] --------------------------------------------------------------------------------------------------
I 04-02 09:18:00 optimizer.py:906]  GCP     n2-standard-8[Spot]   8       32        -              us-east4-a    0.09          ✔     
I 04-02 09:18:00 optimizer.py:906] --------------------------------------------------------------------------------------------------
I 04-02 09:18:00 optimizer.py:906] 
Launching the spot job 'sky-cmd'. Proceed? [Y/n]: Y
Launching managed spot job 'sky-cmd' from spot controller...
Launching spot controller...
I 04-02 09:18:05 optimizer.py:690] == Optimizer ==
I 04-02 09:18:05 optimizer.py:713] Estimated cost: $0.0 / hour
I 04-02 09:18:05 optimizer.py:713] 
I 04-02 09:18:05 optimizer.py:836] Considered resources (1 node):
I 04-02 09:18:05 optimizer.py:906] ----------------------------------------------------------------------------------------------
I 04-02 09:18:05 optimizer.py:906]  CLOUD        INSTANCE     vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE   COST ($)   CHOSEN   
I 04-02 09:18:05 optimizer.py:906] ----------------------------------------------------------------------------------------------
I 04-02 09:18:05 optimizer.py:906]  Kubernetes   8CPU--24GB   8       24        -              kubernetes    0.00          ✔     
I 04-02 09:18:05 optimizer.py:906] ----------------------------------------------------------------------------------------------
I 04-02 09:18:05 optimizer.py:906] 
I 04-02 09:18:05 cloud_vm_ray_backend.py:4238] Creating a new cluster: 'sky-spot-controller-2ea485ea' [1x Kubernetes(8CPU--24GB, cpus=8+, mem=3x, disk_size=50)].
I 04-02 09:18:05 cloud_vm_ray_backend.py:4238] Tip: to reuse an existing cluster, specify --cluster (-c). Run `sky status` to see existing clusters.
W 04-02 09:18:05 cloud_vm_ray_backend.py:2013] sky.exceptions.NotSupportedError: The following features are not supported by Kubernetes:
W 04-02 09:18:05 cloud_vm_ray_backend.py:2013]  Feature           Reason                                     
W 04-02 09:18:05 cloud_vm_ray_backend.py:2013]  host_controllers  Kubernetes cannot host controllers.        
W 04-02 09:18:05 cloud_vm_ray_backend.py:2013]  stop              Kubernetes does not support stopping VMs.  
W 04-02 09:18:05 cloud_vm_ray_backend.py:2039] 
W 04-02 09:18:05 cloud_vm_ray_backend.py:2039] Provision failed for 1x Kubernetes(8CPU--24GB, cpus=8+, mem=3x, disk_size=50) in kubernetes. Trying other locations (if any).
E 04-02 09:18:05 cloud_vm_ray_backend.py:2690] Failed to provision all possible launchable resources. Relax the task's resource requirements: 1x Kubernetes(cpus=8+, mem=3x, disk_size=50)
I 04-02 09:18:05 cloud_vm_ray_backend.py:2694] === Retry until up ===
I 04-02 09:18:05 cloud_vm_ray_backend.py:2694] Retrying provisioning after 32s (backoff with random jittering). Already tried 1 attempt.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Manual tests- sky spot launch --cloud gcp -- echo hi with kubernetes and aws controllers.

[romilbhardwaj]

Will be removed after #3377

[Michaelvll]

Thanks for separating this @romilbhardwaj! LGTM.

[Michaelvll]

cc'ing @cblmemo to add clouds that should be black list for hosting the controllers.