Optimizing & Provisioning Retries at the granularity of regions/zones #975

WoosukKwon · 2022-07-15T08:09:59Z

This PR closes #930

For on-demand instances, the optimizer selects the cloud region (e.g., AWS us-west-1) to run the task on. And a provisioning request is made for that region. If it fails, Sky re-runs the optimizer and retries for the next cheapest cloud region (e.g., it can be GCP us-east1).

For spot instances, the optimizer additionally decides the availability zone (e.g., AWS us-west-1a). A provisioning request is made for that zone. If it fails, Sky re-runs the optimizer and retries for other zones (possibly in other clouds and regions).

This PR also fixes many bugs when a region is specified with sky launch. For example,

sky launch "" -c test --gpus V100 --cloud aws --region us-west-1: us-west-1 does not have p3 instances. However, Sky does not return errors before launching.
sky launch "" -c test --gpus K80 --region southcentralus: The price of Standard_NC6_Promo in the southcentralus region is $ 0.22 / hour, but the optimizer says it is $ 0.18/hour, which is the price of the instance in other regions.

Tests:

Smoke tests
sky launch "" --use-spot --cloud aws --gpus A100:8
sky tpunode --region europe-west4 (NOTE: this command fails because our GCP catalog does not have any N1 machines for the regions outside the US.)
sky tpunode --region europe-west4 --tpu-vm (NOTE: this command succeeds because TPU VMs do not require N1 machines.)

Michaelvll · 2022-07-15T16:54:58Z

Thanks for the fantastic and quick work! Just scanned the PR and have several questions:

What are the bugs fixed in this PR? It would be nice to have a record here in the description.
It would be nice to list some tests in the description.

I will take a deeper look at the modifications soon. : )

Future TODO: We may want to consider adding the sky storages with object store to the input automatically so that the optimizer can take the data movement of the file mounts into consideration.

WoosukKwon · 2022-07-15T23:20:35Z

Future TODO: We may want to consider adding the sky storages with object store to the input automatically so that the optimizer can take the data movement of the file mounts into consideration.

@Michaelvll That's a great idea. Especially in the copy mode, Sky can do that since it knows the amount of data in the bucket. But the problem is 1) I'm not sure if Sky can accurately estimate the egress cost (because I think the formula is pretty complex), 2) Sky should also consider the VM idle time caused by the data egress, which is hard to estimate.

concretevitamin

Very nice @WoosukKwon! Did a quick pass, except for optimizer.py.

A high-level question:

» sky cpunode
I 08-10 19:57:09 optimizer.py:602] == Optimizer ==
I 08-10 19:57:09 optimizer.py:614] Target: minimizing cost
I 08-10 19:57:09 optimizer.py:625] Estimated cost: $0.4 / hour
I 08-10 19:57:09 optimizer.py:625]
I 08-10 19:57:09 optimizer.py:677] Considered resources (1 node):
I 08-10 19:57:09 optimizer.py:705] ------------------------------------------------------------------------------
I 08-10 19:57:09 optimizer.py:705]  CLOUD   INSTANCE         ACCELERATORS   REGION (ZONE)    COST ($)   CHOSEN
I 08-10 19:57:09 optimizer.py:705] ------------------------------------------------------------------------------
I 08-10 19:57:09 optimizer.py:705]  AWS     m6i.2xlarge      -              us-east-1        0.38          ✔
I 08-10 19:57:09 optimizer.py:705]  AWS     m6i.2xlarge      -              us-east-2        0.38
I 08-10 19:57:09 optimizer.py:705]  AWS     m6i.2xlarge      -              us-west-2        0.38
I 08-10 19:57:09 optimizer.py:705]  Azure   Standard_D8_v4   -              eastus           0.38
I 08-10 19:57:09 optimizer.py:705]  Azure   Standard_D8_v4   -              eastus2          0.38
I 08-10 19:57:09 optimizer.py:705]  Azure   Standard_D8_v4   -              northcentralus   0.38
I 08-10 19:57:09 optimizer.py:705]  Azure   Standard_D8_v4   -              westus2          0.38
I 08-10 19:57:09 optimizer.py:705]  Azure   Standard_D8_v4   -              centralus        0.43
I 08-10 19:57:09 optimizer.py:705]  AWS     m6i.2xlarge      -              us-west-1        0.45
I 08-10 19:57:09 optimizer.py:705]  Azure   Standard_D8_v4   -              westus           0.45
I 08-10 19:57:09 optimizer.py:705]  Azure   Standard_D8_v4   -              southcentralus   0.46
I 08-10 19:57:09 optimizer.py:705]  Azure   Standard_D8_v4   -              westcentralus    0.46
I 08-10 19:57:09 optimizer.py:705]  GCP     n1-highmem-8     -              us-west1         0.47
I 08-10 19:57:09 optimizer.py:705]  GCP     n1-highmem-8     -              us-central1      0.47
I 08-10 19:57:09 optimizer.py:705]  GCP     n1-highmem-8     -              us-east1         0.47
I 08-10 19:57:09 optimizer.py:705]  GCP     n1-highmem-8     -              us-east4         0.47
I 08-10 19:57:09 optimizer.py:705]  GCP     n1-highmem-8     -              us-west2         0.47
I 08-10 19:57:09 optimizer.py:705]  GCP     n1-highmem-8     -              us-west4         0.47
I 08-10 19:57:09 optimizer.py:705] ------------------------------------------------------------------------------

» sky gpunode --gpus V100                                                       1 ↵
I 08-10 19:57:34 optimizer.py:602] == Optimizer ==
I 08-10 19:57:34 optimizer.py:614] Target: minimizing cost
I 08-10 19:57:34 optimizer.py:625] Estimated cost: $3.0 / hour
I 08-10 19:57:34 optimizer.py:625]
I 08-10 19:57:34 optimizer.py:677] Considered resources (1 node):
I 08-10 19:57:34 optimizer.py:705] --------------------------------------------------------------------------------
I 08-10 19:57:34 optimizer.py:705]  CLOUD   INSTANCE           ACCELERATORS   REGION (ZONE)    COST ($)   CHOSEN
I 08-10 19:57:34 optimizer.py:705] --------------------------------------------------------------------------------
I 08-10 19:57:34 optimizer.py:705]  GCP     n1-highmem-8       V100:1         us-central1      2.95          ✔
I 08-10 19:57:34 optimizer.py:705]  GCP     n1-highmem-8       V100:1         us-west1         2.95
I 08-10 19:57:34 optimizer.py:705]  GCP     n1-highmem-8       V100:1         us-east1         2.95
I 08-10 19:57:34 optimizer.py:705]  AWS     p3.2xlarge         V100:1         us-east-1        3.06
I 08-10 19:57:34 optimizer.py:705]  AWS     p3.2xlarge         V100:1         us-east-2        3.06
I 08-10 19:57:34 optimizer.py:705]  AWS     p3.2xlarge         V100:1         us-west-2        3.06
I 08-10 19:57:34 optimizer.py:705]  Azure   Standard_NC6s_v3   V100:1         eastus           3.06
I 08-10 19:57:34 optimizer.py:705]  Azure   Standard_NC6s_v3   V100:1         eastus2          3.06
I 08-10 19:57:34 optimizer.py:705]  Azure   Standard_NC6s_v3   V100:1         westus2          3.06
I 08-10 19:57:34 optimizer.py:705]  Azure   Standard_NC6s_v3   V100:1         southcentralus   3.37
I 08-10 19:57:34 optimizer.py:705]  Azure   Standard_NC6s_v3   V100:1         centralus        3.46
I 08-10 19:57:34 optimizer.py:705]  Azure   Standard_NC6s_v3   V100:1         westus           3.98
I 08-10 19:57:34 optimizer.py:705] --------------------------------------------------------------------------------

Feels much longer than before. Do you think it makes sense to still keep 1 row per cloud (not showing REGION (ZONE)) but make the COST ($) column correctly pick the cheapest among valid region-zones per cloud?

sky/clouds/cloud.py

sky/resources.py

concretevitamin · 2022-08-11T02:24:19Z

sky/resources.py

+            return False
+
+        if self.zone is not None and self.zone != other.zone:
+            return False


Add something like L453?

I seems like it was added by #1014?

sky/resources.py

sky/clouds/service_catalog/__init__.py

sky/clouds/service_catalog/aws_catalog.py

sky/clouds/service_catalog/azure_catalog.py

sky/clouds/service_catalog/common.py

sky/backends/cloud_vm_ray_backend.py

sky/resources.py

WoosukKwon · 2022-12-17T05:03:59Z

@concretevitamin @Michaelvll
I have a dumb question on one of our CLI tests: what is the right behavior for sky launch --cloud aws --instance-type p3.2xlarge --gpus V100:0.5 ?

Raise an error
Current master branch:

I 12-16 21:00:22 optimizer.py:606] == Optimizer ==
I 12-16 21:00:22 optimizer.py:629] Estimated cost: $3.1 / hour
I 12-16 21:00:22 optimizer.py:629] 
I 12-16 21:00:22 optimizer.py:685] Considered resources (1 node):
I 12-16 21:00:22 optimizer.py:714] -----------------------------------------------------------------
I 12-16 21:00:22 optimizer.py:714]  CLOUD   INSTANCE     vCPUs   ACCELERATORS   COST ($)   CHOSEN   
I 12-16 21:00:22 optimizer.py:714] -----------------------------------------------------------------
I 12-16 21:00:22 optimizer.py:714]  AWS     p3.2xlarge   8       V100:0.5       3.06          ✔     
I 12-16 21:00:22 optimizer.py:714] -----------------------------------------------------------------

Promote V100:0.5 to V100:1 and launch the VM

concretevitamin · 2022-12-17T05:16:41Z

Hmm, it's a tricky one. The intention to me reads like "I need a p3.2xlarge node, run this task on it with 0.5 V100 as requirements". That said, it's arguable this should be banned. What do you think about implementing this intention ("promote" 0.5 to 1 only in that optimizer display; for the actual task requirements, still use 0.5 for scheduling)?

Michaelvll

Thanks for the great effort @WoosukKwon! LGTM. I think it is ready to ship.

A kind bump for the previous comment: #975 (review)

what is the right behavior for sky launch --cloud aws --instance-type p3.2xlarge --gpus V100:0.5 ?

For this, I agree with @concretevitamin that we may want to show V100:1 in the optimizer's output but still schedule with the V100:0.5.
Btw, with the current branch, I got ValueError: Accelerator "V100" is not available in "eu-west-2". and the error is non-deterministic (changed to ValueError: Accelerator "V100" is not available in "us-east-2". in the second run).

sky/clouds/gcp.py

Michaelvll · 2022-12-17T09:47:45Z

sky/backends/cloud_vm_ray_backend.py

+        if to_provision.zone is None:
+            message = ('Failed to acquire resources in all zones in '
+                       f'{to_provision.region}. Try changing resource '
+                       'requirements or use another region.')


I wondered whether the optimizer will always fill in the region for the to_provision for the public clouds. If it is true, I think it should be fine to keep the current way (I could not think of an exception).
nit: It would be nice if we could comment it somewhere saying that the optimizer will always fill region for the public clouds.

WoosukKwon · 2022-12-26T02:33:54Z

Btw, with the current branch, I got ValueError: Accelerator "V100" is not available in "eu-west-2". and the error is non-deterministic (changed to ValueError: Accelerator "V100" is not available in "us-east-2". in the second run).

Thanks @Michaelvll for pointing this out. The non-deterministic behavior was because we returned Set[Region] in the regions_with_offerings method. When the price is the same for two regions, the optimizer's tie-breaking mechanism respects the order of the regions in the set, which can be non-deterministic. Thus, I changed the return type to List[Region] and made it preserve the order determined by common.py#get_region_zones(), which uses price -> region_str -> zone_str for the sorting key.

WoosukKwon · 2022-12-26T02:36:46Z

Hmm, it's a tricky one. The intention to me reads like "I need a p3.2xlarge node, run this task on it with 0.5 V100 as requirements". That said, it's arguable this should be banned. What do you think about implementing this intention ("promote" 0.5 to 1 only in that optimizer display; for the actual task requirements, still use 0.5 for scheduling)?

Thanks @concretevitamin for the suggestion. I think we need more discussions on this though; I personally lean toward forbidding such cases. For this PR, I followed the current master's behavior. Let's address the issue in another PR.

sky/resources.py

Michaelvll

Thank you for the great effort for the fine-grained optimizer @WoosukKwon! Left two nits. It should be good to go. : )

sky/backends/cloud_vm_ray_backend.py

sky/clouds/gcp.py

WoosukKwon added 5 commits July 15, 2022 00:22

Region/zone-based optimizer & provisioner

25af691

yapf

7071bb3

Fix messages

e47a934

Fix comments

926a781

Fix a comment

05029aa

WoosukKwon requested a review from Michaelvll July 15, 2022 08:10

WoosukKwon added 2 commits July 15, 2022 15:59

Fix version for backward compatibility

6fd509f

yapf

7f32764

concretevitamin reviewed Aug 11, 2022

View reviewed changes

WoosukKwon added 17 commits August 13, 2022 17:32

Merge branch 'master' into fine-grained-optimizer

09c2572

Minor fix

784bf92

Merge branch 'master' into fine-grained-optimizer

f05350b

yapf

fa5ba13

Merge branch 'master' into fine-grained-optimizer

4b11e37

yapf

89181cd

Add blanks back to docstrings

8a71fa5

Add more detailed docstrings for get_hourly_cost

9d4839a

Remove docstrings

7ff8976

Address review & Allow region == None

2ebe120

Minor fix

55ecfcf

Add type annotations

c14c175

Minor fix

c77ccbf

Merge branch 'master' into fine-grained-optimizer

bbd9a9b

Merge branch 'master' into fine-grained-optimizer

5150275

Merge branch 'master' into fine-grained-optimizer

ee90945

Merge branch 'master' into fine-grained-optimizer

2ae3043

WoosukKwon marked this pull request as draft November 21, 2022 22:57

Fix TPU error

82cf40c

WoosukKwon added 6 commits December 16, 2022 16:29

privision loop -> regions_with_offering

ee54ee6

Check if instance_type is None

a33e4e8

Fix a bug

7da4d82

yapf

056eefd

Add back the accelerator region check

57e4ca5

Merge branch 'master' into fine-grained-optimizer

984c8c2

concretevitamin approved these changes Dec 17, 2022

View reviewed changes

sky/backends/cloud_vm_ray_backend.py Outdated Show resolved Hide resolved

sky/resources.py Outdated Show resolved Hide resolved

WoosukKwon added 3 commits December 16, 2022 21:08

Fix err msg

d0c5b24

Roll back the change

3b572fd

Make sets

5b8e844

Michaelvll approved these changes Dec 17, 2022

View reviewed changes

WoosukKwon added 8 commits December 23, 2022 06:44

Merge branch 'master' into fine-grained-optimizer

45997e7

Merge branch 'master' into fine-grained-optimizer

8a64db3

Minor

3be9e4c

Set[Region] -> List[Region] for deterministic tie-breaking

9ed5d85

Enable the validation check for fractional accelerators

f49694f

yapf

3bfdd5d

Fix provisioner msg

ea79e31

Add a comment

f1a41cb

concretevitamin approved these changes Dec 26, 2022

View reviewed changes

sky/resources.py Show resolved Hide resolved

Michaelvll mentioned this pull request Dec 26, 2022

[WIP] Mypy for resources #1558

Closed

4 tasks

Michaelvll approved these changes Dec 28, 2022

View reviewed changes

sky/backends/cloud_vm_ray_backend.py Outdated Show resolved Hide resolved

sky/clouds/gcp.py Show resolved Hide resolved

WoosukKwon added 2 commits December 29, 2022 11:39

Merge branch 'master' into fine-grained-optimizer

5398b34

yapf

e3fa0f2

WoosukKwon merged commit 63d057b into master Dec 29, 2022

WoosukKwon deleted the fine-grained-optimizer branch January 9, 2023 07:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimizing & Provisioning Retries at the granularity of regions/zones #975

Optimizing & Provisioning Retries at the granularity of regions/zones #975

WoosukKwon commented Jul 15, 2022 •

edited

Loading

Michaelvll commented Jul 15, 2022

WoosukKwon commented Jul 15, 2022

concretevitamin left a comment

concretevitamin Aug 11, 2022

WoosukKwon Nov 23, 2022

WoosukKwon commented Dec 17, 2022

concretevitamin commented Dec 17, 2022

Michaelvll left a comment

Michaelvll Dec 17, 2022

WoosukKwon commented Dec 26, 2022

WoosukKwon commented Dec 26, 2022

Michaelvll left a comment

Optimizing & Provisioning Retries at the granularity of regions/zones #975

Optimizing & Provisioning Retries at the granularity of regions/zones #975

Conversation

WoosukKwon commented Jul 15, 2022 • edited Loading

Michaelvll commented Jul 15, 2022

WoosukKwon commented Jul 15, 2022

concretevitamin left a comment

Choose a reason for hiding this comment

concretevitamin Aug 11, 2022

Choose a reason for hiding this comment

WoosukKwon Nov 23, 2022

Choose a reason for hiding this comment

WoosukKwon commented Dec 17, 2022

concretevitamin commented Dec 17, 2022

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll Dec 17, 2022

Choose a reason for hiding this comment

WoosukKwon commented Dec 26, 2022

WoosukKwon commented Dec 26, 2022

Michaelvll left a comment

Choose a reason for hiding this comment

WoosukKwon commented Jul 15, 2022 •

edited

Loading