Refactor failover when resources have some unsupported features on certain cloud #2245
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cblmemo
added a commit
to cblmemo/skypilot
that referenced
this pull request
Jul 16, 2023
Michaelvll
approved these changes
Jul 17, 2023
There was a problem hiding this comment.
This is awesome @cblmemo! We have been thinking of having this kind of refactoring for a while. LGTM.
cblmemo
added a commit
that referenced
this pull request
Aug 1, 2023
* finished AWS & GCP, test passed on GCP * GCP now create rules with cluster-specified tags * Creating new SGs on AWS and configuring user specified rules * new SG name * move to resources & allow tcp only * disable request of newer ports in task.yaml * lint * delete firewall rules & aws SGs after teardown * add smoke test & cluster name length limit * format * add doc * add error when not using aws & gcp * add failover * remove redundant length checking * temporary remove failover and wait for #2245 * move to new provisioner api * add unsupported feature * add ports to resources.get_required_cloud_features * use CloudImplementationFeatures to validate port * remove assert for interrupted launching process * renaming provision package * AWS: only create new SG when port is specified * nit: variable name * remove redundant error message since check is done in resources validation * support port range * refactor GCP implementation * remove unused function * rename gcp rule name to avoid dependency and name too long * remove redundant tags * remove redundant silent argument * change cluster name hash to truncate + hash * format * remove create firewall wait function * rename hash_cluster_name to truncate_and_hash_cluster_name * fix error when launching TPU node * disable TPU with ports for now * format * enable tpu nodes * enable tpu vm * raise error instead of assert * Update sky/skylet/providers/gcp/config.py Co-authored-by: Zhanghao Wu <[email protected]> --------- Co-authored-by: Zhanghao Wu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The failover function in the optimizer helps us get rid of invalid resources for a certain cloud. Say if we specify
--disk-tier highwhensky launch, we wanted the optimizer only pick clouds that support the customizeddisk_tierfeature.Previously, we have to manually modify
_get_feasible_launchable_resourcesfunction in all cloud to handle the special case: return an empty list to the optimizer when one unsupported feature is used, so that the optimizer will skip this cloud since no valid resources are found. This PR refactors the implementation of this function, by describing unsupported features in resources withclouds.CloudImplementationFeatures, and adding a check before we step in cloud-specific_get_feasible_launchable_resourcesfunctions.After this PR is merged, #1910 and #2210 could have similar implementations when docker/ports are specified.
Tested (run the relevant ones):
bash format.shsky launch --disk-tier highand only AWS & GCP are listedsky launch --disk-tier mediumand only AWS & GCP & Azure are listedsky launch --disk-tier lowand only AWS & GCP & Azure are listedpytest tests/test_smoke.pypytest tests/test_smoke.py::test_fill_in_the_namebash tests/backward_comaptibility_tests.sh