Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Serve] --purge is not respected for downing Serve controller #3061

Closed
romilbhardwaj opened this issue Jan 31, 2024 · 0 comments · Fixed by #3094
Closed

[Serve] --purge is not respected for downing Serve controller #3061

romilbhardwaj opened this issue Jan 31, 2024 · 0 comments · Fixed by #3094

Comments

@romilbhardwaj
Copy link
Collaborator

Trying to delete the sky serve controller with sky down <controller> --purge from a changed identity results in a hard ClusterOwnerIdentityMismatchError and the cluster is not purged.

Probably need to update _hint_or_raise_for_down_sky_serve_controller to respect --purge.

This is for a serve controller running on kubernetes (on my local dev branch):

$ sky down sky-serve-controller-2ea485ea --purge
D 01-31 13:12:42 skypilot_config.py:139] Using config path: /Users/romilb/.sky/config.yaml
D 01-31 13:12:42 skypilot_config.py:143] Config loaded:
D 01-31 13:12:42 skypilot_config.py:143] {'serve': {'controller': {'resources': {'cloud': 'kubernetes'}}}}
D 01-31 13:12:42 skypilot_config.py:153] Config syntax check passed.
Traceback (most recent call last):
  File "/Users/romilb/tools/anaconda3/bin/sky", line 8, in <module>
    sys.exit(cli())
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 316, in _record
    return f(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 1222, in invoke
    return super().invoke(ctx)
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 337, in _record
    return f(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 2942, in down
    _down_or_stop_clusters(clusters,
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 3128, in _down_or_stop_clusters
    hint_or_raise(controller_name)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 3003, in _hint_or_raise_for_down_sky_serve_controller
    cluster_status, _ = backend_utils.refresh_cluster_status_handle(
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 337, in _record
    return f(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/backends/backend_utils.py", line 2099, in refresh_cluster_status_handle
    record = refresh_cluster_record(
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/backends/backend_utils.py", line 2069, in refresh_cluster_record
    check_owner_identity(cluster_name)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/backends/backend_utils.py", line 1571, in check_owner_identity
    raise exceptions.ClusterOwnerIdentityMismatchError(
sky.exceptions.ClusterOwnerIdentityMismatchError: 'sky-serve-controller-2ea485ea' (Kubernetes) is owned by account ['gke_skypilot-375900_us-central1-c_gkeusc2_gke_skypilot-375900_us-central1-c_gkeusc2_default'], but the activated account is ['gke_skypilot-375900_us-central1-c_gkeusc2_gke_skypilot-375900_us-central1-c_gkeusc2-skypilot-sa_default'].
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant