Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AWS] Add identity info when check fails #2456

Merged
merged 7 commits into from
Aug 24, 2023
+9 −6
Merged

[AWS] Add identity info when check fails #2456

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
feb1b18
[AWS] Add identity info when check fails
Michaelvll Aug 24, 2023
ae62f0c
More observebility
Michaelvll Aug 24, 2023
316b667
format
Michaelvll Aug 24, 2023
e40011a
Update sky/clouds/service_catalog/data_fetchers/fetch_aws.py
Michaelvll Aug 24, 2023
d82df7a
address comments
Michaelvll Aug 24, 2023
badd220
Merge branch 'add-identity-in-check' of github.com:skypilot-org/skypi…
Michaelvll Aug 24, 2023
016a74b
logging
Michaelvll Aug 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions sky/clouds/aws.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,7 @@ def check_credentials(cls) -> Tuple[bool, Optional[str]]:
# Checks if AWS credentials 1) exist and 2) are valid.
# https://stackoverflow.com/questions/53548737/verify-aws-credentials-with-boto3
try:
cls.get_current_user_identity()
identity = cls.get_current_user_identity()
except exceptions.CloudUserIdentityError as e:
return False, str(e)

Expand Down Expand Up @@ -486,9 +486,10 @@ def check_credentials(cls) -> Tuple[bool, Optional[str]]:
aws_catalog.get_default_instance_type()
except RuntimeError as e:
return False, (
'Failed to fetch the availability zones for the account. It is '
'likely due to permission issues, please check the minimal '
'permission required for AWS: https://skypilot.readthedocs.io/en/latest/cloud-setup/cloud-permissions/aws.html' # pylint: disable=
'Failed to fetch the availability zones for the account '
f'{identity}. It is likely due to permission issues, please '
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
f'{identity}. It is likely due to permission issues, please '
f'{identity!r}. It is likely due to permission issues, please '

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a list of strings. We probably don't want to add !r

'check the minimal permission required for AWS: '
'https://skypilot.readthedocs.io/en/latest/cloud-setup/cloud-permissions/aws.html' # pylint: disable=
f'\n{cls._INDENT_PREFIX}Details: '
f'{common_utils.format_exception(e, use_bracket=True)}')
return True, hints
Expand Down
7 changes: 5 additions & 2 deletions sky/clouds/service_catalog/data_fetchers/fetch_aws.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import pandas as pd

from sky.adaptors import aws
from sky.utils import common_utils
from sky.utils import ux_utils

# Enable most of the regions. Each user's account may have a subset of these
Expand Down Expand Up @@ -133,10 +134,12 @@ def _get_availability_zones(region: str) -> Optional[pd.DataFrame]:
elif e.response['Error']['Code'] == 'UnauthorizedOperation':
with ux_utils.print_exception_no_traceback():
raise RuntimeError(
'Failed to retrieve availability zone. '
'Failed to retrieve availability zones. '
'Please ensure that the `ec2:DescribeAvailabilityZones` '
'action is enabled for your AWS account in IAM. '
'Ref: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html' # pylint: disable=line-too-long
'Ref: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html.\n' # pylint: disable=line-too-long
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more detail on this raise would be good - currently just recieve the error: "Details: [botocore.exceptions.ClientError] An error occurred (UnauthorizedOperation) when calling the DescribeAvailabilityZones operation: You are not authorized to perform this operation."

despite: aws ec2 describe-availability-zones working fine.

Can you log the account at this point also?

Copy link
Collaborator Author

@Michaelvll Michaelvll Aug 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comment @TRT-BradleyB! The upper-level caller in the aws.py should log the identity information as modified above. Do you see that message when running sky check?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did think that was the intention, this is the extent of the message I recieve:

RuntimeError: Failed to retrieve availability zones. Please ensure that the `ec2:DescribeAvailabilityZones` action is enabled for your AWS account in IAM. Ref: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html.
Details: [botocore.exceptions.ClientError] An error occurred (UnauthorizedOperation) when calling the DescribeAvailabilityZones operation: You are not authorized to perform this operation.

Copy link
Collaborator Author

@Michaelvll Michaelvll Aug 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, this is weird. I did see the following messages when running sky check, where the first line of the Reason shows the account info:

$ sky check                                                     
Checking credentials to enable clouds for SkyPilot.
  Checking AWS...I 08-24 10:09:34 aws_catalog.py:79] Fetching availability zones mapping for AWS...
  AWS: disabled          
    Reason: Failed to fetch the availability zones for the account ['my-account', 'my-account']. It is likely due to permission issues, please check the minimal permission required for AWS: https://skypilot.readthedocs.io/en/latest/cloud-setup/cloud-permissions/aws.html
    Details: [builtins.RuntimeError] Failed to retrieve availability zones. Please ensure that the `ec2:DescribeAvailabilityZones` action is enabled for your AWS account in IAM. Ref: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html. Details: [botocore.exceptions.ClientError] ...

Just to confirm, is the error you shown from sky check?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah - no that's my bad I was running sky launch - check indeed has it.

Copy link
Collaborator Author

@Michaelvll Michaelvll Aug 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming! Merging this for now, and we can continue the discussion in #2451 for the availability zones permission issue. ; )

'Details: '
f'{common_utils.format_exception(e, use_bracket=True)}'
) from None
else:
raise
Expand Down