[AWS] Setting AWS credentials to another account in the env var through /etc/profile.d
will cause the cluster in INIT
mode
#2441
Labels
/etc/profile.d
will cause the cluster in INIT
mode
#2441
A user encountered this issue, where they set the AWS credentials through
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
using the/etc/profile.d
, and theirsky status -r
will always show the cluster inINIT
mode.To reproduce:
sky launch -c test-cred --cloud aws --cpus 4
ssh test-cred; sudo echo 'export AWS_ACCESS_KEY_ID=xxx; export AWS_SECRET_ACCESS_KEY=xxx' > /etc/profile.d/aws_keys.sh; sudo chmod +x /etc/profile.d/aws_keys.sh; ray stop
sky stop -y test-cred; sky launch -y -c test-cred
;sky status -r
After step 2, the ray autoscaler started with
sky launch
will use the wrong AWS credential, causing theray status
to show empty Healthy nodes: (reference: https://github.com/ray-project/ray/blob/d2fc4823126927b2c54f89ec72fa3d24b442e6a3/python/ray/autoscaler/_private/autoscaler.py#L396)That causes
sky status -r
's check withray status
fail to fetch the cluster status.The text was updated successfully, but these errors were encountered: