Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JuiceFS connection to s3 delay due to missing IMDSv2 in EKS cluster #5222

Closed
cuichenli opened this issue Oct 11, 2024 · 4 comments · Fixed by #5231
Closed

JuiceFS connection to s3 delay due to missing IMDSv2 in EKS cluster #5222

cuichenli opened this issue Oct 11, 2024 · 4 comments · Fixed by #5231
Assignees

Comments

@cuichenli
Copy link

What happened:If the EKS cluster does not have IMDSv2 enabled, JuiceFS may experience a delay of approximately 4 minutes before successfully connecting to S3. To clarify, if a node in the cluster is not compatible with IMDSv2, the AWS client will attempt to connect to the V2 metadata store using commands similar to curl -X PUT "http://169.254.169.254/latest/api/token"

However, this connection will time out only after 4 minutes. During this time, the JuiceFS client will hang without providing any error message, which can be confusing, even though the connection will eventually succeed.

What you expected to happen: maybe detect if the cluster is IMDSv2 compatiable before trying to get the credentials, or let it fail quicker so user would not wait for 4 minutes.

How to reproduce it (as minimally and precisely as possible): create one eks cluster with IMDSv2 disabled

@cuichenli cuichenli added the kind/bug Something isn't working label Oct 11, 2024
@davies
Copy link
Contributor

davies commented Oct 11, 2024

We may need to upgrade the aws sdk

@zhijian-pro zhijian-pro self-assigned this Oct 12, 2024
@jiefenghuang
Copy link
Contributor

ref: aws/aws-sdk-go#3066 aws/aws-sdk-go#2972
our custom client overwrite that timeout

@zhijian-pro
Copy link
Contributor

@cuichenli
What is the metadata response hop limit setting for your cluster ?
According to the official document, the recommended setting is 2 . You can try this setting to see if it works.
In addition, when we create the environment, we can not completely disable v2. There are only two options,only use v2 or allow v1 & v2. so we can not completely simulate the situation of disabling v2. What is your configuration?

@zhijian-pro
Copy link
Contributor

@cuichenli Can you test if this pull request #5231 solves your problem ?

@zhijian-pro zhijian-pro reopened this Oct 21, 2024
@zhijian-pro zhijian-pro removed the kind/bug Something isn't working label Oct 22, 2024
@davies davies closed this as completed Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants