Update credential precedence to match AWS CLI #621

omus · 2023-05-10T15:10:32Z

I noticed there were some credential precedence ordering differences between AWS.jl and AWS CLI. I ended up doing some experimentation with pairing different AWS CLI settings to determine the precedence ordering used by AWS CLI. Here are the results of those tests:

aws --profile used over env AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
aws --profile used over env AWS_PROFILE
env AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY used over env AWS_PROFILE
env AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY used over config file sso_*
config file sso_* used over ~/.aws/credentials (if exists)
~/.aws/credentials (if exists) used over config file credential_process
config file credential_process used over config file aws_access_key_id/aws_secret_access_key
config file aws_access_key_id/aws_secret_access_key used over EC2 instance metadata
config file aws_access_key_id/aws_secret_access_key used over AWS_CONTAINER_CREDENTIALS_FULL_URI

Using aws-cli/2.11.13 Python/3.11.3 Darwin/22.4.0 source/arm64 prompt/off

Notes:

Defining sso_account_id or sso_role_name in a profile without other sso_* keys results in an error about missing required configuration. Defining sso_start_url and sso_region by themselves doesn't produce this error.
Specifying the AWS credential file with AWS_SHARED_CREDENTIALS_FILE just replaces ~/.aws/credentials
Tested this by specifying bad credentials in one source and valid ones in the other. As I didn't have an SSO setup to test against I could only force these to fail.
Some additional testing was done to verify that the credential preference ordering is linear. I didn't find any examples of non-linear ordering.

omus · 2023-05-10T15:11:24Z

src/AWSCredentials.jl

-        elseif !isnothing(sso_start_url)
-            access_key, secret_key, token, expiry = _aws_get_sso_credential_details(p, ini)
-            return AWSCredentials(access_key, secret_key, token; expiry=expiry)


Potential breakage if someone was using dot_aws_config directly to get credentials for SSO support. If this is a concern I can leave this logic in place.

I think it's probably safe to remove, but I feel like we should keep it in place. Having the rug pulled under you would be quite annoying.

I think it's probably safe to remove, but I feel like we should keep it in place. Having the rug pulled under you would be quite annoying.
– #621 (comment)

I'll add this back in with either a deprecation or just a comment

omus · 2023-05-10T15:16:38Z

src/AWSCredentials.jl

+    if isfile(config_file)
+        ini = read(Inifile(), config_file)


We may want to refactor this credential code in the future to avoid reloading the config_file on each call. As this function and others are called in _aws_get_role we could see some improved performance with this change.

omus · 2023-05-10T15:35:36Z

test/AWSCredentials.jl

-                # The AWS CLI uses the config file `credential_process` setting over
-                # specifying the config file `aws_access_key_id`/`aws_secret_access_key`.
-                @testset "precedence" begin
-                    open(config_file, "w") do io
-                        write(
-                            io,
-                            """
-                            [profile $(test_values["Test-Config-Profile"])]
-                            aws_access_key_id = invalid
-                            aws_secret_access_key = invalid
-                            credential_process = $(abspath(credential_process_file))
-                            """,
-                        )
-                    end
-
-                    result = dot_aws_config(test_values["Test-Config-Profile"])


Diff is pretty messy but this was refactored and moved into the "Credential Precedence" testset

omus · 2023-05-10T15:43:45Z

bors try

bors · 2023-05-10T15:47:36Z

try

Build failed:

Julia 1 - ubuntu-latest - x64

test/AWSCredentials.jl

omus · 2023-05-10T18:34:37Z

bors try

bors · 2023-05-10T18:39:10Z

try

Build failed:

Julia 1 - ubuntu-latest - x64

omus · 2023-05-10T18:58:08Z

bors try

bors · 2023-05-10T19:05:46Z

try

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

src/AWSCredentials.jl

mattBrzezinski · 2023-05-10T19:47:20Z

src/AWSCredentials.jl

-        elseif !isnothing(sso_start_url)
-            access_key, secret_key, token, expiry = _aws_get_sso_credential_details(p, ini)
-            return AWSCredentials(access_key, secret_key, token; expiry=expiry)


I think it's probably safe to remove, but I feel like we should keep it in place. Having the rug pulled under you would be quite annoying.

omus · 2023-05-11T04:31:18Z

bors try

bors · 2023-05-11T04:36:16Z

try

Build failed:

Julia 1 - ubuntu-latest - x64

omus · 2023-05-11T14:01:25Z

bors try

bors · 2023-05-11T14:06:32Z

try

Build failed:

Julia 1 - ubuntu-latest - x64

omus · 2023-05-11T14:53:19Z

bors try

bors · 2023-05-11T15:01:32Z

try

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

omus · 2023-05-11T15:17:48Z

bors try

omus · 2023-05-11T15:22:30Z

Alright, should be done with making any more changes to this PR. Summary of changes since the last review:

Update AWS.jl documentation on credential precedence
Document deviation of having ECS credentials over EC2 credentials
Clean up docstrings
When attempting to connect to the ECS credential provider timeout quickly (service should be provided on the localhost)
Restore SSO support to dot_aws_config but with a deprecation warning. In practise the deprecation will only be seen if calling dot_aws_config directly.
Increase precedence of web token identity credentials (matches documentation) and add precedence test

bors · 2023-05-11T15:28:11Z

try

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

omus · 2023-05-11T18:40:42Z

bors r+

bors · 2023-05-11T18:55:16Z

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

omus added 8 commits May 10, 2023 09:52

Respect when users explicitly set the profile

6d933fd

Prefer SSO to credential files

497dfbe

Test credential files are preferred over credential_process

77d9548

Revise test ensuring credential_process is used over config creds

a466953

Revise docstrings

58e44c3

Comment on credential precedence testset

9a40d2b

Remove SSO support from dot_aws_config

eab2126

Set project version to 1.86.0

5e87387

omus commented May 10, 2023

View reviewed changes

Formatting

e13fca8

omus force-pushed the cv/precedence branch from 27e8b99 to e13fca8 Compare May 10, 2023 15:43

bors bot added a commit that referenced this pull request May 10, 2023

Try #621:

cd50001

omus self-assigned this May 10, 2023

github-actions bot reviewed May 10, 2023

View reviewed changes

test/AWSCredentials.jl Outdated Show resolved Hide resolved

test/AWSCredentials.jl Outdated Show resolved Hide resolved

test/AWSCredentials.jl Outdated Show resolved Hide resolved

omus added 3 commits May 10, 2023 13:34

Add EC2 instance credential precedence

962f34b

SSO fix

8d65dbd

Expiration date format

9483fb8

omus force-pushed the cv/precedence branch from 018f312 to 9483fb8 Compare May 10, 2023 18:34

bors bot added a commit that referenced this pull request May 10, 2023

Try #621:

47f5821

Fix source_profile traversal

67c7f3c

bors bot added a commit that referenced this pull request May 10, 2023

Try #621:

8b0f1e8

omus requested a review from mattBrzezinski May 10, 2023 19:08

mattBrzezinski approved these changes May 10, 2023

View reviewed changes

Fix mistake in precedence tests

4d98c4c

Deprecate SSO support in dot_aws_config

16bbf81

bors bot added a commit that referenced this pull request May 11, 2023

Try #621:

5a0ff4f

Revise ECS explainer comment

a549e41

bors bot added a commit that referenced this pull request May 11, 2023

Try #621:

73e1f65

Handle URI in patch with ConnectError

92a2c09

omus force-pushed the cv/precedence branch 2 times, most recently from bcb9319 to 380f7ba Compare May 11, 2023 14:16

omus added 4 commits May 11, 2023 09:21

Update docstrings for dot_aws_config/dot_aws_credentials

6fd5f61

Update documentation to use correct type

4f017a6

Update docstring on AWS.jl credential precedence

2d53308

Increase precedence of web identity creds

996821f

omus force-pushed the cv/precedence branch from 380f7ba to 996821f Compare May 11, 2023 14:52

bors bot added a commit that referenced this pull request May 11, 2023

Try #621:

700b4ab

Self-review

7b1a425

bors bot added a commit that referenced this pull request May 11, 2023

Try #621:

19c2480

omus mentioned this pull request May 11, 2023

Add integration with aws-vault for credentials #612

Closed

mattBrzezinski approved these changes May 11, 2023

View reviewed changes

bors bot merged commit 9a4322a into master May 11, 2023

bors bot deleted the cv/precedence branch May 11, 2023 18:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update credential precedence to match AWS CLI #621

Update credential precedence to match AWS CLI #621

omus commented May 10, 2023 •

edited

Loading

omus May 10, 2023 •

edited

Loading

mattBrzezinski May 10, 2023

omus May 10, 2023

omus May 10, 2023

omus May 10, 2023

omus May 10, 2023

omus commented May 10, 2023

bors bot commented May 10, 2023

omus commented May 10, 2023

bors bot commented May 10, 2023

omus commented May 10, 2023

bors bot commented May 10, 2023

mattBrzezinski May 10, 2023

omus commented May 11, 2023

bors bot commented May 11, 2023

omus commented May 11, 2023

bors bot commented May 11, 2023

omus commented May 11, 2023

bors bot commented May 11, 2023

omus commented May 11, 2023

omus commented May 11, 2023

bors bot commented May 11, 2023

omus commented May 11, 2023

bors bot commented May 11, 2023

Update credential precedence to match AWS CLI #621

Update credential precedence to match AWS CLI #621

Conversation

omus commented May 10, 2023 • edited Loading

omus May 10, 2023 • edited Loading

Choose a reason for hiding this comment

mattBrzezinski May 10, 2023

Choose a reason for hiding this comment

omus May 10, 2023

Choose a reason for hiding this comment

omus May 10, 2023

Choose a reason for hiding this comment

omus May 10, 2023

Choose a reason for hiding this comment

omus May 10, 2023

Choose a reason for hiding this comment

omus commented May 10, 2023

bors bot commented May 10, 2023

try

omus commented May 10, 2023

bors bot commented May 10, 2023

try

omus commented May 10, 2023

bors bot commented May 10, 2023

try

mattBrzezinski May 10, 2023

Choose a reason for hiding this comment

omus commented May 11, 2023

bors bot commented May 11, 2023

try

omus commented May 11, 2023

bors bot commented May 11, 2023

try

omus commented May 11, 2023

bors bot commented May 11, 2023

try

omus commented May 11, 2023

omus commented May 11, 2023

bors bot commented May 11, 2023

try

omus commented May 11, 2023

bors bot commented May 11, 2023

omus commented May 10, 2023 •

edited

Loading

omus May 10, 2023 •

edited

Loading