Doesn't work with AWS SSO #229

otaviomedeirossb · 2020-07-23T02:04:09Z

I'm trying to make it work with AWS SSO but i'm getting NoCredentialProviders.

I used aws configure sso to create a profile on my local machine and i can manually docker login and pull images from ECR like this:

aws ecr get-login-password --region us-west-2 --profile my-profile | docker login --username AWS --password-stdin my-account-id.dkr.ecr.us-west-2.amazonaws.com

docker pull account-id.dkr.ecr.us-west-2.amazonaws.com/my-repo:latest

My ~/.aws/config looks something like:

[profile my-profile]
sso_start_url = https://my-company.awsapps.com/start
sso_region = us-east-1
sso_account_id = my-account-id
sso_role_name = MyRoleName
region = us-west-2
output = json

Nothing was added to ~/.aws/credentials.

I logged out docker, installed docker-credential-helper-ecr and set:

~/.docker/config.json

{
   "credsStore": "ecr-login"
}

and:

export AWS_PROFILE=my-profile
export AWS_REGION=us-west-2

When i:

docker pull account-id.dkr.ecr.us-west-2.amazonaws.com/repo:latest

I get:

no basic auth credentials

It works fine on another AWS account which is not SSO and i have my access keys set to ~/.aws/credentials.

Does it work with AWS SSO somehow?

The text was updated successfully, but these errors were encountered:

FernandoMiguel · 2020-07-23T16:35:44Z

i assume you did aws sso login ?

otaviomedeirossb · 2020-07-23T17:14:57Z

@FernandoMiguel yes i did and i can pull the images if i manually login to docker.

awilkins · 2020-10-07T13:15:20Z

Not everything can read the credential store that SSO uses, which is a bunch of JSON files in ~/.aws/sso/cache, but they contain the same stuff you'd get from any other sts:AssumeRole - access key id, secure access key, and session token - albeit encoded as a JWT.

Maybe try this small util I wrote that does an SSO login and copies the credentials into your "normal" ~/.aws/credentials file.

https://github.com/awilkins/aws-sso-refresh/

lachlancooper · 2020-10-14T06:41:56Z

The solution I use with other applications that do not yet natively support AWS SSO is an external credential_process, using a tool like https://github.com/benkehoe/aws-sso-credential-process

However, this doesn't currently work with amazon-ecr-credential-helper. In the logs I get an error like the following:

... error="ecr: Failed to get authorization token: ProcessProviderParseError: parse failed of credential_process output: <JSON REDACTED>\ncaused by: unexpected end of JSON input"

This is because the output from the credential process is longer than 1024 characters, but the max buffer size was only increased in aws-sdk-go v1.25.42.

Until proper SSO support is added here, it would be great to bump the SDK version to allow use of credential_process as a workaround.

overbit · 2020-10-20T08:23:40Z

This issue is becoming more important with the release of the latest version of docker v2.4.0. There seems to be a limit of 2500 characters on the auth token that makes the aws ecr get-login-password method not working (check here aws/aws-cli#5636).

rhertogh · 2020-12-04T16:59:52Z

This issue is becoming more important with the release of the latest version of docker v2.4.0. There seems to be a limit of 2500 characters on the auth token that makes the aws ecr get-login-password method not working (check here aws/aws-cli#5636).

It's not a solution in all situations but for those looking for a workaround for trying to push from a local computer the following might work:

Navigate to https://your-company.awsapps.com/start
Under the desired account select "Command line or programmatic access"
Copy the AWS environment vars for the desired operating system
Paste them in a console
Amazon-ecr-credential-helper will now work (at least it does for me 😁)

dougrday · 2020-12-04T17:17:15Z

For those that need to support many teams with this, I've created a temporary workaround credential helper.
https://github.com/dougrday/docker-credential-plaintext

It's working with our teams with AWS SSO, but it's definitely not the "secure" solution. Our tokens only live for 4 hours though, so the security window is fairly small.

I'd definitely prefer that other credential helpers step up and resolve the issue in a more secure fashion.

guilhermeheinen · 2020-12-28T12:13:29Z

@dougrday . Your solution is the only one that worked for me. I understand the risk of exposing the password for a short period of time. That said, it is the only workaround until other credential helpers solve this issue.
Thanks!

ericvilla · 2021-01-14T11:52:46Z

I @otaviomedeirossb, as already stated, AWS CLI v2 will write AWS SSO temporary credentials under ~/.aws/cli/cache folder, which in many cases ends up to a "credentials not found" issue. I'm currently working on an open source project that addresses this problem too. Indeed, it provides support to AWS SSO and lists you all the IAM Roles that your AWS SSO User can access. By clicking a session card associated with a IAM Role, Leapp will generate temporary credentials to access your AWS Organization's Account, inside ~/.aws/credentials file. Let me know if it makes sense to you and if you have questions about it.

TylerLubeck · 2021-01-15T18:50:31Z

I've also taken a stab at this, and ended up with something that doesn't rely on node and doesn't store temp credentials anywhere*. It's working well for me, and hopefully it's of some value to the rest of you as well.

https://github.com/TylerLubeck/docker-credentials-aws-ecr-sso

*Not to say that these things are bad, they just don't fit for my use case

flyinprogrammer · 2021-01-26T01:33:44Z

The solution I use with other applications that do not yet natively support AWS SSO is an external credential_process, using a tool like https://github.com/benkehoe/aws-sso-credential-process

However, this doesn't currently work with amazon-ecr-credential-helper. In the logs I get an error like the following:
... error="ecr: Failed to get authorization token: ProcessProviderParseError: parse failed of credential_process output: <JSON REDACTED>\ncaused by: unexpected end of JSON input"
This is because the output from the credential process is longer than 1024 characters, but the max buffer size was only increased in aws-sdk-go v1.25.42.

Until proper SSO support is added here, it would be great to bump the SDK version to allow use of credential_process as a workaround.

I can confirm this particular issue with credential_process not working with was fixed with this PR: #240 - however a version of amazon-ecr-credential-helper hasn't been released with this update.

As for native support of SSO I imagine this is stalled until something like this: aws/aws-sdk-go#3610 is merged to the SDK, then it will Just Work™ with another SDK upgrade because it seems it'll be part of the default profile credential parser.

Also, sidebar - it seems that Docker Desktop for Mac is overwriting /usr/local/bin/docker-credential-ecr-login when the application is started, which is seemingly terrible and and they should feel bad for doing that. So that's also a mess 🤸

masteinhauser · 2021-02-15T22:42:24Z

Now that aws/aws-sdk-go#3610 has merged and is released as part of aws-sdk-go v1.37.0+, when will this dependency be upgraded in the project? Currently it is v1.36.0, and while there are significant changes the aws-sdk-go CHANGELOG.md doesn't appear to immediately call out risks for this project.

This is one of the final tools we rely on to formally adopt the native AWS SSO credential protocol for us to deprecate our custom shim scripts.

stijndehaes · 2021-04-05T18:05:00Z

I was wondering if a release is planned in the next coming weeks :) This is also one of the last tools I need for AWS SSO adoption. For now I guess I will build it from source, but a release would be better!

tristanpemble · 2021-04-12T23:49:02Z

For anyone wandering in from Google, I can confirm that #267 fixes this for me after building from source.

shavo007 · 2021-07-06T04:24:12Z

hi @tristanpemble is there a new release of this with that fix or you have to build from source?

awilkins · 2021-07-06T11:12:17Z

Last release, Feb 15th, date of the MR that fixes this, Feb 22nd.

Yup, you still have to build from source.

shavo007 · 2021-07-06T11:37:01Z

thanks @awilkins sad

anyone here able to kick off a release and get it on brew?

jobirobi · 2021-07-12T16:09:56Z

Seconding (Nth-ing?) the request for a homebrew release with this change included 🙏

tristanpemble · 2021-08-18T22:36:18Z

we use Nix here. in the off chance that you are also using Nix, here is the overlay that we use:

self: super:
{
  amazon-ecr-credential-helper = super.amazon-ecr-credential-helper.overrideAttrs(old: {
    src = super.fetchFromGitHub {
      owner = "awslabs";
      repo = "amazon-ecr-credential-helper";
      rev = "efd1603705e91c31bf28cbbc5dba8c3e09c63100";
      sha256 = "sha256:0d2nc4rl0pl3mspvavxixa6lsjnnx5g05srpw1f68wqbj0vb7kd1";
    };
  });
}

kyledecot · 2021-09-05T23:09:04Z

This is still a problem for Homebrew users as the current formula is pinned at 0.5.0 (b19192b).

ghost · 2022-01-31T19:53:37Z

It looks like the Homebrew formula is updated now. In my case, I had to force-link it after installing, to overwrite the 0.5.0 version installed with Docker Desktop. I also had to clean up a bunch of pre-SSO config, but the most relevant things were:

Deleting old ECR cache: rm ~/.ecr/cache.json
Deleting old AWS CLI credentials: rm ~/.aws/credentials (not 100% sure if this is required, but it's not needed/used anymore with SSO)
Re-logging to establish new tokens in the expected places: aws sso login

After all of the above, I was able to painlessly pull images from ECR again.

gautam-nutalapati · 2022-09-16T15:24:05Z

I am using amazon-ecr-credential-helper for first time,
I installed 0.6.0 version via brew.
When I pull from private ECR repo, I see Error response from daemon: Head "https://***.dkr.ecr.us-east-1.amazonaws.com/v2/prisidio/service-base-image/manifests/latest": no basic auth credentials

What am I missing? 🤔
I use AWS SSO. Exporting correct profile, and sos sso login as well.

My dockerconfig is:

{
        "credsStore": "ecr-login",
        "credHelpers": {
          "public.ecr.aws": "ecr-login",
          "XXX.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
        }
}

brew info docker-credential-helper-ecr                                                                                                                                   main
==> docker-credential-helper-ecr: stable 0.6.0 (bottled)
Docker Credential Helper for Amazon ECR
https://github.com/awslabs/amazon-ecr-credential-helper
/usr/local/Cellar/docker-credential-helper-ecr/0.6.0 (7 files, 8.6MB)
  Poured from bottle on 2022-09-16 at 09:59:58
...

ejhayes · 2022-09-16T23:40:57Z

@gautam-nutalapati make sure that you are setting environment variable AWS_SDK_LOAD_CONFIG=1 and that your AWS profile has permissions to ECR. Everything else you are doing looks correct.

gautam-nutalapati · 2022-09-20T00:17:30Z

I tried this, but unfortunately same thing Error response from daemon: Head "https://651627600016.dkr.ecr.us-east-1.amazonaws.com/v2/prisidio/service-base-image/manifests/latest": no basic auth credentials
Same dockerconfig as above and my profile is admin profile, I validated list-images to ensure SSO profile is setup correctly, which is.

kzys · 2022-09-20T00:24:07Z

Let us take a look this week.

matuszeman · 2022-09-22T12:45:07Z

@gautam-nutalapati I had the same problem as you reported.
I fixed it by setting AWS profile via environment variable.

AWS_PROFILE=<your-profile> docker pull <ecr-id>.dkr.ecr.eu-central-1.amazonaws.com/<repo>

natenho · 2022-09-24T03:14:32Z

Same problem here, setting AWS_PROFILE= doesn't work aswell. I tried many different combinations and the credential helper did not work. no basic auth credentials

matuszeman · 2022-09-27T11:31:06Z

Does your IAM user have ecr:GetAuthorizationToken permission allowed?

natenho · 2022-10-01T23:38:15Z

@matuszeman yes, I have the permsision GetAuhtorizationToken permission.
I'm also able to perform pulls authenticating via aws ecr get-login-password | docker login, but not via credential helper =/

I noticed a log file inside .ecr folder with this line that might help:

time="2022-10-04T00:25:03-03:00" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

chaoqunya · 2023-02-02T06:33:43Z

This command works

❯ aws ecr get-login-password --region ap-east-1 --profile myprofile | docker login --username AWS --password-stdin XXX.dkr.ecr.ap-east-1.amazonaws.com
❯ docker push XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo:latest
The push refers to repository [XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo]
370e6XXXXXXX: Pushed 

latest: digest: sha256:XXXXXXXXXX size: 1987

But this don't, is there anyone has similar problem?

❯ AWS_PROFILE=myprofile docker push XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo:latest
The push refers to repository [XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo:latest
370e6XXXXXXX: Preparing 
no basic auth credentials
❯ cat ~/.ecr/log/ecr-login.log
time="2023-02-02T10:48:11+08:00" level=debug msg="Could not fetch credentials for cache prefix, disabling cache" error="the SSO session has expired or is invalid: open /Users/me/.aws/sso/cache/XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.json: no such file or directory"
time="2023-02-02T10:48:11+08:00" level=debug msg="Retrieving credentials" region=ap-east-1 registry=XXX serverURL=XXX.dkr.ecr.ap-east-1.amazonaws.com service=ecr
time="2023-02-02T10:48:11+08:00" level=debug msg="Calling ECR.GetAuthorizationToken" registry=XXX
time="2023-02-02T10:48:11+08:00" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: operation error ECR: GetAuthorizationToken, failed to sign request: failed to retrieve credentials: the SSO session has expired or is invalid: open /Users/me/.aws/sso/cache/XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.json: no such file or directory"

rchennault-exelixis · 2023-02-24T20:35:51Z

Yes. I can reproduce.

vtatarin · 2023-03-28T15:24:15Z

FYI, SSO-related issues seem to be gone in a new binary version (077b4a9), which can be built manually from the sources. Not sure why new releases are not posted for such a long time already

gondalez · 2023-03-29T01:25:27Z

FYI, SSO-related issues seem to be gone in a new binary version (077b4a9), which can be built manually from the sources. Not sure why new releases are not posted for such a long time already

Thanks for the heads-up @vtatarin. I've had a few issues with SSO bugs in my team.

@samuelkarp @austinvazquez I noticed you have changed CHANGELOG.md in the past... any insights as to why there are master commits but no releases past 0.6.0 for ?

I'd love to see the newer version with bugfixes released.
I assume that would allow it to make its way into a future Docker Desktop release as well for ultimate 😌

This is what I see locally; my credentials helper is packaged with with docker and pinned at 0.6.0:

» which docker-credential-ecr-login
/usr/local/bin/docker-credential-ecr-login

» ls -lah /usr/local/bin/docker-credential-ecr-login

lrwxr-xr-x  1 root  wheel    75B  1 Dec  2021 /usr/local/bin/docker-credential-ecr-login -> /Applications/Docker.app/Contents/Resources/bin/docker-credential-ecr-login

» docker-credential-ecr-login -v
amazon-ecr-credential-helper
Version:    0.6.0
Git commit: 69c85dc22db6511932bbf119e1a0cc5c90c69a7f

samuelkarp · 2023-03-29T02:39:37Z

@samuelkarp I noticed you have changed CHANGELOG.md in the past...

@gondalez I no longer work at Amazon.

adampl · 2023-04-14T14:20:35Z

Thanks! I've wasted half a day on trying to fix this, until I ended up here...

Installing from source did the job:

go install github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login@latest

austinvazquez · 2023-04-26T15:46:06Z

@gondalez thanks for the interest in the project and the feedback. As others have noted building from main will help the immediate pain. That said let me bring this up with the team to see about an updated release with these fixes.

austinvazquez · 2023-05-10T16:16:31Z

Special thanks to @fangn2 who worked to push this through. v0.7.0 is available with fix for the issue.

zchenyu · 2023-05-22T21:59:53Z

Ran into the same issue. Solved by installing from source per the README

go install github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login@latest

and also explicitly setting AWS_PROFILE.

Is there a way for the helper to use the default profile if AWS_PROFILE is not specified ? It's a bit annoying to manually specify the profile.

paminhoff · 2023-09-20T12:06:41Z

Works for one day once creds expire it breaks. Guess no longer supported properly by AWS. Back to unsecure ways of doing things until something stable comes along. Tried everything posted here including the profile and updating from go, rming files under .ecr and .aws/credentials

gopipal · 2024-03-21T03:49:00Z

@paminhoff I wonder if you need to do a docker logout "<aws-user-id>.dkr.ecr.<aws-cli-region>.amazonaws.com" every time the creds expire after a day/12 hrs(since docker can cache creds) before you login using aws sso login.

I am using the latest aws credentials helper and it works with aws sso

go install github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login@latest
sudo cp $GOPATH/bin /usr/bin/.

This is my ~/.docker/config.json

{
	"credHelpers": {
		"<aws-user-id>.dkr.ecr.<aws-cli-region>.amazonaws.com": "ecr-login"
	}
}

overbit mentioned this issue Oct 16, 2020

Docker login fails returning "The stub received bad data" on Windows when using awscli v2. aws/aws-cli#5636

Open

2 tasks

dougrday mentioned this issue Nov 6, 2020

Windows credential helper doesn't support passwords greater than 2,500 characters in length docker/docker-credential-helpers#190

Open

samuelkarp added contributions welcome enhancement labels Feb 15, 2021

lachlancooper mentioned this issue Feb 22, 2021

Update aws sdk to v1.37.32 #267

Merged

kzys closed this as completed in #267 Mar 24, 2021

oofnikj mentioned this issue Aug 10, 2021

New release that supports AWS SSO #290

Closed

kzys reopened this Sep 20, 2022

fangn2 mentioned this issue Apr 26, 2023

Update version to 0.7.0 #538

Merged

austinvazquez closed this as completed May 10, 2023

saugion mentioned this issue Nov 21, 2023

Error using SSO: The security token included in the request is invalid. #311

Open

Doesn't work with AWS SSO #229

Doesn't work with AWS SSO #229

Comments

otaviomedeirossb commented Jul 23, 2020

FernandoMiguel commented Jul 23, 2020

otaviomedeirossb commented Jul 23, 2020

awilkins commented Oct 7, 2020

lachlancooper commented Oct 14, 2020

overbit commented Oct 20, 2020

rhertogh commented Dec 4, 2020

dougrday commented Dec 4, 2020 • edited Loading

guilhermeheinen commented Dec 28, 2020

ericvilla commented Jan 14, 2021

TylerLubeck commented Jan 15, 2021

flyinprogrammer commented Jan 26, 2021 • edited Loading

masteinhauser commented Feb 15, 2021

stijndehaes commented Apr 5, 2021

tristanpemble commented Apr 12, 2021

shavo007 commented Jul 6, 2021

awilkins commented Jul 6, 2021

shavo007 commented Jul 6, 2021

jobirobi commented Jul 12, 2021

tristanpemble commented Aug 18, 2021

kyledecot commented Sep 5, 2021

ghost commented Jan 31, 2022

gautam-nutalapati commented Sep 16, 2022 • edited Loading

ejhayes commented Sep 16, 2022 • edited Loading

gautam-nutalapati commented Sep 20, 2022

kzys commented Sep 20, 2022

matuszeman commented Sep 22, 2022 • edited Loading

natenho commented Sep 24, 2022

matuszeman commented Sep 27, 2022

natenho commented Oct 1, 2022 • edited Loading

chaoqunya commented Feb 2, 2023 • edited Loading

rchennault-exelixis commented Feb 24, 2023

vtatarin commented Mar 28, 2023 • edited Loading

gondalez commented Mar 29, 2023

samuelkarp commented Mar 29, 2023

adampl commented Apr 14, 2023

austinvazquez commented Apr 26, 2023

austinvazquez commented May 10, 2023

zchenyu commented May 22, 2023

paminhoff commented Sep 20, 2023 • edited Loading

gopipal commented Mar 21, 2024 • edited Loading

dougrday commented Dec 4, 2020 •

edited

Loading

flyinprogrammer commented Jan 26, 2021 •

edited

Loading

gautam-nutalapati commented Sep 16, 2022 •

edited

Loading

ejhayes commented Sep 16, 2022 •

edited

Loading

matuszeman commented Sep 22, 2022 •

edited

Loading

natenho commented Oct 1, 2022 •

edited

Loading

chaoqunya commented Feb 2, 2023 •

edited

Loading

vtatarin commented Mar 28, 2023 •

edited

Loading

paminhoff commented Sep 20, 2023 •

edited

Loading

gopipal commented Mar 21, 2024 •

edited

Loading