AWS S3 state backend fails with AWS SSO new profile format

[chiemerieezechukwu]

Terraform Version

Terraform v1.3.7
on darwin_arm64
+ provider registry.terraform.io/datadog/datadog v3.19.1
+ provider registry.terraform.io/hashicorp/aws v3.75.2
+ provider registry.terraform.io/hashicorp/null v3.2.1

Terraform Configuration Files

terraform {
  required_version = "~>1.0"

  required_providers {
    aws = {
      version = "~>3.0"
      source  = "hashicorp/aws"
    }
    null = {
      version = "~>3.0"
      source  = "hashicorp/null"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.18"
    }
  }

  backend "s3" {
    region         = "eu-central-1"
    bucket         = "xxxx-bucket-xxxx"
    key            = "state-key"
    dynamodb_table = "lock-table"
    encrypt        = "true"
  }
}

Debug Output

...
2023-01-05T17:27:57.944+0100 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2023-01-05T17:27:57.945+0100 [INFO]  CLI command args: []string{"apply"}
2023-01-05T17:27:57.954+0100 [TRACE] Meta.Backend: built configuration for "s3" backend with hash value 3591387704
2023-01-05T17:27:57.955+0100 [TRACE] Preserving existing state lineage "8e30792b-97ac-5674-d17a-955b1985e845"
2023-01-05T17:27:57.955+0100 [TRACE] Preserving existing state lineage "8e30792b-97ac-5674-d17a-955b1985e845"
2023-01-05T17:27:57.956+0100 [TRACE] Meta.Backend: working directory was previously initialized for "s3" backend
2023-01-05T17:27:57.956+0100 [TRACE] Meta.Backend: using already-initialized, unchanged "s3" backend configuration
2023-01-05T17:27:57.959+0100 [INFO]  Attempting to use session-derived credentials
╷
│ Error: error configuring S3 Backend: Error creating AWS session: profile "xxx" is configured to use SSO but is missing required configuration: sso_region, sso_start_url
│ 
│ 
╵

Expected Behavior

Terraform should work with the new AWS config format

Actual Behavior

Terraform throws error with the new AWS config format

Steps to Reproduce

1. Configure S3 remote state backend (including creating the bucket, making sure your AWS permissions are sufficient, etc.)
2. Configure your AWS client to use AWS IAM Identity Center (SSO) new format. Example ~/.aws/config entry:

[profile xxx]
sso_session = my-sso
sso_account_id = 111122223333
sso_role_name = SampleRole

[sso-session my-sso]
sso_region = us-east-1
sso_start_url = https://my-sso-portal.awsapps.com/start
sso_registration_scopes = sso:account:access

reference: see here

3. Login with AWS_PROFILE=xxx aws sso login
4. confirm that your bucket is accessible (e.g. AWS_PROFILE=xxx AWS_REGION=yyy aws s3 ls <your-bucket-name>)
5. Run e.g. terraform init, and receive the error message above.

Additional Context

No response

References

No response

[crw]

Thanks for the report, I'll notify the AWS provider team.

[simon97k]

Any updates on this?

[jlarfors]

In the meantime, a workaround I found is to use the legacy non-refreshable configuration as per the AWS docs: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html#sso-configure-profile-legacy

I created my profile manually according to the above docs, which now contains the sso_region and sso_start_url fields, and ran aws sso login --profile <legacy-profile-name> and can now use my S3 backend.

Hopefully this gets fixed soon!

[jparta]

An issue exists since early December: hashicorp/terraform-provider-aws#28263

[tonyhhyip]

The sso_session is not supported by AWS Go SDK v1, which is used by Terraform. To fix this issue, it would require migrating to v2 SDK.

[omerfsen]

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html#sso-configure-profile-legacy

I manually export

and it works.. Though I need to do it every 8 hours ;) manually.

[omerfsen]

hashicorp/terraform-provider-aws#28263 (comment) provides a workaround with S3 backend for now..

[pjaudiomv]

hashicorp/terraform-provider-aws#28263 (comment) provides a workaround with S3 backend for now..

yea doesn't help when you use many aws accounts though or a different aws account for backend vs provider :(. you have to login to every account separately vs just one with session chaining.

[lucasjrt]

@crw this is not a problem on AWS provider, but on Terraform core, as it is from S3 remote state.

The problem is located here:

terraform/internal/backend/remote-state/s3/backend.go

Lines 393 to 396 in 984d8c1

	sess, err := awsbase.GetSession(cfg)
	if err != nil {
	return fmt.Errorf("error configuring S3 Backend: %w", err)
	}

It is a function call to the code from aws-sdk-go-base HashiCorp module version v0.7.1 (see go.mod), which is a version released in 2021. Version 2.0.0 is in beta test, so it is not that easy to update the code as there is no stable release on this new version and it is not so simple to release a v0.7.2 just to fix this issue.

To fix this, it is needed to understand when is it going to have a stable release from version 2.0.0 of the module aws-sdk-go-base, and only then fix this issue on Terraform repository itself.

It is not necessary to update Terraform AWS SDK version to V2 itself, only updating to the newest version of the aws-sdk-go-base module should already fix the problem.

[crw]

@lucasjrt The AWS Provider team supports the S3 Backend. See: https://github.com/hashicorp/terraform/blob/main/CODEOWNERS

[Fotkurz]

In the meantime, a workaround I found is to use the legacy non-refreshable configuration as per the AWS docs: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html#sso-configure-profile-legacy

I created my profile manually according to the above docs, which now contains the sso_region and sso_start_url fields, and ran aws sso login --profile <legacy-profile-name> and can now use my S3 backend.

Hopefully this gets fixed soon!

This fixed the issue for me. Setting manually the sso session on .aws/config and removing the sso_session block.

[BOPOHA]

The sso_session is not supported by AWS Go SDK v1, which is used by Terraform. To fix this issue, it would require migrating to v2 SDK.

thanks. it is the most important comment in this thread!

[full-stack-devops]

I experience the same issue. I do not see a security flawed workaround as a long-term viable solution for production workloads.

[pnickerson-cashstar]

I tried #32465 (comment) to work around the issue.

After doing that, I got this error:

│ Error: SSOProviderInvalidToken: the SSO session has expired or is invalid
│ caused by: open /home/[username]/.aws/sso/cache/abc123.json: no such file or directory

The fix was to edit .aws/config again, and remove the sso_session line from the profile's section.

[michelleparent]

Any updates here?

[cookiesowns]

Getting this fixed will be very beneficial to the rest of the terraform community as it means we can standardize using the AWS SSO token provider configuration, instead of the legacy format.

[kneemaa]

I found a temporary workaround. I went back and manually added the sso_start_url and sso_region to the profile i needed to use

original

[sso-session mySession]
sso_start_url = https://SSO_DOMAIN/start#/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[default]
sso_session = mySession
sso_account_id = xxxxxxxxxxxxxx
sso_role_name = aws-sso-non-admin
region = us-east-2
output = json

updated

[sso-session mySession]
sso_start_url = https://SSO_DOMAIN/start#/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[default]
sso_session = mySession
sso_start_url = https://SSO_DOMAIN/start#/     <<< Add
sso_region = us-east-1                                         <<< Add
sso_account_id = xxxxxxxxxxxxxx
sso_role_name = aws-sso-non-admin
region = us-east-2
output = json

[LatinRickshaw]

I found a temporary workaround. I went back and manually added the sso_start_url and sso_region to the profile i needed to use

original

[sso-session mySession]
sso_start_url = https://SSO_DOMAIN/start#/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[default]
sso_session = mySession
sso_account_id = xxxxxxxxxxxxxx
sso_role_name = aws-sso-non-admin
region = us-east-2
output = json

updated

[sso-session mySession]
sso_start_url = https://SSO_DOMAIN/start#/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[default]
sso_session = mySession
sso_start_url = https://SSO_DOMAIN/start#/     <<< Add
sso_region = us-east-1                                         <<< Add
sso_account_id = xxxxxxxxxxxxxx
sso_role_name = aws-sso-non-admin
region = us-east-2
output = json

I also had to remove sso-session from my target profile.

[jurski-eu]

It solved my problem - thanks!

P.S.
But still sounds like a „dirty” workaround 🤪

[flokli]

The workaround entirely removes usage of the sso_session key and the whole [sso-session foo] section, which is the old AWS SSO profile format.

This issue still is about adding support for the new format, which is still very much valid.

[arianvp]

Note that only the new SSO profile format supports automatic access key rotation, whilst the old one doesn't and always creates an access_token that expires after 8 hours. This is quite annoying and makes using the old format not an option for us

https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-legacy.html
https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html

[holms]

Is this really still an issue? SSO is official way in AWS to use programmatic access to AWS, even terraform docs from AWS tutor redirected me to SSO steps when creating first user in IAC

[Carlos4ndresh]

I had this issue just today, this with provider v4, I'll test with provider v5; or maybe it's not related to the provider?

[pjaudiomv]

I had this issue just today, this with provider v4, I'll test with provider v5; or maybe it's not related to the provider?

provider v4.54.0 and later should work. the issue is the s3 backend does not support session tokens. @hashicorp/terraform-aws is there a plan to fix this or is it on the road map?

[Carlos4ndresh]

I had this issue just today, this with provider v4, I'll test with provider v5; or maybe it's not related to the provider?

provider v4.54.0 and later should work. the issue is the s3 backend does not support session tokens. @hashicorp/terraform-aws is there a plan to fix this or is it on the road map?

Yeah, I just checked latest TF and still happening, it doesn't even reach the stage for provider installation.

[AMMullan]

I had this issue just today, this with provider v4, I'll test with provider v5; or maybe it's not related to the provider?

provider v4.54.0 and later should work. the issue is the s3 backend does not support session tokens. @hashicorp/terraform-aws is there a plan to fix this or is it on the road map?

Yeah, I just checked latest TF and still happening, it doesn't even reach the stage for provider installation.

Agree with @Carlos4ndresh - this has nothing to do with the provider if the provider isn't even grabbed by the point the error occurs. Doing a terraform init pulls in the module and then fails immediately.

[mmerickel]

Confirmed this is still an issue on 1.5.2 - it is a problem with the S3 backend built in to the terraform process - not the AWS provider.

[RafaelClovr]

Hi, I have opened a related issue here: #33431

[mmerickel]

Per hashicorp/terraform-provider-aws#28263 (comment)

The support for sso-session for aws-sdk-go has been merged and is now in a release.
https://github.com/aws/aws-sdk-go/releases/tag/v1.44.298

so hopefully the new sdk can get merged into the next terraform release.

[et304383]

@jbardin what is the ETA on Hashicorp pushing a new stable Terraform version out that uses the updated Go SDK? I don't even see mention to this in the 1.6 alpha releases.

[crw]

@et304383 See #33469 (comment), although this was posted in July there have been multiple S3 backend PRs merged since this comment, so I believe this work is underway. Per that comment, it will take a few releases to fully upgrade. Thanks!

[PeterBocan]

Hey Peeps, is there any official doc around this and/if we can expect this to be fixed? I now need to work with TF with a different AWS profile which makes it a bit awkward.

[pjaudiomv]

Hey Peeps, is there any official doc around this and/if we can expect this to be fixed? I now need to work with TF with a different AWS profile which makes it a bit awkward.

The fix is in 1.6.0-beta1

[dancorne]

Hey Peeps, is there any official doc around this and/if we can expect this to be fixed? I now need to work with TF with a different AWS profile which makes it a bit awkward.

Our workaround has been using aws-vault (v7+) and initiating with eg aws-vault exec $profile_name -- terraform apply.

Native support would be nicer ofc...

[mbokman-e2open]

A previous comment stated that the fix is available in 1.6.0-beta1, which for me was not the case. For me, this has started working per 1.6.0-beta3, running on a Mac which I doubt matters.

[kevinkopf]

I can confirm everything works after upgrading to Terraform v1.6.0-rc1

[toadjaune]

I can confirm everything works fine on the released stable 1.6.0 ! 🎉

This issue can probably be closed now :)

[charltonstanley]

If you're still running into this as I was even after updating to 1.6+, check that your aws provider is greater than or equal to 4.54.0. Mine was at 4.53.0 and it wasn't working. However, it seems to be fixed starting in 4.54.0, but it's (strangely) not mentioned the release notes for the provider: https://github.com/hashicorp/terraform-provider-aws/releases/tag/v4.54.0

[alejlatorre]

In the meantime, a workaround I found is to use the legacy non-refreshable configuration as per the AWS docs: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html#sso-configure-profile-legacy

I created my profile manually according to the above docs, which now contains the sso_region and sso_start_url fields, and ran aws sso login --profile <legacy-profile-name> and can now use my S3 backend.

Hopefully this gets fixed soon!

This helped me

[dongho-jung]

If you're still running into this as I was even after updating to 1.6+, check that your aws provider is greater than or equal to 4.54.0. Mine was at 4.53.0 and it wasn't working. However, it seems to be fixed starting in 4.54.0, but it's (strangely) not mentioned the release notes for the provider: https://github.com/hashicorp/terraform-provider-aws/releases/tag/v4.54.0

this saved my day 🙇

⚠️ check that your aws provider is greater than or equal to 4.54.0 ⚠️

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.