[version 1.42] No valid credential sources found for AWS Provider. #6320

shifres · 2018-11-01T16:32:28Z

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.10
+ provider.aws v1.42.0
+ provider.template v1.0.0

Affected Resource(s)

provider.aws

Terraform Configuration Files

provider "aws" {
  region = "${var.region}"
}

Output

terraform plan
.
.
.
module.bla.bla.bla.provider.aws: No valid credential sources found for AWS Provider.
    Please see https://terraform.io/docs/providers/aws/index.html for more information on
    providing credentials for the AWS Provider

Expected Behavior

Shouldn't get No valid credential sources found for AWS Provider. error

Steps to Reproduce

run terraform init --reconfigure to download the latest provider.aws version: provider.aws: version = "~> 1.42"

terraform plan

Important Factoids

I'm running terraform locally and using a ~/.aws/credentials file.
When I lock the aws.provider version to 1.41 everything runs properly.

provider "aws" {
  version = "1.41"
  region = "${var.region}"
}

The text was updated successfully, but these errors were encountered:

bflad · 2018-11-01T17:00:17Z

Hi @shifres 👋 Sorry you're running into trouble here.

It might be very helpful to turn on Terraform debug logging in your scenario (e.g. TF_LOG=debug terraform plan) to see if the underlying error(s) cannot be found instead of our less helpful error here.

There wasn't any intended changes to the provider and its authentication in this previous release so we may want to check the upstream AWS Go SDK changes to see if something occurred there as well. This one looks suspicious, but would also be very environment specific (two // in credentials): aws/aws-sdk-go#2239

shifres · 2018-11-01T17:18:50Z

Thanks for the quick response @bflad
This is the relevant output from the debug:

2018/11/01 13:09:33 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
.
.
.
2018/11/01 13:09:33 [DEBUG] [aws-sdk-go] <GetUserResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
  <GetUserResult>
    <User>
      <Path>/</Path>
      <UserName>XXXXXXX</UserName>
      <Arn>arn:aws:iam::<ACCOUNT_NUMBER>:user/XXXXXXX</Arn>
      <UserId>XXXXXXXXXXXX</UserId>
      <CreateDate>2018-10-23T21:33:39Z</CreateDate>
    </User>
  </GetUserResult>
  <ResponseMetadata>
    <RequestId>e71e4ac9-ddf8-11e8-89ee-df548ea8d37e</RequestId>
  </ResponseMetadata>
</GetUserResponse>
2018/11/01 13:09:33 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeAccountAttributes Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: ec2.us-east-1.amazonaws.com
User-Agent: aws-sdk-go/1.14.31 (go1.11.1; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10
Content-Length: 87
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXX/20181101/us-east-1/ec2/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=e4b3b0bdd2d97fa99385488ac9dc06e730ad3e31e32a1cd5fa60551fa5f7d06e
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20181101T170933Z
Accept-Encoding: gzip

Action=DescribeAccountAttributes&AttributeName.1=supported-platforms&Version=2016-11-15
-----------------------------------------------------
2018/11/01 13:09:33 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 403
Content-Type: text/xml
Date: Thu, 01 Nov 2018 17:09:32 GMT
X-Amzn-Requestid: e7280ea6-ddf8-11e8-b3fc-3ba677e6daf4
.
.
.
Action=GetCallerIdentity&Version=2011-06-15
-----------------------------------------------------
2018-11-01T13:09:33.005-0400 [DEBUG] plugin.terraform-provider-aws_v1.42.0_x4: 2018/11/01 13:09:33 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2018/11/01 13:09:33 [ERROR] root.spill.certificates_alternative_names: eval: *terraform.EvalConfigProvider, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.certificates_alternative_names: eval: *terraform.EvalSequence, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.certificates_alternative_names: eval: *terraform.EvalOpFilter, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.certificates_alternative_names: eval: *terraform.EvalSequence, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018-11-01T13:09:33.021-0400 [DEBUG] plugin.terraform-provider-aws_v1.42.0_x4: 2018/11/01 13:09:33 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2018/11/01 13:09:33 [ERROR] root.spill.elb: eval: *terraform.EvalConfigProvider, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.elb: eval: *terraform.EvalSequence, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.elb: eval: *terraform.EvalOpFilter, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.elb: eval: *terraform.EvalSequence, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [TRACE] [walkRefresh] Exiting eval tree: module.spill.module.elb.provider.aws
2018/11/01 13:09:33 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2018/11/01 13:09:33 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
2018/11/01 13:09:33 [INFO] Initializing DeviceFarm SDK connection
2018/11/01 13:09:33 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:

Let me know if there's anything further I can do in order to assist.

tdewitt · 2018-11-01T18:11:53Z

I can reproduce with 1.42.0 and it works as expected using 1.41.0. I'm running Linux. Coworkers using Apple laptops do not experience the same problem.

shifres · 2018-11-01T18:27:16Z

@tdewitt , can you reproduce the problem or you don't experience any problems with version 1.42 ?
I am using Apple btw.

tdewitt · 2018-11-01T18:42:05Z

On my Linux laptop, using version 1.42.0 of the aws provider plugin, I can reproduce.
On my Linux laptop, using version 1.41.0 of the aws provider plugin, I cannot reproduce and things work as expected.
A coworker, on his Apple laptop, using version 1.42.0 of the aws provider plugin, he cannot reproduce.

For reference, this is how the provider block is defined:

terraform {                                                                     
  required_version = ">= 0.11.0"

  backend "s3" {
    encrypt = "true"
    acl     = "bucket-owner-full-control"
  }
}

/**
 * Provider
 */

provider "aws" {
  version = "~> 1.3"
  region  = "${var.aws_region}"

  assume_role {
    role_arn = "${lookup(var.assume_roles, var.aws_account_alias)}"
  }
}

I changed the version to <= 1.41.0 to get working again
We use role assumption for everything

~/.aws/config:

[default]                                                                       
output=json

[s3]
calling_format = boto.s3.connection.OrdinaryCallingFormat

~/.aws/credentials:

[name]
aws_access_key_id = REDACTED
aws_secret_access_key = REDACTED # does have a / in the string

[name::dev]
source_profile = name
role_arn = arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME

bflad · 2018-11-01T18:48:49Z

It'll be super useful for this issue to know how credentials are being sourced (e.g. environment variables, ~/.aws/config, `~/.aws/credentials) -- thanks for sharing that information, @tdewitt!

Out of curiosity, does it help to wrap your secret access key in quotes?

[name]
aws_access_key_id = REDACTED
aws_secret_access_key = "REDACTED" # does have a / in the string

tdewitt · 2018-11-01T18:57:47Z

This is my workflow:

$ awsenv name
AWS environment is name
AWS_PROFILE=name
AWS_DEFAULT_REGION=us-west-2

# change to template dir
$ terraform init -backend-config=role_arn=arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME -backend-config region=us-west-2 -backend-config bucket=company-us-west-2-remote-state-develop -backend-config key=base/terraform.tfstate -backend-config dynamodb_table=us-west-2-remote-state-develop -lock=false

$ terraform plan -var-file ../../vars/variables.tfvars -var-file ../../vars/develop/shared.tfvars -var-file ../../vars/develop/default/shared.tfvars -var-file ../../vars/develop/default/us-west-2.tfvars -out out.us-west-2.tfstate

$ terraform apply out.us-west-2.tfstate

I just verified with the coworker who isn't having a problem (using an Apple) that he also has a forward slash in his secret key. Quoting the secret key doesn't change the behavior (still fails on my laptop).

bflad · 2018-11-01T19:55:05Z

Just providing for reference that we have heard about some odd behavior with secret access keys that contain /: #663 (comment)

Antidotally, I've heard a case internally about this as well, but since there was an easy workaround (rotating your credentials) I never personally spent any time diving further into it. 😅

shifres · 2018-11-01T20:04:38Z

I've changed the AWS_PROFILE env var to credentials that don't contain any special characters (like /, + etc...) and I still encounter the same problem.
I have no problem reading from the TF s3 backend (which requires valid credentials) but still encounter the error: No valid credential sources found for AWS Provider for the AWS provider.
My ~/.aws/config file contains:

[default]
output = json
region = us-east-1

and my ~/.aws/credentials file format is:

[name]
aws_access_key_id=<KEY>
aws_secret_access_key=<SECRET>

jhumbug · 2018-11-01T21:04:41Z

Getting the error as well for 1.42

Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: Error creating AWS session: SharedConfigLoadError: failed to load config file, /Users/****/.aws/credentials
caused by: INIParseError: invalid state with ASTKind {section_stmt {1 STRING 0 [97 119 115 45 97 100 117 108 116 115 119 105 109]} true []} and TokenType {4 NONE 0 [58]}

Manually setting the version to 1.41 works.

shifres · 2018-11-01T21:31:57Z

I see similar error to @jhumbug when I setup AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

bflad · 2018-11-01T22:05:10Z

I think TokenType {4 NONE 0 [58]} means its getting stuck on a : character. You could try wrapping that with quotes. Here's another bug reported upstream: aws/aws-sdk-go#2246

br4nd0n · 2018-11-01T22:41:56Z

I'm running this on a Mac laptop, high sierra 10.13.6, and I can reproduce this with AWS provider 1.42. AWS provider 1.41 works fine for me.

jritsema · 2018-11-02T16:19:21Z

@bflad in case it's helpful in resolving. We use https://github.com/turnerlabs/samlkeygen which populates our credentials file with temporary keys. For example:

[account1:devops]
aws_access_key_id = REDACTED
aws_secret_access_key = REDACTED
aws_session_token = REDACTED
aws_security_token = REDACTED
last_updated = 2018-11-02T13:48:32Z
expiration = 2018-11-02T22:48:32Z

mjreed-wbd · 2018-11-02T16:30:35Z

Based on the comments, the closed upstream bug only resolves the issue for colons in section headers, not in values. But creds file values can contain colons - not only timestamps like the ones above from samlkeygen, but also role ARNs (which show up when you use a credentials profile to assume a role from another profile, as in @tedewitt's post above).

jritsema · 2018-11-02T18:27:53Z

aws/aws-sdk-go#2247 (pending release) seems to fix our issue. Hopefully that makes it into a v1.42.1 or v1.43.0 release.

bflad · 2018-11-02T22:30:41Z

Version 1.43.0 of the AWS provider will include AWS Go SDK v1.15.68+, which contains both upstream INI parsing fixes. If there are still other problems with the parser, please continue to file them upstream. 👍

We're still tracking to release on our "normal" cadence middle of next week, but for visibility I'm actually going to keep this issue open until then even though the fixes may already be merged in case other issues emerge.

jritsema · 2018-11-02T22:42:58Z

Thanks @bflad for turning this around so quickly! And for all the awesome work you guys do!!!

bflad · 2018-11-07T23:00:43Z

This has been released in version 1.43.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

alienth · 2018-11-09T18:59:39Z

@bflad Still encountering this on 1.43.0. I have to pin to 1.41.0 to get my TF stuff to work. Symptom is the exact same:

* provider.aws: No valid credential sources found for AWS Provider.

My AWS credentials file contains : in values due to expiry time. Nothing else special going on there.

bflad · 2018-11-09T19:19:06Z

@alienth I would suggest opening a SDK issue upstream (similar to aws/aws-sdk-go#2246) and a new tracking issue here as well. 👍

bflad · 2018-11-09T19:22:09Z

FYI, just locking this issue to encourage fresh bug reports if there are any more. 😄

bflad added waiting-response Maintainers are waiting on response from community or contributor. provider Pertains to the provider itself, rather than any interaction with AWS. labels Nov 1, 2018

bflad removed the waiting-response Maintainers are waiting on response from community or contributor. label Nov 1, 2018

bflad added bug Addresses a defect in current functionality. upstream Addresses functionality related to the cloud provider. labels Nov 1, 2018

bflad mentioned this issue Nov 2, 2018

deps: aws/[email protected] #6330

Merged

bflad mentioned this issue Nov 2, 2018

deps: aws/[email protected] #6345

Merged

bflad added this to the v1.43.0 milestone Nov 2, 2018

bflad closed this as completed Nov 7, 2018

hashicorp locked and limited conversation to collaborators Nov 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[version 1.42] No valid credential sources found for AWS Provider. #6320

[version 1.42] No valid credential sources found for AWS Provider. #6320

shifres commented Nov 1, 2018 •

edited

Loading

bflad commented Nov 1, 2018

shifres commented Nov 1, 2018 •

edited

Loading

tdewitt commented Nov 1, 2018 •

edited

Loading

shifres commented Nov 1, 2018 •

edited

Loading

tdewitt commented Nov 1, 2018

bflad commented Nov 1, 2018

tdewitt commented Nov 1, 2018

bflad commented Nov 1, 2018

shifres commented Nov 1, 2018

jhumbug commented Nov 1, 2018

shifres commented Nov 1, 2018

bflad commented Nov 1, 2018

br4nd0n commented Nov 1, 2018

jritsema commented Nov 2, 2018 •

edited by breathingdust

Loading

mjreed-wbd commented Nov 2, 2018

jritsema commented Nov 2, 2018

bflad commented Nov 2, 2018

jritsema commented Nov 2, 2018

bflad commented Nov 7, 2018

alienth commented Nov 9, 2018 •

edited

Loading

bflad commented Nov 9, 2018

bflad commented Nov 9, 2018

[version 1.42] No valid credential sources found for AWS Provider. #6320

[version 1.42] No valid credential sources found for AWS Provider. #6320

Comments

shifres commented Nov 1, 2018 • edited Loading

Community Note

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Output

Expected Behavior

Steps to Reproduce

Important Factoids

bflad commented Nov 1, 2018

shifres commented Nov 1, 2018 • edited Loading

tdewitt commented Nov 1, 2018 • edited Loading

shifres commented Nov 1, 2018 • edited Loading

tdewitt commented Nov 1, 2018

bflad commented Nov 1, 2018

tdewitt commented Nov 1, 2018

bflad commented Nov 1, 2018

shifres commented Nov 1, 2018

jhumbug commented Nov 1, 2018

shifres commented Nov 1, 2018

bflad commented Nov 1, 2018

br4nd0n commented Nov 1, 2018

jritsema commented Nov 2, 2018 • edited by breathingdust Loading

mjreed-wbd commented Nov 2, 2018

jritsema commented Nov 2, 2018

bflad commented Nov 2, 2018

jritsema commented Nov 2, 2018

bflad commented Nov 7, 2018

alienth commented Nov 9, 2018 • edited Loading

bflad commented Nov 9, 2018

bflad commented Nov 9, 2018

shifres commented Nov 1, 2018 •

edited

Loading

shifres commented Nov 1, 2018 •

edited

Loading

tdewitt commented Nov 1, 2018 •

edited

Loading

shifres commented Nov 1, 2018 •

edited

Loading

jritsema commented Nov 2, 2018 •

edited by breathingdust

Loading

alienth commented Nov 9, 2018 •

edited

Loading