AWS client not picking up ECS credentials

[mjacksonw]

Environment:

• Vault Version: 0.8.0
• Operating System/Architecture: Alpine Linux in a docker container on ECS, running (more or less) from from the docker-vault image.

Vault Config File:

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 0
  tls_cert_file = "/cert/fullchain.pem"
  tls_key_file = "/cert/privkey.pem"
}

storage "dynamodb" {
  ha_enabled      = "true"
  region          = "us-east-1"
  table           = "vault-data"
  advertise_addr  = "https://{hostname}:8200"
  recovery_mode   = 1
}

Startup Log Output:

vault server -config /etc/vault/vault.conf 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x66a126]

goroutine 1 [running]:
net/http.(*Client).deadline(0x0, 0xc42000c198, 0xc420113620, 0x1)
	/goroot/src/net/http/client.go:186 +0x26
net/http.(*Client).Do(0x0, 0xc4203d6200, 0xc4204e61e8, 0xc4204e61e0, 0xc42049b8c0)
	/goroot/src/net/http/client.go:497 +0x89
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/corehandlers.sendFollowRedirects(0xc420029000, 0x1c18c18, 0xc420029000, 0xc4203d6100)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go:134 +0x3b
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/corehandlers.glob..func3(0xc420029000)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go:126 +0x85
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc420029190, 0xc420029000)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/request/handlers.go:195 +0x87
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc420029000, 0x0, 0x0)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:480 +0x191
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/credentials/endpointcreds.(*Provider).getCredentials(0xc42049bac0, 0xc42019c500, 0x7fe31afda000, 0x0)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/credentials/endpointcreds/provider.go:156 +0x12f
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/credentials/endpointcreds.(*Provider).Retrieve(0xc42049bac0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7fe31af8a988, ...)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/credentials/endpointcreds/provider.go:114 +0x5e
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/credentials.(*ChainProvider).Retrieve(0xc4204c7d70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x19b8ee0, ...)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/credentials/chain_provider.go:77 +0xc9
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/credentials.(*Credentials).Get(0xc42049fb60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/credentials/credentials.go:208 +0x13a
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/signer/v4.Signer.signWithBody(0xc42049fb60, 0x0, 0x27de660, 0xc42000c170, 0x10000, 0x1c1c078, 0x0, 0xc4203d6000, 0x27e7320, 0xc42019c4a0, ...)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:338 +0x259
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/signer/v4.signSDKRequestWithCurrTime(0xc4204b9c00, 0x1c1c078, 0x0, 0x0, 0x0)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:472 +0x2f4
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/signer/v4.SignSDKRequest(0xc4204b9c00)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:416 +0x52
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc4204b9d70, 0xc4204b9c00)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/request/handlers.go:195 +0x87
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/request.(*Request).Sign(0xc4204b9c00, 0x1c18c98, 0xc4204b9c00)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:337 +0xb0
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc4204b9c00, 0x0, 0x0)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:473 +0x13d
github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/service/dynamodb.(*DynamoDB).DescribeTable(0xc42000c180, 0xc42000c188, 0x0, 0x0, 0x0)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/aws/aws-sdk-go/service/dynamodb/api.go:922 +0x4d
github.com/hashicorp/vault/physical/dynamodb.ensureTableExists(0xc42000c180, 0xc4204b7b31, 0xa, 0x5, 0x5, 0xc42000c180, 0x0)
	/gopath/src/github.com/hashicorp/vault/physical/dynamodb/dynamodb.go:689 +0xc0
github.com/hashicorp/vault/physical/dynamodb.NewDynamoDBBackend(0xc4204c7b90, 0x27f8020, 0xc42049b800, 0x8, 0xc420068d78, 0x1, 0x0)
	/gopath/src/github.com/hashicorp/vault/physical/dynamodb/dynamodb.go:204 +0x56a
github.com/hashicorp/vault/command.(*ServerCommand).Run(0xc4204a0fc0, 0xc42000e160, 0x2, 0x2, 0x0)
	/gopath/src/github.com/hashicorp/vault/command/server.go:215 +0xcf6
github.com/hashicorp/vault/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc4204a0ea0, 0xc4204c6f60, 0x27, 0x1c18598)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/mitchellh/cli/cli.go:235 +0x2d1
github.com/hashicorp/vault/cli.RunCustom(0xc42000e150, 0x3, 0x3, 0xc4204c6f30, 0x0)
	/gopath/src/github.com/hashicorp/vault/cli/main.go:44 +0x4ea
github.com/hashicorp/vault/cli.Run(0xc42000e150, 0x3, 0x3, 0xc4200001a0)
	/gopath/src/github.com/hashicorp/vault/cli/main.go:11 +0x56
main.main()
	/gopath/src/github.com/hashicorp/vault/main.go:10 +0x64

Expected Behavior:
Vault (via the golang aws SDK, I suppose) should get AWS credentials from the environment, connect properly, and renew the credentials periodically as necessary.

Actual Behavior:
The server panics, fails to start, and crashes. Providing credentials manually either via the config file or the more traditional environment variables works fine.

Steps to Reproduce:
Start an ECS container and run the vault server against a DynamoDB storage backend. Do not make any effort to specify any AWS credentials.

Important Factoids:
This is in a docker container run via ECS. The AWS_CONTAINER_CREDENTIALS_RELATIVE_URI value is set, and when you pull it via curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, it returns the proper role and a set of credentials.

If you parse that payload and export them to environment variables, vault server will run properly, but it will fail as soon as those credentials expire (24h?), requiring the server to be restart, which means it'll be sealed. (I was hoping that the "reload via SIGHUP" might work for this in a pinch, but obviously no one's complained about AWS credentials needing a reload)

References:

• Incorrect AWS ID is used in ECS AWS auth #2979 – perhaps this is related in some way? It looks like the golang sdk might have trouble with the relative URI?
• Use RemoteCredProvider instead of EC2RoleProvider  #2983 – Seems like some work was just done on this front, though I'm not sure if/how this use case intersects with that.