data sources that generate temporary credentials should not persist values in plan tfstate

[llamahunter]

Terraform Version

Terraform v0.12.24

Terraform Configuration Files

data "aws_eks_cluster" "example" {
  name = "example"
}

data "aws_eks_cluster_auth" "example" {
  name = "example"
}

provider "kubernetes" {
  host                   = "${data.aws_eks_cluster.example.endpoint}"
  cluster_ca_certificate = "${base64decode(data.aws_eks_cluster.example.certificate_authority.0.data)}"
  token                  = "${data.aws_eks_cluster_auth.example.token}"
  load_config_file       = false
}

Debug Output

From terraform apply:

Error: Unauthorized

  on .terraform/modules/prometheus_operator/modules/prometheus-operator/main.tf line 36, in resource "kubernetes_namespace" "this":
  36: resource "kubernetes_namespace" "this" {

From EKS authorization log:

time="2020-05-06T05:17:59Z" level=warning msg="access denied" client="127.0.0.1:55512" error="input token was not properly formatted: X-Amz-Date parameter is expired (15 minute expiration) 2020-05-06 01:09:00 +0000 UTC" method=POST path=/authenticate

Note the very stale X-Amz-Date token (2020-05-06 01:09:00Z) relative to the current time (2020-05-06T05:17:59Z). The token date corresponds to the time at which the terraform plan was run, several hours earlier.

Crash Output

Expected Behavior

data.aws_eks_cluster_auth.example.token should be refreshed on apply. Authentication tokens should not be cached as part of the plan.

Actual Behavior

data.aws_eks_cluster_auth.example.token is cached in the plan, and attempted to be reused later on apply. But, tokens only have validity for 15 minutes.

Steps to Reproduce

1. terraform plan and save plan output
2. wait more than 15 minutes
3. terraform apply existing cached plan

Additional Context

EKS clusters get their authentication tokens via a backdoor mechanism with AWS IAM. The current AWS IAM role is used to communicate with a service in the EKS cluster to obtain a K8S token that can be used for only 15 minutes to access the cluster. The aws_eks_cluster_auth data source is the way to get this token in terraform. It should not be cached as part of the plan, but regenerated on every terraform call.

References

I filed the below issue on the terraform-provider-aws project, but they said this is actually a shortcoming of the Terraform Core.

• aws_eks_cluster_auth should not cache token in plan terraform-provider-aws#13189

[gblues]

This is biting us in the butt (also AWS EKS)--the kubernetes auth token is generated at the very start of the planning process, so when the plan process takes >15 minutes, the apply process fails.

[mnowrot]

Is there any way to influence this 15 minutes timeout? (Perhaps from the AWS EKS side?)

[gblues]

There isn't.

However, what you can do (and what we have done to work around this) is to wrap all kubernetes operations in helm charts, because the Helm provider can refresh credentials at execute time and therefore the plan can be executed at any point (assuming no state changes occur that would invalidate the plan in the meantime).

Thanks to @bear454 for demonstrating this workaround in the SUSE/cap-terraform PR that Github has helpfully linked above.

[llamahunter]

That kind of defeats the purpose of using the kubernetes terraform provider to track resources.

[bear454]

Well, if it's broken, sometimes you have to work around it.

[dschunack]

Is anyone working on this ?

[llamahunter]

anyone addressing this? It's a pain for AWS eks k8s clusters.

[stevehipwell]

/bump

[rotilho]

@danieldreier you labeled it as an enhancement but it sounds more like a bug. It's impossible at the moment to apply a saved plan due to this issue.

Can you relabel it as a bug?

[flixx]

Hello,
this limitation makes it hard to use Atlantis for manage kubernetes resources.
With atlantis a terraform plan is created and saved once a Pull Request is created.

Then after the Pull Request was reviewed, the plan can be applied with a comment.
An error happens if the PR was opened more then 15min ago.

There is an issue for this in the atlantis project here:
runatlantis/atlantis#800

Would be really nice to have this fixed.

Maybe as an idea how this could be solved:

Terraform could re-create the plan always when using terraform apply - even when a saved plan is used as an input.
If a saved plan was given, terraform could validate if the re-generated plan still matches with the input plan and otherwise error out. To avoid backwards compatibility issues this behavior might be controlled with a flag (--refresh-plan)

[joemiller]

Is there a known workaround here? This is a significant problem for some CI workflows.

[SirineBenbrahim]

+1 on this issue. This makes it impossible to automate workflows with auth tokens.

[Unichron]

For the kubernetes provider I solved it with the help of exec plugins: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#exec-plugins

Although it depends on some binary being available, it should be relatively easy to achieve in most CI/CD workflows.

Example for GKE:

provider "kubernetes" {
  host                   = "https://${data.google_container_cluster.this.endpoint}"
  cluster_ca_certificate = base64decode(data.google_container_cluster.this.master_auth[0].cluster_ca_certificate)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "gke-gcloud-auth-plugin"
  }
}