Unauthorized after update to v1.10 using token auth #679
Comments
|
Having the same issue with authentication in version 1.10 Terraform Configuration File Terraform Version: v0.11.14 for what it's worth it seemed to always use a static Authorization: Bearer token independent of the value in the provider I was having this issue in |
|
We see this too. It seems like the smoking gun is this PR. If change is still desired, should this release have been an x-release (2.0)? There could be a way to reimplemented that change using resource syntax that is additive and thus would only constitute a y release like you have it now. For now, we are telling our developers to pin to 1.9.0. |
|
I also just ran into this issue and have figured out what I believe is the exact cause. When the terraform is run in a Kubernetes pod, the Kubernetes configuration is defaulted to be the in cluster configuration, which sets all of these fields in the return &Config{
// TODO: switch to using cluster DNS.
Host: "https://" + net.JoinHostPort(host, port),
TLSClientConfig: tlsClientConfig,
BearerToken: string(token),
BearerTokenFile: tokenFile,
}, nilIt is the if v, ok := d.GetOk("token"); ok {
cfg.BearerToken = v.(string)
}Instead, it should do: if v, ok := d.GetOk("token"); ok {
cfg.BearerToken = v.(string)
cfg.BearerTokenFile = nil
} // Path to a file containing a BearerToken.
// If set, the contents are periodically read.
// The last successfully read value takes precedence over BearerToken.
BearerTokenFile string |
|
In our builds, we tip toed around this issue by setting the $ export KUBERNETES_SERVICE_HOST=
$ terraform init
$ terraform planThis causes the following check in the kubernetes rest client to think it is not running in a cluster https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/vendor/k8s.io/client-go/rest/config.go#L404-L407 host, port := os.Getenv("KUBERNETES_SERVICE_HOST"), os.Getenv("KUBERNETES_SERVICE_PORT")
if len(host) == 0 || len(port) == 0 {
return nil, ErrNotInCluster
} |
|
I think I'm having a similar issue, but with a different result. I'm using Google Container Cluster to both run atlantis and to configure other clusters, using the following configuration: I'm seeing the following error: Reverting back to version One thing to note, this configuration is not managing the local cluster, so having a service account will not fix the issue, actually if I configure a service account it starts to fail with the A service account is not configured as it's expected to use GCP service account to access the remote clusters to do the management. |
|
@caquino Does setting the |
|
When seeing the I have tried to reproduce this with the configuration mentioned in the issue description. I had to replace the I am getting the same |
|
Follow-up Using the following configuration works as expected. data "google_client_config" "current" {}
data "google_container_cluster" "this" {
name = var.cluster_name
location = var.cluster_location
project = var.project_id
}
provider "kubernetes" {
version = "1.10"
cluster_ca_certificate = base64decode(
data.google_container_cluster.this.master_auth[0].cluster_ca_certificate,
)
host = "https://${data.google_container_cluster.this.endpoint}"
token = data.google_client_config.current.access_token
load_config_file = false
}
resource "kubernetes_namespace" "this" {
metadata {
name = var.namespace
}
}Versions used: Provider request log to Kubernetes API: |
|
Please provide more details about your cluster configuration (SA used, version) and I can attempt to reproduce the supposed issue again. |
|
@alexsomesan The problem only occurs when terraform is run on a different Kubernetes cluster than the cluster being updated by the terraform script. |
|
@iciclespider Do you mean "inside a different cluster" ? |
@alexsomesan Yes. Terraform is executed inside a different cluster. In my case, I am running Jenkins in a private, on premise Kubernetes cluster, and the build jobs are attempting to update an AWS EKS cluster. When the kubernetes rest api is called, the correct host and ca certificate is used, but the token used is the token found in the |
My case is similar, I running terraform inside of a cluster and I'm trying to update another gke cluster. I have the right permission for the SA and using the same SA with 1.9 I don't have any issue. I hope this help. |
|
We also run Terraform (via Atlantis) from inside a Kubernetes cluster. Change #497 appears to break our existing configuration, as we're creating a provider for a cluster from inside a different cluster. The end result is that Terraform has new implicitly loaded values for a Kubernetes provider's arguments, based on it being inside of a cluster, that we did not set ourselves. Pinning to 1.9 helps, but it seems like we need an option to disable in-cluster configuration, or an opt-in to enable in-cluster configuration (considering this appears to be a breaking change). The other option is to document that in-Kubernetes users may need to hide Kubernetes environment variables from Terraform, which seems like the worst choice here. Would it be best to only load the in-cluster config if no arguments are set on a Kubernetes provider? |
|
Same case here. We are using a Google service account for authenticating against GKE, and we provision the token to the Kubernetes provider via We found that, instead of using the passed in token, Atlantis is using the token found in the I agree with @nilium , It would be better to only load the in-cluster config if no arguments are explicitly set on Kubernetes provider. For now, we reverted the version to v1.9.0 to keep our Atlantis server happy. Hope this issue gets fixed soon. |
|
Same here... From gitlab-runner inside EKS, connecting to another EKS === |
|
Same here, #497 seems like the cause, maybe it would be best to have a |
|
@alexsomesan the problem is the following: if terraform runs in a kubernetes cluster in a pod that has service account credentials mounted and a config file is not used (as in the above examples) there is no way to ignore the pod service account credentials. |
|
I'm going to solve this by moving our Atlantis installation to a Workload Identity enabled cluster in a namespace where we can control the default GSA, but, IMO, InClusterConfig shouldn't be forced if we're passing in an access token. I believe the default logic should be to respect what is being passed in, check for InClusterConfig if nothing is being passed in and then fail. |
|
For those encountering this issue when running terraform in kubernetes pods, there's a work-around provided by @iciclespider: Setting the A fix is being worked on. I believe @iciclespider also nailed the root cause and the solution. |
|
Ran into the exact same issue with v1.10. As @pdecat mentioned above, the fix provided by @iciclespider should fix the root cause. Given that it's just an one line patch, @alexsomesan could you push a quick commit to have it fixed? Let us know if you think it's better for any of us to submit a PR for it. This is affecting all terraform runs within k8s pods that are using token auth, so it's rather serious. At least it's breaking all of our Jenkins pipelines. |
|
Hi @houqp, after more investigation, it is not the single line patch I expected as this would have only corrected the case where a service account token is available at I've submitted #686 which should fix both cases. Obviously, feedback is welcome, you may even build the provider yourself to test the fix. I understand this is causing issues but as there is a work-around, it should be avoided to rush a fix that would only make things worse. |
|
good catch @pdecat and thanks for creating that PR! |
|
Thanks so much for this @iciclespider !! |
|
So even I am getting the same error. I am getting it with Kubernetes Provider and EKS. I got it for :
The mentioned workaround fixed the issue for me. Happy to see that root cause and fix both are available. Thanks a lot @iciclespider @pdecat |
|
When will we see a new release? I'm running into Error: Failed to configure: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory issues in my pipeline (bitbucket pipelines) too when using load_config_file = false With 1.9 it works, so downgraded for now, but something seems very off when telling the provider to not load the config file. I'm curious to see if the PR merged to solve this issue will also solve mine. |
|
I'm having the exact same issue, however I'm not running Terraform inside an EKS cluster. Even weirder is that I can |
|
UPDATE: Once I |
|
Looking forward to have @pdecat work included in the next release! (Or I guess specifying Want to add a data point about the issue. I got My terraform does not run in different kubernetes clusters, but is indeed run in different containers / VMs. I run on my laptop as well as on travisCI builds. Just one week ago I split my terraform code to different repositories and have them run on travisCI as part of some CI/CD automation. Then this This is what I've end up provider "kubernetes" {
# cannot downgrade from 1.10 to 1.9 by specifying `~> 1.9`, it has to be exact `1.9`; otherwise terraform will still upgrade to 1.10 even if `.terraform` directory is deleted
version = "1.9"
host = digitalocean_kubernetes_cluster.project_digitalocean_cluster.endpoint
load_config_file = false
token = digitalocean_kubernetes_cluster.project_digitalocean_cluster.kube_config[0].token
cluster_ca_certificate = base64decode(
digitalocean_kubernetes_cluster.project_digitalocean_cluster.kube_config[0].cluster_ca_certificate
)
}Since I'm also using a module that contains kubernetes provider, I downgraded the kubernetes provider to 1.9 in that module as well. Not sure if setting |
ghost
locked and limited conversation to collaborators
Hi there,
After upgrading to version 1.10, token authentication stopped working.
Terraform Version
0.12.11
Affected Resource(s)
Terraform Configuration Files
Expected Behavior
Get a success authentication.
Actual Behavior
It fail, getting a "Unauthorized "
Important Factoids
Using version 1.9 there is no problem
The text was updated successfully, but these errors were encountered: