Create a datasource for URL documents

[briantopping]

It would be useful to have a datasource that could pull documents from the internet. So instead of having 1,000 lines of YAML embedded in the .tf, just provide the URL to the YAML and voila!

[sebastianhutter]

Have a look at the "file" function of terraform.
it can't use an URL but you can put your kubernetes manifest into a separate file:

resource "kubectl_manifest" "mymanifest" {
  yaml_body = file("${path.module}/mymanifest.yaml")
}

[briantopping]

Hi @sebastianhutter thanks for the tip! I guess that helps keep everything idempotent if someone pastes a url from master branch. So I guess it depends on the audience and intent.

What I had envisioned was being able to set the desired version number in variables, interpolate the URL and load from there. It's a power tool, and sometimes power tools chop off fingers... 🤪

[sebastianhutter]

I like power tools ;-)

You can do that with terraform natively though.

There is a"HTTP" datasource which downloads a file from an URL and which you can then feed into your resources:

data "http" "manifestfile" {
  url = "https://mycoolnawsomemanifesturl.com/${var.manifestversion}/manifest.yaml"
}

resource "kubectl_manifest" "mymanifest" {
  yaml_body = data.manifestfile.body
}

(untested)

Keeping the tasks separate allows the terraform plugins to remain small and efficient - it also allows terraform to manage dependencies properly - so if the download of the file fails the resource wont be created etc.

[smiletrl]

ln case there're multiple configs in the manifest.yaml, "kubectl_path_documents" might be necessary, since "kubectl_manifest" by default only runs for the first config. "null_resource" can be used to download that yaml file firstly. Something like

resource "null_resource" "controller_rancher_installation" {

  provisioner "local-exec" {
    command = <<EOT
      echo "Downloading rancher config"
      curl -L https://some-url.yaml -o rancherconfig.yaml
    EOT
  }
}

data "kubectl_path_documents" "rancher_manifests" {
    // somehow pattern only supports search in local file directory, not from remote url.
    pattern = "./rancherconfig.yaml"
    depends_on = [null_resource.controller_rancher_installation]
}

[pduchnovsky]

ln case there're multiple configs in the manifest.yaml, "kubectl_path_documents" might be necessary, since "kubectl_manifest" by default only runs for the first config. "null_resource" can be used to download that yaml file firstly. Something like

resource "null_resource" "controller_rancher_installation" {

  provisioner "local-exec" {
    command = <<EOT
      echo "Downloading rancher config"
      curl -L https://some-url.yaml -o rancherconfig.yaml
    EOT
  }
}

data "kubectl_path_documents" "rancher_manifests" {
    // somehow pattern only supports search in local file directory, not from remote url.
    pattern = "./rancherconfig.yaml"
    depends_on = [null_resource.controller_rancher_installation]
}

This wouldn't work then with kubectl_manifest which as per docu should be used with count

resource "kubectl_manifest" "test" {
    count     = length(data.kubectl_path_documents.manifests.documents)
    yaml_body = element(data.kubectl_path_documents.manifests.documents, count.index)
}

This is the error you'd get due to file not being available 'yet':

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

I haven't found a workaround yet..

[smiletrl]

@pduchnovsky as the error says, this error can be fixed by

% terraform apply -target=null_resource.controller_rancher_installation

% terraform apply -target=kubectl_path_documents.rancher_manifests

// Test with this command, or simply run terraform apply. It should work well with `terraform apply` after the dependent file resource has been applied before.
% terraform apply -target=kubectl_manifest.test

See more at https://www.hashicorp.com/blog/resource-targeting-in-terraform.

[pduchnovsky]

@pduchnovsky as the error says, this error can be fixed by

% terraform apply -target=null_resource.controller_rancher_installation

% terraform apply -target=kubectl_path_documents.rancher_manifests

// Test with this command, or simply run terraform apply. It should work well with `terraform apply` after the dependent file resource has been applied before.
% terraform apply -target=kubectl_manifest.test

See more at https://www.hashicorp.com/blog/resource-targeting-in-terraform.

If you use terraform cloud as we do for fully automated deployment this is not possible though.

[smiletrl]

ah, not sure how your deployment process works. In my imagination, maybe run this command in terraform apply -target=kubectl_path_documents.rancher_manifests before terraform apply in the jenkins phase or circleci step?

[pduchnovsky]

ah, not sure how your deployment process works. In my imagination, maybe run this command in terraform apply -target=kubectl_path_documents.rancher_manifests before terraform apply in the jenkins phase or circleci step?

terraform cloud performs changes on environment automatically based on commits to monitored repository, also they do not allow terraform apply on remote backend therefore this workaround wouldn't be viable.
I downloaded manifests we use and pushed it to repository as a solution for now, not the best but until terraform can actually work with remote manifests this is the only solution I got.

[tommyjcarpenter]

@sebastianhutter you need data.http.manifestfile.body

[nickjj]

I know this thread is a few months old but being able to apply a single URL yaml file with multiple configs would be really useful to have as a built-in feature.

For example I wanted to kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml. I tried a bunch of strategies unsuccessfully and ultimately reverted back to executing kubectl apply manually outside of Terraform but it would have been pretty sweet to keep it as part of the Terraform configuration to avoid having to run a separate command. This separation is worse than it sounds because it means lugging around a shell script to help make these commands easier to run and they hold state (version variables, etc.).

[luisdavim]

I tried using the kubectl_file_documents data source together with the http data source but keep running into #61 (comment)

[bbetter173]

I too am seeing a similar issue - in my case, the YAML documents are generated by Rancher on a per-cluster basis, so using static ones isn't an option as there is still a dependency on the http data source for, at the very least, a unique token. We're not using Terraform Cloud though, so targeting the kubectl_file_documents is a disappointing, but usable workaround in this case.

[dirodriguezm]

I managed to make it work like this, if anyone is interested.

provider "kubectl" {
  host                   = module.eks_cluster.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks_cluster.cluster_certificate_authority_data)
}

data "http" "metrics_server_raw" {
  url = "https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml"
}

data "kubectl_file_documents" "metrics_server_doc" {
  content = data.http.metrics_server_raw.body
}

resource "kubectl_manifest" "metrics_server" {
  for_each  = data.kubectl_file_documents.metrics_server_doc.manifests
  yaml_body = each.value
}

[panaut0lordv]

With the latest http provider use:
content = data.http.metrics_server_raw.response_body
instead of
content = data.http.metrics_server_raw.body

[akhilrajmailbox]

###### Providers ######
terraform {
  required_version = ">=1.8"
  required_providers {
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = "1.14.0"
    }
  }
}

provider "kubectl" {
    config_path    = "~/.kube/config"
}

###### Resources ######
data "http" "manifestfile" {
  url = "https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/helm/aws-load-balancer-controller/crds/crds.yaml"
}

data "kubectl_file_documents" "alb_crds" {
  content = data.http.manifestfile.response_body
}

resource "kubectl_manifest" "alb_crds" {
  for_each  = data. kubectl_file_documents.alb_crds.manifests
  yaml_body = each.value
}