Add option to manually configure CRD OpenAPI specifications to Provider

[apjoseph]

Description

The current behavior of the provider requiring the Kubernetes cluster to exist if any Kubernetes resources require a custom CRD is extremely baffling to Terraform users. Worse yet, the only indication of the issue is found in this cryptic error message:

╷
│ Error: Failed to construct REST client
│
│   with module.cluster-resources.kubernetes_manifest.main_gateway,
│   on cluster-resources/development.tf line 28, in resource "kubernetes_manifest" "main_gateway":
│   28: resource "kubernetes_manifest" "main_gateway" {
│
│ cannot create REST client: no client config
╵ 

This message doesn't give any indication of the actual problem and may lead to defenestration of developer laptops. The provider docs do clearly describe the issue -but the provided solution of using multiple apply operations undermines the problem IAC tools are meant to solve. Any person initializing the Terraform configuration must remember that they need to manually find the module that contains the Kubernetes CRD resources and set count = 0 , do an apply, delete that parameter -and then apply again.

Therefore, I think a good short-term compromise solution is to simply allow manually adding in the CRD OpenAPI specifications in the provider block. This way, the provider wouldn't need to access the cluster unless it can't find a definition in the provided specs.

If the cluster does not yet exist and a CRD is not found in the provided map and the provider needs to look up the resource from the cluster, then a message should be sent to the user indicating that a CRD OpenAPI spec should be provided.

Potential Terraform Configuration

I ran into this issue today while trying to set up the new Gateway API on GKE -which this provider doesn't currently support natively. I think the following config would be a relatively straightforward solution to the problem:

provider "github" {}

data "github_repository_file" "sigs_gateway_gateway_crd" {
  repository = "kubernetes-sigs/gateway-api"
  branch     = "v0.5.1"
  file       = "config/crd/standard/gateway.networking.k8s.io_gateways.yaml"
}

data "github_repository_file" "sigs_gateway_httproute_crd" {
  repository = "kubernetes-sigs/gateway-api"
  branch     = "v0.5.1"
  file       = "config/crd/standard/gateway.networking.k8s.io_httproutes.yaml"
}

locals {
  crd_map = {
    gateway_gateway = {
      spec = yamldecode(github_repository_file.sigs_gateway_gateway_crd.content)
    }
    gateway_httproute = {
      spec = yamldecode(github_repository_file.sigs_gateway_httproute_crd.content)
    }
  }
}

provider "kubernetes" {
  host                   = "https://${data.google_container_cluster.default.endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(data.google_container_cluster.default.master_auth[0].cluster_ca_certificate)
  crd_map                = local.crd_map
}

References

Would solve #1625 #1775 and countless other duplicate issues that are sure to keep recurring until a solution to the custom CRD problem is implemented.

[marziply]

Similar to #1917

[jan-hudec]

Similar to #1917

Not similar, proposing a solution to.

Regarding the solution, wouldn't it be better to provide the spec to the specific kubernetes_manifest resources themselves?

[marziply]

Similar to #1917

Not similar, proposing a solution to.

Regarding the solution, wouldn't it be better to provide the spec to the specific kubernetes_manifest resources themselves?

That's literally what I've said on the issue I linked.