Installing manifests that depend on CRDs created in previous resources. Plan fails as not yet present in cluster and no way to apply without targeting and using old provider.

[MrVentzi]

Terraform, Provider, Kubernetes versions

Terraform version: v0.14.11
Provider version: v0.3.3
Kubernetes version: 1.18.16-gke.2100

Affected Resource(s)

Terraform Configuration Files

resource "helm_release" "istio_operator" {
  name      = "istio-operator"
  chart     = "/data/shared/istio-1.9.3/manifests/charts/istio-operator"
  namespace = var.istio_namespace

  set {
    name = "hub"
    value = var.release_hub
  }

  set {
    name = "operatorNamespace"
    value = var.operator_namespace
  }

  set {
    name = "watchedNamespaces"
    value = var.istio_namespace
  }

}

resource "time_sleep" "destroy_istio_resources_before_operator" {
  depends_on = [helm_release.istio_operator]

  destroy_duration = "60s"
}

resource "kubernetes_manifest" "istio" {
  provider = kubernetes-alpha
  manifest = yamldecode(templatefile(var.istio_tpl_yml_path_override != null ? var.istio_tpl_yml_path_override : "${path.module}/${var.istio_tpl_yml_path}", {
    istio_namespace            = var.istio_namespace
    replicas                   = var.control_plane_replicas
    release_hub                = var.release_hub
    release_tag                = var.release_tag
    priority_class_name        = var.priority_class_name
    node_pool_affinity         = var.node_pool_affinity
    tolerations                = var.tolerations
    resources                  = var.resources
    access_log_format_override = var.access_log_format_override
    remove_node_affinity       = var.remove_node_affinity
    remove_tolerations         = var.remove_tolerations
  }))

  wait_for = {
    fields = {
      "status.status" = "HEALTHY"
    }
  }

  depends_on = [helm_release.istio_operator, time_sleep.destroy_istio_resources_before_operator]
}

Debug Output

Panic Output

Steps to Reproduce

Expected Behavior

What should have happened?
Be able to plan/apply by honouring depends_on and waiting for each resource to be applied in turn.

Installing the istio_operator adds the install.istio.io group, in turn, when that is available, installing istio will add a few more such as security.istio.io

Actual Behavior

What actually happened?

The provider doesn't know how to plan the resources as their dependencies have not yet been created.

Error: rpc error: code = Unknown desc = no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"

Error: Failed to determine GroupVersionResource for manifest

  on .terraform/modules/istio/istio-control-plane.tf line 58, in resource "kubernetes_manifest" "istio":
  58: resource "kubernetes_manifest" "istio" {

no matches for kind "IstioOperator" in group "install.istio.io"

To get around it:

1. pin down the kubernetes-alpha version to 0.2.1
2. terraform apply -target helm_release.istio_operator
3. wait a bit
4. terrafomr apply -target kubernetes_manifest.istio
5. wait a bit
6. apply the rest of the state

IF running 0.3.3 provider version then we get:

1. terraform apply -target helm_release.istio_operator
2. wait a bit
3. error below:

Error: rpc error: code = Unknown desc = no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"

Error: No valid OpenAPI definition

  on .terraform/modules/istio/istio-control-plane.tf line 58, in resource "kubernetes_manifest" "istio":
  58: resource "kubernetes_manifest" "istio" {

Resource IstioOperator.v1alpha1.install.istio.io does not have a valid OpenAPI
definition in this cluster.

Usually this is caused by a CustomResource without a schema.

which is mentioned here: #181 (comment)

$ kubectl api-resources | grep IstioOperator
istiooperators                    iop          install.istio.io               true         IstioOperator

which means we cannot even target the istio install with latest provider.

Any ideas are welcome as currently we have no way of applying the state cleanly.

Thanks

Important Factoids

References

• GH-1234

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ivanov-aleksander]

I also hace such error. In my case i have such kubernetes_manifest resource

resource "kubernetes_manifest" "vertical-pod-autoscaler" {
  provider = kubernetes-alpha

  manifest = {
    "apiVersion" = "autoscaling.k8s.io/v1"
    "kind"       = "VerticalPodAutoscaler"
     ...

Terraform plan fails with error

╷
│ Error: No valid OpenAPI definition
│ 
│   with kubernetes_manifest.vertical-pod-autoscaler,
│   on vpa.tf line 4, in resource "kubernetes_manifest" "vertical-pod-autoscaler":
│    4: resource "kubernetes_manifest" "vertical-pod-autoscaler" {
│ 
│ Resource VerticalPodAutoscaler.v1.autoscaling.k8s.io does not have a valid OpenAPI definition in this cluster.
│ 
│ Usually this is caused by a CustomResource without a schema.

Cluster has verticalpodautoscalers api resource with Kind: VerticalPodAutoscaler but with NAME verticalpodautoscalers

kubectl api-resources --api-group=autoscaling.k8s.io   
NAME                               SHORTNAMES      APIVERSION              NAMESPACED   KIND
verticalpodautoscalers             vpa             autoscaling.k8s.io/v1   true         VerticalPodAutoscaler

It looks like kubernetes-alpha checks Kind instead of CRD. In my exaple it checks Resource VerticalPodAutoscaler.v1.autoscaling.k8s.io instead of Resource VerticalPodAutoscalers.v1.autoscaling.k8s.io

You should use api resource name instead of kind for validating OpenAPI definition