kubernetes_manifest fails terraform plan due to not yet created namespace

[mkemmerz]

Terraform version: 1.1.4
Kubernetes provider version: 2.8.0 (same error with 2.7.1)
Kubernetes version: 1.22.6

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

resource "kubernetes_namespace" "namespace" {
  metadata {
    name = var.namespace_name
    labels = {
      name = var.namespace_name // used for enforcing network policies
    }
  }
}

resource "kubernetes_manifest" "azure_identity" {
  manifest = {
    "apiVersion" = "aadpodidentity.k8s.io/v1"
    "kind"       = "AzureIdentity"
    "metadata" = {
      "name"      = "id-${kubernetes_namespace.namespace.metadata.0.name}"
      "namespace" = kubernetes_namespace.namespace.metadata.0.name
      "annotations" = {
        "aadpodidentity.k8s.io/Behavior" = "namespaced"
      }
    }
    "spec" = {
      "type" = 0
      "resourceID" = var.azure_identity_resource_id
      "clientID" = var.azure_identity_client_id
    }
  }
}

Debug Output

terraform plan '-var-file=./input.tfvars.json' '--out=plan.out'

Warning: This custom resource does not have an associated OpenAPI schema.

  with module.xxx-kubernetes.kubernetes_manifest.azure_identity,
  on .terraform/modules/xxx-kubernetes/main.tf line 97, in resource "kubernetes_manifest" "azure_identity":
  97: resource "kubernetes_manifest" "azure_identity" {

We could not find an OpenAPI schema for this custom resource. Updates to
this resource will cause a forced replacement.
Error: Dry-run failed for non-structured resource

  with module.namespace-kubernetes.kubernetes_manifest.azure_identity,
  on .terraform/modules/xxx-kubernetes/main.tf line 97, in resource "kubernetes_manifest" "azure_identity":
  97: resource "kubernetes_manifest" "azure_identity" {

A dry-run apply was performed for this resource but was unsuccessful:
namespaces "xxx" not found

Panic Output

Steps to Reproduce

1. terraform plan -->

Expected Behavior

Terraform plan should display the resources to be created later.
Terraform creates a new Kubernetes namespace.
Terraform calls the k8s and creates an AzureIdentity object.

We just recently upgraded from k8s 1.21 to 1.22, not sure if this is important.

Actual Behavior

Terraform stops its process.

References

• Plan/apply fails when namespace does not yet exist terraform-provider-kubernetes-alpha#57
• Plan/apply fails when namespace does not yet exist terraform-provider-kubernetes-alpha#57 (comment)

[jbg]

This is more of a design limitation than a bug. When a resource is non-structured (the AzureIdentity CRD doesn't bother to include a schema), kubernetes_manifest uses a dry-run apply on the API server to confirm that it will create successfully (because it can't validate that itself, due to the lack of a schema). This fails because the namespace doesn't exist yet.

The workaround/solution is to plan & apply twice, either in the first stage with the kubernetes_manifest.azure_identity resource not added to your config yet, or in the first stage using -target to target only the namespace.

[mkemmerz]

@jbg Thanks a lot for the reply!
Your suggested solution worked for me (using the -target to create the namespace first). Now we noticed all of the warnings, mentioned in blogs and the documentation about using it in combination with managed K8s solutions (we are using Azure Kubernetes Services).

I guess we will have to rework our modules on this point as we hit a limitation of the kubernetes provider.

[jbg]

Those warnings are about a different issue, but one which manifests in a similar way (needing connectivity to the cluster at plan time). In your case, if Azure had included a schema in that CRD, you shouldn't have hit this specific problem.

Something weird though is that schema was actually added to the aad-pod-identity CRDs in April 2021. So unless you're using a really old version of the CRDs, it should have already worked without needing to do a dry-run apply at plan time.

[mkemmerz]

Thanks again, I will have a look on the CRDs, maybe this resolves our issue

[mkemmerz]

as addition on how I fixed it on the cluster:

• Run kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts/aad-pod-identity/crds/crd.yaml once on the cluster to update the CRDs

The core issue was that we missed to update the CRD of the AAD Pod Identity after upgrading to version 1.8.0.

This was introduced on our cluster by upgrading the Helm Chart from version 4.0.0 to 4.1.0.

[jbg]

Note that you can use the kubernetes_manifest resource to manage the CRDs in Terraform, rather than manually applying manifests, which may help to keep track of CRDs in your cluster and avoid missing updates in future.

[mkemmerz]

Do you have some doumentation about how to do this? Sounds very good

[jbg]

https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest

You can use a tool like https://github.com/jrhouston/tfk8s to convert YAML manifests (even files containing multiple objects) to HCL.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.