terraform plan fails with 'Invalid value: "": field is immutable' error due to wrong content type header

[anrock-sc]

Terraform, Provider, Kubernetes versions

Terraform version: v0.14.2
Provider version: v0.2.1
Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.7-gke.1500", GitCommit:"99f90aac4943d3d1c8d9705458c2711240fcfc2b", GitTreeState:"clean", BuildDate:"2021-02-03T09:17:28Z", GoVersion:"go1.15.5b5", Compiler:"gc", Platform:"linux/amd64"}

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

terraform {
  backend "gcs" {
    bucket = "<BUCKET_NAME>"
    prefix = "services"
  }

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 3.4"
    }
    kubernetes-alpha = {
      source  = "hashicorp/kubernetes-alpha"
      version = ">= 0.2.1"
    }
  }

  required_version = ">= 0.13"
}

locals {
  kubectl_context = join("_", ["gke", var.project, var.region, var.cluster])
}

provider "google" {
  project = var.project
  region  = var.region
}

provider "kubernetes-alpha" {
  config_path    = "~/.kube/config"
  config_context = local.kubectl_context

  server_side_planning = true
}

Debug Output

https://gist.github.com/anrock-sc/34ff83ed56236e32280bd4de9f9459f5

Panic Output

N/A

Steps to Reproduce

Terrafrom resource:

resource "kubernetes_manifest" "homepage-redirect-service" {
  provider = kubernetes-alpha

  manifest = {
    apiVersion = "v1"
    kind       = "Service"
    metadata = {
      annotations = {
        "cloud.google.com/neg" = "{\"ingress\": true}"
      }
      name      = "homepage-redirect"
      namespace = "default"
    }
    spec = {
      type = "NodePort"
      ports = [
        {
          name       = "http"
          port       = 80
          targetPort = 80
          protocol   = "TCP"
        },
      ]
      selector = {
        app = "server"
      }
    }
  }
}

Service yaml on server:

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
    cloud.google.com/neg-status: '{"network_endpoint_groups":{"80":"k8s1-23af038a-default-homepage-redirect-80-70cc1b1e"},"zones":["europe-west1-b","europe-west1-c","europe-west1-d"]}'
  creationTimestamp: "2020-08-19T21:42:42Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cloud.google.com/neg: {}
      f:spec:
        f:clusterIP: {}
        f:externalTrafficPolicy: {}
        f:ports:
          k:{"port":80,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:nodePort: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: Terraform
    operation: Apply
    time: "2020-08-19T21:44:21Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cloud.google.com/neg-status: {}
    manager: glbc
    operation: Update
    time: "2020-08-19T22:13:47Z"
  name: homepage-redirect
  namespace: default
  resourceVersion: "44632089"
  selfLink: /api/v1/namespaces/default/services/homepage-redirect
  uid: d1aecb5f-9558-4b3b-aeea-95a8630b15cf
spec:
  clusterIP: 10.0.1.23
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: 31042
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: server
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

Run terraform plan -target=kubernetes_manifest.homepage-redirect-service

Expected Behavior

Terraform plan completes successfully.

Actual Behavior

Error: rpc error: code = Unknown desc = update dry-run for 'default/homepage-redirect' failed: Service "homepage-redirect" is invalid: spec.clusterIP: Invalid value: "": field is immutable

Important Factoids

We are on the GKE rapid release channel, however when this bug started to occur, roughly around the 22.02.2021, we didn't updated or changed anything on client or server side.

When comparing kubectl http traffic to the terraform alpha provider traffic, we could boil down the issue to the content-type header value which seems to be different.

This is the curl command that matches are the headers send by the kubectl:

curl -k -v -XPATCH  -H "Accept: application/json" -H "Content-Type: application/strategic-merge-patch+json" -H "User-Agent: kubectl/v1.19.3 (darwin/amd64) kubernetes/1e11e4a" 'https://<IP>/api/v1/namespaces/default/services/homepage-redirect?dryRun=All&fieldManager=Terraform' -H "Authorization: Bearer $TOKEN" --data @patch.json

patch.json

 {
  "apiVersion": "v1",
  "kind": "Service",
  "metadata": {
   "annotations": {
    "cloud.google.com/neg": "{\"ingress\": true}"
   },
   "name": "homepage-redirect",
   "namespace": "default"
  },
  "spec": {
   "ports": [
    {
     "name": "http",
     "port": 80,
     "protocol": "TCP",
     "targetPort": 80
    }
   ],
   "selector": {
    "app": "server"
   },
   "type": "NodePort"
  }
 }

Result:

> PATCH /api/v1/namespaces/default/services/homepage-redirect?dryRun=All&fieldManager=Terraform HTTP/2
> Host: <IP>
> Accept: application/json
> Content-Type: application/strategic-merge-patch+json
> User-Agent: kubectl/v1.19.3 (darwin/amd64) kubernetes/1e11e4a
> Authorization: Bearer ***
> Content-Length: 376
>
< HTTP/2 200
< audit-id: 635701a3-fde3-4ffb-b387-e0adc7e4d81b
< cache-control: no-cache, private
< content-type: application/json
< content-length: 1449
< date: Tue, 23 Feb 2021 11:35:33 GMT
<
<CONFIG_JSON_FILE (Similar to the application.yaml from above)>

Note that for kubectl the content-type header is set to Content-Type: application/strategic-merge-patch+json However, for the terraform alpha provider the content type-header application/apply-patch+yaml is set resulting in an Unprocessable Entity Error.

Here the curl command with the content-type header the terraform alpha provider would use:

curl -k -v -XPATCH  -H "Accept: application/json" -H "Content-Type: application/apply-patch+yaml" -H "User-Agent: kubectl/v1.19.3 (darwin/amd64) kubernetes/1e11e4a" 'https://<IP>/api/v1/namespaces/default/services/homepage-redirect?dryRun=All&fieldManager=Terraform' -H "Authorization: Bearer $TOKEN" --data @patch.json

Result:

> PATCH /api/v1/namespaces/default/services/homepage-redirect?dryRun=All&fieldManager=Terraform HTTP/2
> Host: <IP>
> Accept: application/json
> Content-Type: application/apply-patch+yaml
> User-Agent: kubectl/v1.19.3 (darwin/amd64) kubernetes/1e11e4a
> Authorization: Bearer ***
> Content-Length: 376
>
< HTTP/2 422
< audit-id: 0814b2ef-e33d-4477-a970-59538f1571ce
< cache-control: no-cache, private
< content-type: application/json
< content-length: 384
< date: Tue, 23 Feb 2021 11:36:59 GMT
<
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Service \"homepage-redirect\" is invalid: spec.clusterIP: Invalid value: \"\": field is immutable","reason":"Invalid","details":{"name":"homepage-redirect","kind":"Service","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"\": field is immutable","field":"spec.clusterIP"}]},"code":422}

It seems to be related to the patch definitions in https://github.com/hashicorp/terraform-provider-kubernetes-alpha/blob/master/vendor/k8s.io/apimachinery/pkg/types/patch.go#L24-L29

References

This bug relates to:

• Helm 3 - upgrade nginx - spec.clusterIP: Invalid value: "": field is immutable helm/helm#6378 (comment)
• Changes to immutable fields not applined when resource is tainted #132
• applying a service twice with clusterIP set to blank returns an error kubernetes/kubernetes#68369

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[alexsomesan]

@anrock-sc Thanks a lot for the detailed analysis!

I really appreciate the fact that you spotted the difference in content-type header between this provider and other Kubernetes tools. In fact, that is by design. This provider uses a different mechanism for interacting with the Kubernetes API, called server-side-apply. The content type header application/apply-patch+yaml is actually how the provider instructs the API server that it wants to operate in this mode.

Server-side-apply is a relatively new feature in Kubernetes. You can read about it in more detail here: https://kubernetes.io/docs/reference/using-api/server-side-apply/

On point to the failure you are seeing, it is actually not caused by the patch type but rather by a limitation in the provider that causes the value of the clusterIp attribute to not be correctly tracked because it was not part of the original configuration at the time of apply. One of the reasons why this provider is designated as "alpha" right now.

This limitation and some other issues is what led us to almost completely re-engineer this provider in the recent moths. That is why there was a pause in releases recently. I expect to be able to release version v0.3.0 in about a week or so. It will be drastically more robust and will also solve this problem that you are seeing. If you want to try out the work in progress you can build from this branch: https://github.com/hashicorp/terraform-provider-kubernetes-alpha/tree/tftypes. If you do, make sure to remove server_side_planning from the provider configuration. Also, I cannot stress this enough, this is very new software - do not use in any production environments.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!