GKE Nodepool keeps on updating frequently

[abhivaidya07]

What happened?

Deployed GKECluster and Nodepool using the below yaml, but the GKECluster remains in RECONCILING state because Nodepool keeps on updating frequently though there is no change after update.

Crossplane yaml

apiVersion: container.gcp.crossplane.io/v1beta1
kind: GKECluster
metadata:
  name: crossplane-cluster
spec:
  forProvider:
    addonsConfig:
      networkPolicyConfig:
        disabled: false
    ipAllocationPolicy:
      useIpAliases: true
    network: sample-team-common-vpc
    subnetwork: sample-gke-us-ct1-subnet
    location: us-central1-c
    loggingService: "logging.googleapis.com/kubernetes"
    masterAuth:
      clientCertificateConfig:
        issueClientCertificate: false
    monitoringService: "monitoring.googleapis.com/kubernetes"
    defaultMaxPodsConstraint:
       maxPodsPerNode: 110
    networkPolicy:
      enabled: false
    podSecurityPolicyConfig:
      enabled: true
---
apiVersion: container.gcp.crossplane.io/v1alpha1
kind: NodePool
metadata:
  name: crossplane-np
spec:
  forProvider:
    autoscaling:
      autoprovisioned: false
      enabled: false
    clusterRef:
      name: crossplane-cluster
    config:
      machineType: e2-medium
      diskSizeGb: 100
      diskType: pd-ssd
      imageType: cos_containerd
      labels:
        test-label: crossplane-created
      machineType: e2-medium
      oauthScopes:
      - "https://www.googleapis.com/auth/devstorage.read_only"
      - "https://www.googleapis.com/auth/logging.write"
      - "https://www.googleapis.com/auth/monitoring"
      - "https://www.googleapis.com/auth/servicecontrol"
      - "https://www.googleapis.com/auth/service.management.readonly"
      - "https://www.googleapis.com/auth/trace.append"      
    initialNodeCount: 1
    maxPodsConstraint:
        maxPodsPerNode: 110
    locations:
      - "us-central1-c"

Status of GKECluster

kubectl get gkecluster
NAME                 READY   SYNCED   STATE         ENDPOINT         LOCATION        AGE
crossplane-cluster   True    True     RECONCILING   xxxxxxxxxx       us-central1-c   31m

Status of Nodepool

kubectl get nodepool
NAME            READY   SYNCED   STATE     CLUSTER-REF          AGE
crossplane-np   True    False    RUNNING   crossplane-cluster   22m

Describing Nodepool throws this message

Message:               update failed: cannot update GKE node pool: googleapi: Error 400: Operation operation-1630410489595-0a0ca8e7 is currently upgrading cluster crossplane-cluster. Please wait and try again once it is done., failedPrecondition
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
    Last Transition Time:  2021-08-31T11:46:31Z
    Reason:                Available
    Status:                True
    Type:                  Ready
Events:
  Type     Reason                           Age                  From                                          Message
  ----     ------                           ----                 ----                                          -------
  Warning  CannotResolveResourceReferences  12m (x3 over 12m)    managed/nodepool.container.gcp.crossplane.io  cannot resolve references: spec.forProvider.cluster: referenced field was empty (referenced resource may not yet be ready)
  Warning  CannotCreateExternalResource     10m (x6 over 12m)    managed/nodepool.container.gcp.crossplane.io  cannot create GKE node pool: googleapi: Error 400: Operation operation-1630409771409-90b76123 is currently creating cluster crossplane-cluster. Please wait and try again once it is done., failedPrecondition
  Warning  CannotCreateExternalResource     9m4s (x2 over 9m5s)  managed/nodepool.container.gcp.crossplane.io  cannot create GKE node pool: googleapi: Error 400: Operation operation-1630409954358-14936243 is currently deleting a node pool for cluster crossplane-cluster. Please wait and try again once it is done., failedPrecondition
  Warning  CannotCreateExternalResource     5m2s (x5 over 8m4s)  managed/nodepool.container.gcp.crossplane.io  cannot create GKE node pool: googleapi: Error 400: Operation operation-1630410014653-f916f4ea is currently upgrading cluster crossplane-cluster. Please wait and try again once it is done., failedPrecondition
  Normal   CreatedExternalResource          4m2s                 managed/nodepool.container.gcp.crossplane.io  Successfully requested creation of external resource
  Normal   UpdatedExternalResource          2m (x6 over 4m1s)    managed/nodepool.container.gcp.crossplane.io  Successfully requested update of external resource
  Warning  CannotUpdateExternalResource     115s (x4 over 2m)    managed/nodepool.container.gcp.crossplane.io  cannot update GKE node pool: googleapi: Error 400: Operation operation-1630410391403-4cfd5a9a is currently upgrading cluster crossplane-cluster. Please wait and try again once it is done., failedPrecondition

How can we reproduce it?

Deploy the above YAML and check on GKE console, the cluster goes in updating state frequently.

What environment did it happen in?

Crossplane version: v1.3
GCP provider pkg version: v0.17.1
Kubernetes version (use kubectl version): 1.20.8-gke.2100
Kubernetes distribution: GKE

[turkenh]

@abhivaidya07 thanks for the ticket. I was able to reproduce the issue and just debugged what is going on here.

It seems that the field causing updates is spec.forProvider.config.imageType in NodePool object due to observation returns COS_CONTAINERD but desired is cos_containerd. Interestingly, gcloud API accepts lowercase but converts to uppercase on server side which is perceived as a configuration difference on crossplane side.

As a workaround, I would suggest you use the uppercase version as below.

apiVersion: container.gcp.crossplane.io/v1alpha1
kind: NodePool
metadata:
  name: crossplane-np
spec:
  forProvider:
    autoscaling:
      autoprovisioned: false
      enabled: false
    clusterRef:
      name: crossplane-cluster
    config:
      machineType: e2-medium
      diskSizeGb: 100
      diskType: pd-ssd
-    imageType: cos_containerd
+    imageType: COS_CONTAINERD

[abhivaidya07]

@turkenh I ACK that everything is working as expected. Thank you 👍
Closing this issue !!