k8s.IsDeploymentAvailable() sometimes returning true even if de deployment is not complete

[marcellmartini]

Describe the bug

If the test runs too fast, the IsDeploymentAvailable() doesn't correctly check if the deployment is really available. This is an intermittent error. The error does not occur If the deployment was made in the default namespace.

Full Error

$ go test -timeout 30s -tags kubeall -run ^TestKubernetesDeployment$ ./
=== RUN   TestKubernetesDeployment
=== PAUSE TestKubernetesDeployment
=== CONT  TestKubernetesDeployment
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/logger.go:66: Running command kubectl with args [--namespace 08fqcs apply -f ../deploy.yaml]
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/logger.go:66: deployment.apps/nginx created
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/retry.go:91: Wait for deployment nginx to be provisioned.
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/deployment.go:95: Deployment is now available
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/tunnel.go:178: Creating a port forwarding tunnel for resource deploy/nginx routing local port 0 to remote port 80
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/tunnel.go:207: Error finding available pod: Deployment nginx is not available, reason: NewReplicaSetCreated, message: Created new replica set "nginx-765cb4ff6b"
    /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/tunnel.go:173:
                Error Trace:    /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/tunnel.go:173
                                                        /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/deploy_test.go:34
                Error:          Received unexpected error:
                                Deployment nginx is not available, reason: NewReplicaSetCreated, message: Created new replica set "nginx-765cb4ff6b"
                Test:           TestKubernetesDeployment
TestKubernetesDeployment 2023-04-28T23:19:53-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/logger.go:66: Running command kubectl with args [--namespace 08fqcs delete -f ../deploy.yaml]
TestKubernetesDeployment 2023-04-28T23:19:54-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/logger.go:66: deployment.apps "nginx" deleted
TestKubernetesDeployment 2023-04-28T23:19:54-04:00 /home/shotler/github/personal-projects/devops-tools/k8s/operatos/3-deployment/test/client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
--- FAIL: TestKubernetesDeployment (0.15s)
FAIL
FAIL    github.com/marcellmartini/devops-tools/k8s/operatos/3-deployment        0.156s

To Reproduce

Depoyment.yam

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          resources:
            limits:
              memory: "128Mi"
              cpu: "500m"
          ports:
            - containerPort: 80

Deployment test

func TestKubernetesDeployment(t *testing.T) {
	t.Parallel()

	kubeResourcePath := "./deploy.yaml"

	uniqueID := strings.ToLower(random.UniqueId())
	options := k8s.NewKubectlOptions("", "", uniqueID)
	k8s.CreateNamespace(t, options, uniqueID)
	defer k8s.DeleteNamespace(t, options, uniqueID)

	k8s.KubectlApply(t, options, kubeResourcePath)
	defer k8s.KubectlDelete(t, options, kubeResourcePath)

	k8s.WaitUntilDeploymentAvailable(t, options, "nginx", 60, 1*time.Second)

	// Open a tunnel to Deployment from any available port locally
	tunnel := k8s.NewTunnel(options, k8s.ResourceTypeDeployment, "nginx", 0, 80)
	defer tunnel.Close()
	tunnel.ForwardPort(t)

	// Setup a TLS configuration to submit with the helper, a blank struct is acceptable
	tlsConfig := tls.Config{}

	// Try to access the nginx service on the local port, retrying until we get a good response for up to 5 minutes
	http_helper.HttpGetWithRetryWithCustomValidation(
		t,
		fmt.Sprintf("http://%s", tunnel.Endpoint()),
		&tlsConfig,
		60,
		5*time.Second,
		verifyNginxWelcomePage,
	)
}

Expected behavior

$ go test -timeout 30s -tags kubeall -run ^TestKubernetesDeployment$ ./
=== RUN   TestKubernetesDeployment
=== PAUSE TestKubernetesDeployment
=== CONT  TestKubernetesDeployment
TestKubernetesDeployment 2023-04-28T23:24:14-04:00 ./client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:24:14-04:00 ./logger.go:66: Running command kubectl with args [--namespace zy5a2w apply -f ../deploy.yaml]
TestKubernetesDeployment 2023-04-28T23:24:14-04:00 ./logger.go:66: deployment.apps/nginx created
TestKubernetesDeployment 2023-04-28T23:24:14-04:00 ./retry.go:91: Wait for deployment nginx to be provisioned.
TestKubernetesDeployment 2023-04-28T23:24:14-04:00 ./client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:24:14-04:00 ./retry.go:103: Wait for deployment nginx to be provisioned. returned an error: Deployment nginx is not available, reason: NewReplicaSetCreated, message: Created new replica set "nginx-765cb4ff6b". Sleeping for 1s and will try again.
TestKubernetesDeployment 2023-04-28T23:24:15-04:00 ./retry.go:91: Wait for deployment nginx to be provisioned.
TestKubernetesDeployment 2023-04-28T23:24:15-04:00 ./client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:24:15-04:00 ./retry.go:103: Wait for deployment nginx to be provisioned. returned an error: Deployment nginx is not available, reason: MinimumReplicasUnavailable, message: Deployment does not have minimum availability.. Sleeping for 1s and will try again.
TestKubernetesDeployment 2023-04-28T23:24:16-04:00 ./retry.go:91: Wait for deployment nginx to be provisioned.
TestKubernetesDeployment 2023-04-28T23:24:16-04:00 ./client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:24:16-04:00 ./retry.go:103: Wait for deployment nginx to be provisioned. returned an error: Deployment nginx is not available, reason: MinimumReplicasUnavailable, message: Deployment does not have minimum availability.. Sleeping for 1s and will try again.
TestKubernetesDeployment 2023-04-28T23:24:17-04:00 ./retry.go:91: Wait for deployment nginx to be provisioned.
TestKubernetesDeployment 2023-04-28T23:24:17-04:00 ./client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:24:17-04:00 ./retry.go:103: Wait for deployment nginx to be provisioned. returned an error: Deployment nginx is not available, reason: MinimumReplicasUnavailable, message: Deployment does not have minimum availability.. Sleeping for 1s and will try again.
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./retry.go:91: Wait for deployment nginx to be provisioned.
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./deployment.go:95: Deployment is now available
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./tunnel.go:178: Creating a port forwarding tunnel for resource deploy/nginx routing local port 0 to remote port 80
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./tunnel.go:210: Selected pod nginx-765cb4ff6b-lwzjp to open port forward to
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./tunnel.go:223: Using URL https://192.168.49.2:8443/api/v1/namespaces/zy5a2w/pods/nginx-765cb4ff6b-lwzjp/portforward to create portforward
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./tunnel.go:240: Requested local port is 0. Selecting an open port on host system
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./tunnel.go:246: Selected port 37927
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./tunnel.go:272: Successfully created port forwarding tunnel
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./retry.go:91: HTTP GET to URL http://localhost:37927
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./http_helper.go:59: Making an HTTP GET call to URL http://localhost:37927
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./logger.go:66: Running command kubectl with args [--namespace zy5a2w delete -f ../deploy.yaml]
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./logger.go:66: deployment.apps "nginx" deleted
TestKubernetesDeployment 2023-04-28T23:24:18-04:00 ./client.go:42: Configuring Kubernetes client using config file /home/shotler/.kube/config with context
--- PASS: TestKubernetesDeployment (4.18s)
PASS
ok      github.com/marcellmartini/devops-tools/k8s/operatos/3-deployment        4.194s

[denis256]

Fix released in https://github.com/gruntwork-io/terratest/releases/tag/v0.41.22