fix: disable reconciliation before scaling in SaaS

[deepthidevaki]

While testing E2E test with scaling, noticed that the current zbchaos cluster scale command do not work in SaaS because reconciliation is not disabled.

[ChrisKujawa]

Hm can we add a test for this?

[deepthidevaki]

@Zelldon
Any idea how to properly mock both zeebe crd and statefulset in tests? I get several errors

1. In pause reconciliation - Object 'Kind' is missing in '{"metadata":{"labels":{"cloud.camunda.io/pauseReconciliation":"true"},"name":"testNamespace"}}'
2. When scaling - E0111 11:41:29.861024 51681 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x18fb140), concrete:(*runtime._type)(0x1ae5360), asserted:(*runtime._type)(0x1ae7400), missingMethod:""} (interface conversion: runtime.Object is *v1.StatefulSet, not *v1.Scale) goroutine 10 [running]:

[ChrisKujawa]

Hey @deepthidevaki did you checked already the pods tests? Like this https://github.com/zeebe-io/zeebe-chaos/blob/main/go-chaos/internal/pods_test.go#L68-L70

[deepthidevaki]

Yes. I'm already using the available helper methods.

	// given
	k8Client := CreateFakeClient()
	k8Client.createSaaSCRD(t)
	k8Client.CreateStatefulSetWithLabelsAndName(t, &metav1.LabelSelector{}, "zeebe")

	// when
	_, err := k8Client.ScaleZeebeCluster(3)

[ChrisKujawa]

Do you have your changes somewhere that I can take a look?

[deepthidevaki]

There are no new changes. You can add the following test in statefulset_test.go for reproducing this.

func Test_ShouldPauseReconciliationBeforeScaling(t *testing.T) {
	// given
	k8Client := CreateFakeClient()
	k8Client.createSaaSCRD(t)
	k8Client.CreateStatefulSetWithLabelsAndName(t, &metav1.LabelSelector{}, "zeebe")

	// when
	_, err := k8Client.ScaleZeebeCluster(3)
	require.NoError(t, err)
	statefulset, err := k8Client.GetZeebeStatefulSet()

	// then
	require.NoError(t, err)
	assert.NotNil(t, statefulset)
}

[ChrisKujawa]

@deepthidevaki are we sure that we use the right API here? For me it looks like the GetScale returns some Scale object related to autoscaling

https://github.com/kubernetes/client-go/blob/v0.29.0/kubernetes/typed/apps/v1/statefulset.go#L57

https://miminar.fedorapeople.org/_preview/openshift-enterprise/registry-redeploy/rest_api/apis-apps/v1.StatefulSet.html#Get-apis-apps-v1-namespaces-namespace-statefulsets-name-scale

https://github.com/kubernetes/api/blob/master/core/v1/types.go#L4774

[deepthidevaki]

I think so. It has worked fine during our manual chaos tests. And it is also running fine with our daily automated tests (because reconciliation is already paused by a previous experiment).

[ChrisKujawa]

Ok then I guess the only problem is that we have to setup the fake statefulset correctly

[ChrisKujawa]

@deepthidevaki I played a bit with it and did my changes here https://github.com/zeebe-io/zeebe-chaos/tree/ck-scaling-test

Basically what we have to do is to use the update API if we want to test it properly. GetScale is not supported by the FakeClient.

Furthermore, we have to update the statefulset which we create to set a correct replica value.

The awaiting of the available replicas I moved out, since this makes it harder to test, the status is not updated with the fake clients. Likely we need to call this separate in the other places.

I also saw in the backup commands that we call the disabling of reconcile separately from the scale (so before), we could also do this in our scale command.

[ChrisKujawa]

I realized that we do this always for backups in the command already, I would be fine if we do this in the scale command as well and not here.

https://github.com/zeebe-io/zeebe-chaos/blob/main/go-chaos/cmd/backup.go#L95-L98

[deepthidevaki]

Makes sense. But then I wouldn't test anything specific for this PR, as testing the whole command will be cumbersome.

[deepthidevaki]

Thanks @Zelldon for looking into it further. But I think, I will leave the testing out because not pausing is done in the command. So writing a unit test is more difficult.

[ChrisKujawa]

Thanks @deepthidevaki 🚀 👍🏼