Telemetry webhook validation

[Miles-Garnsey]

What this PR does:

Validates telemetry specifications within the K8ssandraCluster, having regard for the installation status of Prometheus across each K8s DC in the cluster and the telemetry requested for it.

This is preferred to the current situation where validation occurs during reconciliation for two reasons:

1. It ensures that users get fast feedback on the failure via the resource being rejected, instead of them needing to consult logs to determine why service monitors are not created.
2. It ensures that invalid resources do not pass through into the cluster. When this happens, additional updates to the resource cannot propagate, which can cause stability risks if a broken resource is not noticed and then subsequent changes need to be made.

Which issue(s) this PR fixes:
Fixes #235

Checklist

• Changes manually tested
• Automated Tests added/updated
• Documentation added/updated
• CHANGELOG.md updated (not required for documentation PRs)
• CLA Signed: DataStax CLA

[Miles-Garnsey]

I'm trying to figure out what is causing the tests to fail on this one. It isn't really straight forward, especially considering some tests are apparently still flaky. I'm uncertain if I can progress this without #472 being merged.

One source of new flakes appears to be the envtest StopDatacenter, which fails due to the following error at line 184:

                	            	Operation cannot be fulfilled on k8ssandraclusters.k8ssandra.io "stop-dc-test": the object has been modified; please apply your changes to the latest version and try again

This definitely should not be happening, as this is a transient failure due to optimistic locking and should never be fatal. It doesn't happen locally but occurs on GHA because the Patch call has no timing tolerance. Wrapping in an eventually now to see if that improves things.

[Miles-Garnsey]

I've wrapped the patch operation in an Eventually:

patch := client.MergeFromWithOptions(kc.DeepCopy(), client.MergeFromWithOptimisticLock{})
	kc.Spec.Cassandra.Datacenters[0].Stopped = true
	require.Eventually(t, func() bool {
		err := f.Client.Patch(ctx, kc, patch)
		return err == nil
	}, timeout, interval, "timeout waiting to patch dc1 while stopping")

The StopDatacenter test still fails on GHA, and now fails locally as well (BONUS!) This is going well.

[jsanda]

You shouldn't need to wrap writes in Eventually calls unless you are doing so to add retry logic. It's not needed for consistency though.

[Miles-Garnsey]

You shouldn't need to wrap writes in Eventually calls unless you are doing so to add retry logic. It's not needed for consistency though.

Yes, but I need to add retry logic because of the optimistic locking error above Operation cannot be fulfilled on k8ssandraclusters.k8ssandra.io "stop-dc-test": the object has been modified; please apply your changes to the latest version and try again

[burmanm]

Writing old object 100 times will not make a difference, it's still the old object. You need to refresh it (Get)

[Miles-Garnsey]

Writing old object 100 times will not make a difference, it's still the old object. You need to refresh it (Get)

Oh wow, how did I miss that. Thanks @burmanm you just saved me a fair bit of time honestly.

[Miles-Garnsey]

I think this PR is ready for preliminary review.

I've fixed 5 instances where tests were fragile due to being timing sensitive.

There are 8 tests still failing. All fail due to timing related issues, 6/8 are in the multi-cluster suite.

I'm going to investigate the two that are failing in the single-cluster suite, I'm hoping the multi-cluster failures will be dealt with by #472, but we can look to make the calls to the remote cluster concurrent if that doesn't resolve things.

(Edited as a different set of tests is now failing...)

[adejanovski]

This is largely outdated and we never managed to reach a consensus so I'll go ahead and close it.
We can rediscuss this if we think we have a need and a path forward.