maxConcurrentReplacements causing deletion update strategy

[simenl]

What happened?

As a mitigation to storage roles being recruited to log processes [forum post], we tried setting maxConcurrentReplacements to reduce the number of concurrent exclusions.
However, this caused the Deletion strategy to incorrectly be applied on the remaining processes, for updates that requires the Replacement strategy. Consequently this resulted in unschedulable pods, as we updated the node selector to an availability zone incompatible with the existing persistent volume on the process.

What did you expect to happen?

Processes that requires replacement, should not be eligible for the delete update strategy. Even if they were not selected for replacement (yet) due to maxConcurrentReplacements.

How can we reproduce it (as minimally and precisely as possible)?

1. Create a foundationdb cluster with maxConcurrentReplacements and multiple storage pods/processes:

spec:
  automationOptions:
    maxConcurrentReplacements: 1

2. Make an update to the CRD that requires a replacement. E.g. changing the the node selector.

3. Observe that some of the storage pods will be updated through the delete update strategy

Anything else we need to know?

No response

FDB Kubernetes operator

We run FoundationDB on kubernetes through the fdb-kubernetes-operator: v1.28.1.
FoundationDB version: 7.1.43

Kubernetes version

version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.7-gke.500"}

Cloud provider

GCP

[johscheuer]

Thanks for the report, do you think you are able to work on a fix and test for this?

[simenl]

Yes 👍
I'm thinking of reusing the checks from ReplaceMisconfiguredProcessGroups in getPodsToUpdate. Does that seem sound?

[johscheuer]

That sounds reasonable 👍

[johscheuer]

Hello @simenl, just wanted to see how things are going on and if you're able to work on a fix for this?

[simenl]

Hi @johscheuer,
sorry, I got caught up with other things.

I've put up a draft: #1954