Scale down operation fails and is never requeued if getReplicasForPod
fails transiently
#707
Labels
Milestone
getReplicasForPod
fails transiently
#707
Bug Report
What did you do? / Minimal Reproducible Example
We first deployed a five-replica Solr cluster, then modified the CR to scale down to three replicas.
The scale down triggers the solr-operator to perform the ManagedScaleDown operation, invoking the
handleManagedCloudScaleDown
function. Inside the function, the operator invokes theevictSinglePod
function. If the call to the Solr transiently fails (due to transient network issue or transient unavailability of Solr), the operator writes an error message and never retries. Even if the transient fault is resolved later, the operator is stuck and never perform the scale down operation.To reproduce this bug,
replicas: 5
toreplicas:3
in the CR to initiate the scale down.What did you expect to see?
The operator should be able to scale down the cluster after the transient faults go away
What did you see instead?
The operator never requeues the request
Root Cause
The root cause of this bug is at https://github.com/apache/solr-operator/blob/b2c951052828f88e9fc415f54aec189b805117e9/controllers/solr_cluster_ops_util.go#L251C57-L251C71
where the operator does not handle the error returned by
evictSinglePod
but just directly returns normally.The fix should be simple to add an error handling branch to requeue the request.
The text was updated successfully, but these errors were encountered: