From d2ca4983efee570b45601fb01bf69ea17b0b0610 Mon Sep 17 00:00:00 2001 From: Alvaro Aleman Date: Mon, 3 Feb 2025 11:58:18 -0500 Subject: [PATCH] Remove simplistic advice about multiple controllers reconciling same CR This advice is simplyfing things and making an "It depends" situation look like there was a clear good and a clear bad way that is the same in all situations. Pretty much none of the issues stated will get better if each controller gets its own CR: * Race conditions: Conflict errors can always happen and all controllers need to be able to deal with them. If a full reconciliation is too expensive, they can use something like `retry.OnConflict` * Concurrency issues with different interpretations of state: This example sounds like just buggy software. Copying the state to a new CR doesn't eliminate this problem * Maintenance and support difficulties: This is definitely not going to get any better by adding more CRDs into the mix, if anything, it will get more complicated * Status tracking complications: This is why conditions exist and Kubernetes api guidelines explicitly state that controllers need to ignore unknown conditions: `Objects may report multiple conditions, and new types of conditions may be added in the future or by 3rd party controllers.`, [ref][0] * Performance issues: If multiple controllers do the same thing, that is a bug regardless of all other considerations and can easily lead to correctness and performance issues. The `workqueue` locks items while they are reconciled to avoid exactly that, but obviously it doesn't work cross-controller To illustrate the situation, think about the `Pod` object, in the lifecycle of a pod we usually have at least cluster-autoscaler, scheduler and kubelet. Making cluster-autoscaler act on a `PodScaleRequest` and scheduler on a `PodScheduleRequest` would be a complication, not a simplification. [0]: https://github.com/kubernetes/community/blob/322066e7dba7c5043071392fec427a57f8660734/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties --- docs/book/src/reference/good-practices.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/docs/book/src/reference/good-practices.md b/docs/book/src/reference/good-practices.md index 98dedc31f9e..c29e0a31d10 100644 --- a/docs/book/src/reference/good-practices.md +++ b/docs/book/src/reference/good-practices.md @@ -45,16 +45,6 @@ Having one controller manage many Custom Resources (CRs) in an Operator can lead In conclusion, while it might seem efficient to have a single controller manage multiple CRs, it often leads to higher complexity, lower scalability, and potential stability issues. It's generally better to adhere to the single responsibility principle, where each CR is managed by its own controller. -## Why is it recommended to avoid a scenario where multiple controllers are updating the same Custom Resource (CR)? - -Managing a single Custom Resource (CR) with multiple controllers can lead to several challenges: -- **Race conditions**: When multiple controllers attempt to reconcile the same CR concurrently, race conditions can emerge. These conditions can produce inconsistent or unpredictable outcomes. For example, if we try to update the CR to add a status condition, we may encounter a range of errors such as “the object has been modified; please apply your changes to the latest version and try again”, triggering a repetitive reconciliation process. -- **Concurrency issues**: When controllers have different interpretations of the CR’s state, they may constantly overwrite each other’s changes. This conflict can create a loop, with the controllers ceaselessly disputing the CR’s state. -- **Maintenance and support difficulties**: Coordinating the logic for multiple controllers operating on the same CR can increase system complexity, making it more challenging to understand or troubleshoot. Typically, a system’s behavior is easier to comprehend when each CR is managed by a single controller. -- **Status tracking complications**: We may struggle to work adequately with status conditions to accurately track the state of each component managed by the Installer. -- **Performance issues**: If multiple controllers are watching and reconciling the Installer Kind, redundant operations may occur, leading to unnecessary resource usage. -These challenges underline the importance of assigning each controller the single responsibility of managing its own CR. This will streamline our processes and ensure a more reliable system. - ## Why You Should Adopt Status Conditions We recommend you manage your solutions using Status Conditionals following the [K8s Api conventions][k8s-api-conventions] because: