diff --git a/keps/238-controller-revision/README.md b/keps/238-controller-revision/README.md new file mode 100644 index 00000000..1494f401 --- /dev/null +++ b/keps/238-controller-revision/README.md @@ -0,0 +1,393 @@ +# KEP-238: Controller-Revision + + + + + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [LWS controller](#lws-controller) + - [Case 1: No exsting controller revisions in history](#case-1-no-exsting-controller-revisions-in-history) + - [Case 2 & 3: Regular create and update](#case-2--3-regular-create-and-update) + - [Pod Controller](#pod-controller) + - [Controller Revision Implementation](#controller-revision-implementation) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + + +## Summary + + + +This KEP aims to add controller revision to store previous states of the LWS object. +It fixes bug #238, and opens the road for future rollback support. + +## Motivation + + +If a replica is restarted during rolling update, and the replica hasn't been updated +yet, the worker pod spec that is used is the updated one, while the leader pod spec +is the original one. We can fix this by storing the worker pod spec using controller revision. + +Another issue is that when upgrading between different LWS controller versions, a rolling update is triggered, even when the deployed LWS object hasn't changed. This can be fixed by storing the +LWS spec in the revision, then applying the patch and making a semantic comparison, instead of a +string one like it is done currently when computing the template hash. + +### Goals + + + + - Stores the state of the LWS object in order to use the correct version when recreating a replica. + - Change the mechanism with which we determine whether an LWS object has changed. + + +### Non-Goals + + + +This KEP does not add rollback support, though the design takes into account that it will be +added in the future. + +## Proposal +We propose adding controller revision to the LWS controller. This allows us to store previous +iterations of the LWS object, in order to make it possible to recreate pods with the right pod +spec when restarted during rolling update, and be able to compare different LWS objects to determine if the spec has changed. + + + +### User Stories (Optional) + + + +#### Story 1 + +#### Story 2 + +### Notes/Constraints/Caveats (Optional) + + + +### Risks and Mitigations + + + +## Design Details +There are a few constraints that need to be taken into account in the design: +- The label `templateHash` is not a reliable way to determine whether or not an LWS object has been updated, as seen in #281 +- The same LWS object can generate two different controllerRevision names + - For instance, everytime the LWS is queried, it will have different `lws.ResourceVersion`, causing two revisions to be named differently even if they have the same LWS object +- Adding a new default pod label will trigger rolling update when updating from a different LWS controller version, so no new pod labels can be added. + +Taking those into account, we'll combine controller revisions and template hashes. Each controller revision will have a templateHash as a label, essentially creating a map of revisions with template hashes as keys. + + +### LWS controller +In order to fix #281, we will use the hash key in the leaderSts to make a controller revision lookup, and get the LWS object that was used to create it. To determine if an update has happened, we'll compare the fields that are used to generate the template hash. + + +```golang +func templateUpdated(sts, lws) bool { + controllerRevision := GetLeaderWorkerSetRevisionFromTemplateHash(sts.Labels[templateHash]) + baselineLws:= controllerutils.ApplyRevision(lws, controllerRevision) + return !utils.EqualLeaderWorkerTemplates(baselineLws, lws) +} +``` + +There are three cases where we need to create a new controllerRevision: +- There are no existing controller revisions in the history +- A new LWS object is created +- An update has been made to the LWS object + +#### Case 1: No exsting controller revisions in history +This is the case when upgrading from a version that did not have controller revisions. Since a controller revision is needed to determine whether or not an update has happened, it will be created in `rollingUpdateParameters`, so that way an update isn't triggered until a controller revision has been created. + +```golang +func rollingUpdateParameters() { + if !existingControllerRevisions { + createLeaderWorkerSetRevision(lws, leaderSts.labels[templateHash]) + return + } + + if templateUpdated(leaderSts, lws) { + // triggers the rolling update + } +} +``` + +#### Case 2 & 3: Regular create and update +Because an update can be triggered by changing the value of a label, what templateHash value is used when building the leader sts also has to be safeguarded. + +```golang +func SSAwithStatefulSet(lws, partition, lws) { + templateHash := utils.LeaderWorkerTemplateHash(lws) + if leaderStsExists && !templateUpdated(leaderSts, lws){ + templatehash := leaderSts.Labels[templateHash] + } + createLeaderWorkerSetRevision(lws, templateHash) + constructLeaderStatefulSetApplyConfiguration(lws, partition, replicas, templateHash) +} +``` + +Once the update has been determined to be done, we'll reset the history to only have the revision of the current LWS object in the history. + +```golang +func updateConditions() { + if updatedAndReadyCount == int(*lws.Spec.Replicas) { + conditions = append(conditions, makeCondition(leaderworkerset.LeaderWorkerSetAvailable)) + truncateHistory() + } +} +``` + +### Pod Controller +Now that there is a map between the template hash of the leader and controller revision, a lookup can be done to select the worker pod spec that will be used. + + +```golang +func Reconcile() { + controllerRevision := GetLeaderWorkerSetRevisionFromTemplateHash(pod.Labels[templateHash]) + constructWorkerStatefulSetApplyConfiguration(controllerRevision) +} + +func constructWorkerStatefulSetApplyConfiguration(currentRevision) { + currentLws := controllerutils.ApplyRevision(lws, controllerRevision) + podTemplateSpec := **currentLws.WorkerTemplate.DeepCopy() +} +``` + +### Controller Revision Implementation +A new package will be created for the functions that interact with controllerRevision, similar to how it is done in [upstream](https://github.com/kubernetes/kubernetes/blob/cb31e42b85d0a1e2be6639ecd4f7b9414887552a/pkg/controller/history/controller_history.go). + +Functions for creating patches, applying revisions, and truncating history will be added to `controller_utils` since both the pod and lws controllers need to access these functions. + +We'll store the whole LWS spec to make it easier to compare LWS objects when determining if it has been updated. + +In order to ensure that there only exists one revision per templateHash, two controller revisions will be determined to be equal if they have the same template hash label. + + + + + +### Test Plan + + + +[X] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests +- Test controller revision implemenation functions + + + + +##### Integration tests + +Test that currentRevision, updatedRevision, and collisionCount work as intended. + +Test that worker pod is recreated with the correct pod template if restarted during rolling update. + + + + + + +##### e2e tests + + + + +### Graduation Criteria + + + +## Implementation History + + + +## Drawbacks + + + +## Alternatives +An implementation similar to the one done for StatefulSet was considered. In this implementation, we would create +a label to store what controller revision was used to generate the pods, and replacing the label `leaderworkerset.sigs.k8s.io/template-revision-hash`. +However, this requires updating the value of those labels (or adding a new label alltogether). Doing this, triggers a rolling update when upgrading +from an LWS version that does not have controller revision. + +Moreover, there are features that need to trigger a rolling update (e.g NetworkConfig). In order to replicate this behavior when using +a label to store controller revision, NetworkConfig must be part of the controller revision patch. However, this would mean that NetworkConfig would +also be rolled back. + + \ No newline at end of file diff --git a/keps/238-controller-revision/kep.yaml b/keps/238-controller-revision/kep.yaml new file mode 100644 index 00000000..28475c33 --- /dev/null +++ b/keps/238-controller-revision/kep.yaml @@ -0,0 +1,17 @@ +title: Controller Revision +kep-number: 238 +authors: + - edwinhr716 +status: implementable +creation-date: 2024-11-20 +reviewers: + - kerthcet + - ahg-g +approvers: + - kerthcet + - ahg-g + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v0.5.0"