Multiple enhancements - leader election, events, time window, delay after upgrading

[brandond]

Add some long-requested or at least long overdue features:

• Add support for running multiple replicas of the controller #336
  Add leader election, to allow multiple replicas of the controller.
• No history of a updated plan #238
  Emit events. Controller events (startup, leader election) are attached to the Node running the controller, Plan events are attached to the Plan.
• Provide optional upgrade window setting to limit when a plan can run #63
  Add support for spec.window to set time window for upgrades.
• Add the ability to specify a waiting time in seconds before upgrading next node #256
  Add support for spec.postCompleteDelay - configurable delay after job completion before node is marked done.

New fields:

• spec.window: TimeWindow from github.com/kubereboot/kured/pkg/timewindow.
  Plan will not start creating Jobs for Nodes outside this time window. If there are already Nodes in progress (status.applying is not empty) when the window closes, the Plan will continue to run until finished.

spec:
  window:
    days:
      - "saturday"
      - "sunday"
    startTime: "8:00"
    endTime: "16:00"
    timeZone: "UTC"

• spec.postCompleteDelay: positive time.Duration string.
  By default, Nodes are marked as done as soon as the Job is complete. When this is set, the Job must be Completed for at least this long before the Node is marked as done via Plan hash label. Job TTLSecondsAfterFinished is set to at least 1 minute longer than this delay, to ensure that the controller gets a chance to see the Job complete.

spec:
  postCompleteDelay: "1m30s"

Example of events:

brandond@dev01:~$ kubectl events -n system-upgrade --for plan/server-plan
LAST SEEN               TYPE     REASON               OBJECT             MESSAGE
7m33s                   Normal   Validated            Plan/server-plan   Plan is valid
7m33s                   Normal   Resolved             Plan/server-plan   Resolved latest version from Spec.Channel: v1.30.6-k3s1
7m32s (x2 over 7m33s)   Normal   Waiting              Plan/server-plan   Waiting for start of Spec.Window to sync Jobs for version v1.30.6-k3s1. Hash: 848b017d8bfbda18dddf2c0405a16c8658da7da2e867e167506e99b1
5m32s                   Normal   SyncJob              Plan/server-plan   Jobs synced for version v1.30.6-k3s1 on Nodes k3s-server-2. Hash: 848b017d8bfbda18dddf2c0405a16c8658da7da2e867e167506e99b1
4m59s                   Normal   JobCompleteWaiting   Plan/server-plan   Job completed on Node k3s-server-2, waiting 1m30s PostCompleteDelay
3m29s                   Normal   JobComplete          Plan/server-plan   Job completed on Node k3s-server-2
3m29s                   Normal   SyncJob              Plan/server-plan   Jobs synced for version v1.30.6-k3s1 on Nodes k3s-server-1. Hash: 848b017d8bfbda18dddf2c0405a16c8658da7da2e867e167506e99b1
2m55s                   Normal   JobCompleteWaiting   Plan/server-plan   Job completed on Node k3s-server-1, waiting 1m30s PostCompleteDelay
85s                     Normal   JobComplete          Plan/server-plan   Job completed on Node k3s-server-1
85s                     Normal   Complete             Plan/server-plan   Jobs complete for version v1.30.6-k3s1. Hash: 848b017d8bfbda18dddf2c0405a16c8658da7da2e867e167506e99b1

brandond@dev01:~$ kubectl events
LAST SEEN   TYPE     REASON    OBJECT             MESSAGE
20m         Normal   Starting  Node/k3s-server-2  system-upgrade-controller 08d5fdf (08d5fdf) starting leader election for system-upgrade/system-upgrade-controller
20m         Normal   Started   Node/k3s-server-2  system-upgrade-controller 08d5fdf (08d5fdf) running as system-upgrade/system-upgrade-controller