Rollout does not progress current stepIndex when "stable" RS is gone

[jessesuen]

I intentionally borked a rollout by deleting the stable replicaset in the middle of an update.

The rollout ended up in the following condition:

Name:            canary-demo
Namespace:       jesse-test
Status:          ◌ Progressing
Strategy:        Canary
  Step:          0/2
  SetWeight:     1
  ActualWeight:  100
Images:          argoproj/rollouts-demo:yellow (canary)
Replicas:
  Desired:       2
  Current:       2
  Updated:       2
  Ready:         2
  Available:     2

NAME                                     KIND        STATUS         AGE    INFO
⟳ canary-demo                            Rollout     ◌ Progressing  170m
├──# revision:11
│  └──⧉ canary-demo-6758949f55           ReplicaSet  ✔ Healthy      2m33s  canary
│     ├──□ canary-demo-6758949f55-fdzm5  Pod         ✔ Running      2m33s  ready:1/1
│     └──□ canary-demo-6758949f55-kmmtr  Pod         ✔ Running      2m33s  ready:1/1
├──# revision:10
│  └──⧉ canary-demo-645d5dbc4c           ReplicaSet  • ScaledDown   17m
└──# revision:9
   └──⧉ canary-demo-877894d5b            ReplicaSet  • ScaledDown   18m

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"argoproj.io/v1alpha1","kind":"Rollout","metadata":{"annotations":{},"name":"canary-demo","namespace":"jesse-test"},"spec":{"replicas":2,"revisionHistoryLimit":2,"selector":{"matchLabels":{"app":"canary-demo"}},"strategy":{"canary":{"steps":[{"setWeight":1},{"pause":{}}]}},"template":{"metadata":{"labels":{"app":"canary-demo"}},"spec":{"containers":[{"image":"argoproj/rollouts-demo:blue","imagePullPolicy":"Always","name":"canary-demo","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"resources":{"requests":{"cpu":"5m","memory":"32Mi"}}}]}}}}
    rollout.argoproj.io/revision: "11"
  creationTimestamp: "2019-11-22T23:15:13Z"
  generation: 102
  name: canary-demo
  namespace: jesse-test
  resourceVersion: "41606712"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/jesse-test/rollouts/canary-demo
  uid: efd8a9fc-0d7d-11ea-b443-42010aa80075
spec:
  replicas: 2
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: canary-demo
  strategy:
    canary:
      steps:
      - setWeight: 1
      - pause: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: canary-demo
    spec:
      containers:
      - image: argoproj/rollouts-demo:yellow
        imagePullPolicy: Always
        name: canary-demo
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 5m
            memory: 32Mi
status:
  HPAReplicas: 2
  availableReplicas: 2
  blueGreen: {}
  canary:
    stableRS: 644d97bdb6
  conditions:
  - lastTransitionTime: "2019-11-23T01:46:50Z"
    lastUpdateTime: "2019-11-23T01:46:50Z"
    message: Rollout has minimum availability
    reason: AvailableReason
    status: "True"
    type: Available
  - lastTransitionTime: "2019-11-23T02:03:34Z"
    lastUpdateTime: "2019-11-23T02:03:40Z"
    message: ReplicaSet "canary-demo-6758949f55" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  currentPodHash: 6758949f55
  currentStepHash: 566d47875b
  currentStepIndex: 0
  observedGeneration: bffd6db9
  readyReplicas: 2
  replicas: 2
  selector: app=canary-demo
  updatedReplicas: 2

Notice that the rollout has a "stable" RS of 644d97bdb6. This RS does not exist anywhere because I deleted it.

When it was in this condition, any updates to the rollout will skip all steps, which is actually the right behavior, because there is nothing "stable" anymore that exists. The problem is, that the currentStepIndex remains at 0, which means the rollout is forever Progressing.

[jessesuen]

To reproduce:

1. deploy a rollout (version 1)
2. update rollout to version 2, and get it to a paused step
3. delete the stable rs
4. update the rollout to version 3

After step 4, the rollout is in a state where it will always skip all steps, but will always be progressing.

Workaround for this is to run:

kubectl argo rollouts promote canary-demo --skip-all-steps

This will bump the current step index to the last step.