fix: rolling deployments race

[stuartwdouglas]

If we delete the deployments/runners straight away it may still be in some controllers route tables. By adding a small delay we make sure that all the controllers will have updated their table.

This is a pretty nasty hack, but will likely be temporary.

fixes: #2789

[deniseli]

Should we open an issue to track undoing of this, or no need since you already have a longer running plan to fix?

[stuartwdouglas]

I will open an issue, I realized that it is bigger than just this case, it also affects things like subscriptions where one controller can decide the deployment is ready and activate message delivery, and another controller can then start processing messages before it's route table is updated.

[alecthomas]

I think this will all get resolved by the provisioner. eg. the provisioner won't provision subscriptions until routing tables are provisioned

[stuartwdouglas]

Currently I don't start subscriptions until route tables are provisioned, however it is local to a single controller, so we don't have a way of knowing that all route tables in all controllers are updated.

[alecthomas]

Right, but I think as part of this provisioning change we'll need to ensure that routing tables are actually propagated.

[alecthomas]

What I mean is, each thing created by the provisioner will need to guarantee that it's fully propagated before it's counted as provisioned.