Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

operator-controller should not uninstall/rollback a failed install/upgrade #994

Closed
Tracked by #950
joelanford opened this issue Jul 1, 2024 · 4 comments · Fixed by #1020
Closed
Tracked by #950

operator-controller should not uninstall/rollback a failed install/upgrade #994

joelanford opened this issue Jul 1, 2024 · 4 comments · Fixed by #1020
Assignees

Comments

@joelanford
Copy link
Member

Right now, operator-controller is using helm-operator-plugins, which automatically uninstalls failed installations and rolls back failed upgrades.

This is problematic because the installation or upgrade may have progressed to the point of no return. For example:

  • a pre-install/pre-upgrade hook may have run a migration and a subsequent object failed to update.
  • after any hooks run, a deployment may have started that ran a migration, but then subsequent objects fail to update.

In my opinion, a better behavior is to just stop, inform a user of the problem, and require human intervention to resolve and progress.

@joelanford
Copy link
Member Author

I have a change in helm-operator-plugins that adds a knob to turn this behavior off. From a PoC standpoint, this looks very straightforward.

@joelanford
Copy link
Member Author

PR up to make it possible for operator-controller to configure this behavior in helm-operator-plugins: operator-framework/helm-operator-plugins#348

@bentito
Copy link
Contributor

bentito commented Jul 5, 2024

I think some high-frequency environments, like particularly those managed by SD, are going to want the simplicity of the default Helm rollback behavior. As long as we leave the knob for them to allow automatic rollback on failed upgrade, that should be enough.

@joelanford
Copy link
Member Author

I'm not sure I agree. I think the important thing for a managed service SRE user is that:

  1. The failure is apparent
  2. There is a way to centrally apply a fix and have that fix propagate to the affected installations/upgrades.

If OLM automates rollbacks, it could make an SRE's job harder because it could make the problem worse (as described in the description), and it could make troubleshooting more difficult because logs/metrics/object status in the failed release would be wiped out by the rollback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants