Make it possible to run multiple che-server in parallel

[l0rd]

Description

This is about starting multiple wsmaster servers (behind a reverse proxy) to serve the same group of workspaces for the purpose of:

• blue-green deployment to minimize downtime during new versions rollout
• load balancing

[skabashnyuk]

Can we consider this issue is related to wsmaster servers of the same version?

[l0rd]

@skabashnyuk this mechanism should work even for wsmaster with different versions except, of course, if there are breaking changes in the API.

What I mean is that the goal is to realease to production many times a day. To achieve it we need blue-green deployment. So for the deployments that do not introduce breaking changes in the API (99% of the deployment) we should be able run 2 wsmaster in parallel and for deployments that do introduce breaking changes (the remaining 1%) we should use different techniques like canary release or pre-release migration jobs (but this epic is not about these!).

[garagatyi]

I like the idea!
FYI this should relate to multi-user packaging only since single-user packaging uses DB embedded into Che instance.

[garagatyi]

Here are issues I see as needed to solve this epic:

• get rid of the volume mounted to /data path of Che master container (not sure whether it contains anything useful or is present only because it was done in that way in Docker-based deployment)
• implement runtime picking up Implement recovery for Kubernetes/OpenShift infrastructures #5919
• rework auth subsystem to be stateless Machine tokens become invalid when workspaces are recovered #7785
• make sure data transactions into DB allow us to run 2 instances of Che
• make workspace activity checking feature stateless
• feature toggle technics for testing blue-green deployments (is this already done for rh-che)

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[l0rd]

/remove-lifecycle stale

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[l0rd]

Reopening. To be revisited once #15425 is completed

[skabashnyuk]

@l0rd can you explain the goal of this task?
Is it suppose to see multiple DevWorksapce controllers in parallel or what? Can we update the title to properly highlight the goals?

[l0rd]

@skabashnyuk no I am still thinking about the che server and the devworkspace controller proxy.

[skabashnyuk]

so then it's about making che-server stateless? Right?

[l0rd]

@skabashnyuk yes

[ibuziuk]

I belive we need to close this issue as outdated since the workspace engine is now DevWorkspace operator - https://newsroom.eclipse.org/eclipse-newsletter/2022/may/eclipse-che-gets-new-dev-environments-engine

[l0rd]

@ibuziuk are we able to do a rolling update of Che without any outage (i.e. users can start a workspace while the che-server is getting updated)?

[ibuziuk]

currently, che deployment is using rolling update

 strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%

the plan is to retest the flow with workspace startup during rollout update once we decommission db and che-server would become stateless - #21374

[ibuziuk]

@l0rd I would propose closing this issue in favor of #22067
Basically, it is possible to run multiple che-server in parallel (demoed on the last sprint review). What we are missing is options for configuring replicas for each operand on the CR level

[l0rd]

@ibuziuk makes sense 👍