Workspaces websockets not working properly on AWS and EKS Kubernetes clusters using nginx as reverse proxy

[bmboucher]

Describe the bug

I've been running 7.20.0 for a while, after upgrading to 7.27.2 I can't get the websockets endpoints to connect. The dashboard and the REST endpoints are available (I can successfully test them through Swagger), but the dashboard shows an error on loading and the console shows a bunch of websocket-related errors:

https://issues.redhat.com/browse/CRW-2082

Che version

• latest
• nightly
• other: 7.27.2

Steps to reproduce

Fails on first loading the dashboard after logging in.

Runtime

• kubernetes
  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-eks-49a6c0", GitCommit:"49a6c0bf091506e7bafcdb1b142351b69363355a", GitTreeState:"clean", BuildDate:"2020-12-23T22:10:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

• chectl
• Helm
• OperatorHub
• I don't know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• Dev Sandbox (workspaces.openshift.com)
• other: please specify

Eclipse Che Logs

che-dev.zip

Release Notes Text

A bug that prevented workspaces websockets to work properly on AWS and EKS Kubernetes clusters using nginx as reverse proxy has been fixed.

[benoitf]

looks like a duplicate of #19403

[sleshchenko]

WebSocket failure should not lead to the share state.
@bmboucher could you check other failed requests on Network tab of Developer Tools?

[bmboucher]

WebSocket failure should not lead to the share state.
@bmboucher could you check other failed requests on Network tab of Developer Tools?

I don't see any other failed requests; when I load the dashboard page I get some 304/302 redirect responses and a couple of 200 responses whenever it hits the "/api" endpoints. It's just websockets that is not connecting. I've confirmed that our older deployment (7.20.0) on similar infrastructure does establish the ws connection just fine.

[bmboucher]

UPDATE: I reverted to version 7.26.0 and it fixed the "unable to get available storage types" error; I think that one is coming from #19403 as @benoitf indicated. I do have the same issue with websockets, but now I see the expected error notification at the top of the page:

[sleshchenko]

I've confirmed that our older deployment (7.20.0) on similar infrastructure does establish the ws connection just fine.

@bmboucher that's strange.
WebSocket is finished just after it's established.
Could you share websocket frames? Anything interesting in Che Server logs?

[bmboucher]

OK, I figured it out - first off, apologies that I was a little inaccurate in my problem description. The networking setup was not identical between the instance that was working and the broken one as I had thought - we had moved to a new nginx deployment and that was either confused about where to route WS traffic or whether it was enabled at all.

The solution was to add the following annotation to our che-ingress:

nginx.org/websocket-services: che-host

We installed from the Helm chart which doesn't contain this annotation so you might want to consider adding it; I'm not sure when NGINX began requiring it or if chectl would have solved the problem.

[tolusha]

@bmboucher
Thank you for investigation.
Some info about nginx.org/websocket-services annotation
nginx/kubernetes-ingress#322

[tolusha]

Fixed by eclipse-che/che-server#54

[kphilly1]

Do you when the next Docker image version of Che (minor/micro), which includes these fixes, will be published? Looks like it's close based on the latest activity here. If not today or tomorrow, is there documentation on how we can build the affected Docker images locally (with code from 'main' branch)? Thank you very much.

[tolusha]

The fix is included in the next version of Eclipse Che.
You can deploy it using chectl installed from the next channel.

[kphilly1]

Thanks for your reply @tolusha . So we can pull the 'next' version of the Docker images for the various Che components from quay.io?

https://quay.io/repository/eclipse/che-server?tab=tags

If so, we can try pulling the 'next' version and trying. Thanks again!

[tolusha]

@kphilly1
yes, you can

[kphilly1]

Thanks @tolusha !