Docker Launcher: Websockets are unable to comunicate with the wsagent when the server and the browser run on different networks

[l0rd]

In scenarios where a wsagent has two different IP addresses:

• An internal IP address that is used by Che server
• An external IP address that is used by the websockets in the browser

the creation of a new workspace and project fails.

I could reproduce that using Docker for Mac but the scenario should be reproducible wherever you have an internal/external IPs (e.g. amazon EC2). There is a workaround (rerouting the traffic to the internal IP address to the external IP address on the Docker host) and it's described here. This issue is about removing the need to use this workaround and is related to #1482.

Reproduction Steps:

1. Run Che inside a container using Docker for Mac without setting the lo0 alias
2. Access the dashbord using the URL http://localhost:8080
3. Create a new workspace and project

Expected behavior:

The workspace and project should be created successfully. The websocket should use the URL in the address bar to contact the wsagent (with a different port).

Observed behavior:

The creations hangs after the wsagent is started:

[STDOUT] 2016-07-04 15:40:41,950[main] [INFO] [o.a.catalina.startup.Catalina 642] - Server startup in 39359 ms
[STDOUT] 2016-07-04 15:40:43,204[nio-4401-exec-4] [WARN ] [i.ContainerResponse$BodyWriter 90] - Client has aborted connection. Response writing omitted.

And after a few minutes a popup message with the following error:

Workspace Connection Error
It seems that your workspace is running, but we cannot connect your browser to it. This commonly happens when Che was not configured properly. If your browser is connecting to workspaces running remotely, then you must start Che with the --remote: flag where the is the IP address of the node that is running your Docker workspaces.Please restart Che with this flag. You can read about what this flag does and why it is essential at: https://eclipse-che.readme.io/docs/configuration#envrionment-variables

Opening the developers tools in Chrome I could see that the websocket is tring to connect to an host (192.168.65.2) that is not accessible from the Mac OS and that is different from the host I've put in the brwoser address bar (localhost)

Che version: 4.4.0-RC1-SNAPSHOT
OS and version: Mac 10.11.5
Docker version:

Client:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   906eacd
 Built:        Fri Jun 17 20:35:33 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   a7119de
 Built:        Wed Jun 29 10:03:33 2016
 OS/Arch:      linux/amd64
 Experimental: true

Che install: Docker container

Additional information:

• Problem started happening recently, didn't happen in an older version of Che: Yes
• Problem can be reliably reproduced, doesn't happen randomly: Yes

[ghost]

@l0rd browser and Che server use the same URL to reach ws-agent.

Do you propose a different mechanism? Smth like being able to set these endpoints separately for server and client (browser)? Or the browser should use URL currently in the address bar (localhost in your case)?

[ghost]

@l0rd normally, setting CHE_DOCKER_MACHINE_HOST is enough to run Che on a remote Linux machine.

[TylerJewell]

Also, we have started working on a way to proxy all requests through a single nginx proxy. This work will be done quickly and would be a way to hide the server and workspaces on different interfaces and ip addresses with nginx being external. It would be simpler overall, right?

[ghost]

@TylerJewell I believe @l0rd means smth slightly different. While nginx solves the problem of ports exposure, it does not seem to solve host problem where Che server and browser use the same URL to reach ws-agent.

@garagatyi should have his say here.

[TylerJewell]

Yeah I understood his configuration, but I am questioning if we should consider the use case and instead ask agent ip to be consistently applied inside the workspace and then allow nginx to handle the nuances of the internal mapping.

I totally understand that he wants to address the docker for Mac scenario where nginx would apply, but just question the scenario generally.

[l0rd]

From the point of view of a user the URL on the address bar is the perfect
choice because there is no need to specify another parameter when starting
Che. And it makes even more sense if you plan to use a reverse proxy.

Your solution @TylerJewell would work too but is more complicated imho.

By the way have you looked at traefikproxy? That's the modern reverse proxy
for containers and is developed by my friend @emilevauge :-)

On Monday, 4 July 2016, Tyler Jewell [email protected] wrote:

Yeah I understood his configuration, but I am questioning if we should
consider the use case and instead ask agent ip to be consistently applied
inside the workspace and then allow nginx to handle the nuances of the
internal mapping.

I totally understand that he wants to address the docker for Mac scenario
where nginx would apply, but just question the scenario generally.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1644 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAlC779uMwGBdt9fdx5KZxy4qt6um2E8ks5qSTYDgaJpZM4JEfO-
.

[TylerJewell]

@skabashnyuk - check out this reverse proxy. It is very cool. Perfect for Che.

Mario - the work that we need for setting up reverse proxy is internal to Che. Inside of Codenvy we have a mechanism to allow the workspace ip and ephemeral ports to be embedded into the track back URL that the Che server passes back to the browser. We are going to update Che with this logic so that users can have it activated. If activated then the browser will need to talk to a proxy that can decode it.

[garagatyi]

I don't think that having different URL for apps in the containers for Che master and browser is easier than proxy.
@l0rd Do you have another suggestion how to achieve that except usage of proxy?
I think you've told us about possibility of usage overlay network for that. Am I write? How do you see that?

[l0rd]

I'm not sure that a reverse proxy alone would resolve this particular issue. Let's imagine that we have setup a reverse proxy in front of Che:

• When I open the browser to load the Che dashboard it works as expected:

Browser => localhost:8080 (Reverse Proxy) => 192.168.65.2:8080 (Che server)

• When the websocket try to contact the wsagent it fails because it uses the bad address:

websocket => 192.168.65.2:33041 (ERROR: Unable to connect)

The websocket is configured to connect to CHE_DOCKER_MACHINE_HOST but it's not able to ping that IP address. This is reproducible with Docker for Mac but imo that's not the only scenario.

There are two way to address that issue:

• The websocket should use the same URL that's in the browser address bar
• The websocket should use an URL that is provided as a property/env variable when starting Che

Overlay network is a distinct subject: it simplifies communications between server and agent in distributed environments (there is no need to retrieve IP addresses anymore) but do not simplify the communication to/from the browser (that would be outside the overlay network).

[bmicklea]

@benoitf please help @l0rd with the JS and UD impacts.

[JamesDrummond]

Can we remove the milestone and add sprint/next-sprint?