Che Docker Launcher - Fails to Recognize Valid Server Boot

[TylerJewell]

The che launcher seems to be unable to confirm that the che-server has booted under certain Linux configurations.

1: I started a VM using this centos7.2 Vagrantfile. The results of docker version are:

Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Wed Apr 27 00:34:42 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Wed Apr 27 00:34:42 2016
 OS/Arch:      linux/amd64

and its ifconfig command (trimmed for readability):

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.42.1  netmask 255.255.0.0  broadcast 0.0.0.0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.56.110  netmask 255.255.255.0  broadcast 192.168.56.255

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0

I confirmed that all IP addresses are pingable from within che-launcher.

2: When launching che che-launcher with the following syntax, the launcher will timeout as saying the that server is not booted. However, the che-server is successfully launched and can be accessed from outside the VM.

docker run -v /var/run/docker.sock:/var/run/docker.sock \ 
                   -CHE_PORT=7000 codenvy/che-launcher:nightly start

3: Upon inspection, it seems that the che-launcher is unable to curl the server running in another container. I tried a variety of configurations. But essentially:

$ docker run -v /var/run/docker.sock:/var/run/docker.sock 
  -e CHE_PORT=7000 -it --entrypoint=/bin/sh codenvy/che-launcher:nightly

# Inside the che-launcher container:
$  docker run -d --name che-server -v /var/run/docker.sock:/var/run/docker.sock \
    -v /home/user/che/lib:/home/user/che/lib-copy \
    -p 7000:8080 --user=root \
    codenvy/che-server:nightly -s:uid run

### All fail with "host is unreachable"
$ curl -I http://10.0.2.15:7000/api/
$ curl -I http://172.17.42.1:7000/api/
$ curl -I http://192.168.56.110:7000/api/

### Fail with "Failed to connect - connection refused"
$ curl -I http://172.17.0.2:7000/api/

4: And the details of docker inspect che-server

        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "123b45110f99bc78a91ae05fd25f55dc70355a9215ffdba5e3994eed2b198cd7",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "8000/tcp": null,
                "8080/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "7000"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/123b45110f99",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "6cb9ce567cae759a8398649d7262d89a0ee86adb6c39baec0982cc74a3f135e7",
            "Gateway": "172.17.42.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "15a611ac33906aba27ad40135c535eb70dd46ac0633d11c08092b7a67cdff361",
                    "EndpointID": "6cb9ce567cae759a8398649d7262d89a0ee86adb6c39baec0982cc74a3f135e7",
                    "Gateway": "172.17.42.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02"
                }
            }
        }

5: And the details of docker inspect che-launcher.

        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "cd8544ec894b08abc98545960f2e1c91b38780e6086a1127ce0934f06be7e97a",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/cd8544ec894b",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "6ee7da15fefcca9126e4cc574f46b84db25d6cc2bc48679095e0e03be5fd48d2",
            "Gateway": "172.17.42.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.1",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:01",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "15a611ac33906aba27ad40135c535eb70dd46ac0633d11c08092b7a67cdff361",
                    "EndpointID": "6ee7da15fefcca9126e4cc574f46b84db25d6cc2bc48679095e0e03be5fd48d2",
                    "Gateway": "172.17.42.1",
                    "IPAddress": "172.17.0.1",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:01"
                }
            }
        }

[TylerJewell]

@benoitf @l0rd

[TylerJewell]

Update - There seems to be some sort of firewall or port routing issues. If I manually start / stop firewalld, to verify that it is not running and then re-enter the che-launcher container, all curl commands are successful.

Chasing this down further, there is this very long thread on Docker's site about firewalld and Docker.

I started a new VM that does not have Docker installed. I disabled iptables, disabled firewalld, and then installed Docker from scratch on the VM.

In this scenario, the che-launcher was able to launch successfully.

So, I think this is turning into a documentation issue about how to get Docker working with firewalld. That is clearly the culprit.

[l0rd]

That's annoying :-( Have you tried to to ping the che-server using the Docker internal IP (docker inspect -f='{{.NetworkSettings.IPAddress}}') and the internal port (8080) too:

docker exec -ti che-server curl -I http://$(docker inspect -f='{{.NetworkSettings.IPAddress}}' che-server):8080/api

Besides the documentation we could also infer the distribution Che is running on and inform users about that issue in the launcher output. Wdyt?

[TylerJewell]

@l0rd - agree that it is annoying. Yes, I tried every IP address combination. I spent a good couple hours looking at ifconfig of the che-launcher, che-server, and the host VM.

The issue is really just firewalld, and the docs on Docker's site do not make it really clear on what has to happen in order for the firewalld settings to be properly set so that this conflict does not exist.

On my walk this afternoon with my dog, I was thinking about whether we wanted to notify users of particular distribution issues. There are two that I am aware of:
1: If che-launcher AND boot2docker AND windows AND CHE_DATA_FOLDER != %userprofile% then CHE_DATA_FOLDER value will be ignored. This happens because boot2docker on windows only mounts %userprofile%. Any other directory is ignored.

2: If centos / rhel 7.x AND firewalld or iptables alive, then we will not be able to verify that the server is booted. However, the che-server will boot and work properly.

[ghost]

@TylerJewell I am stuck at server booting stage. I checked logs in the server container and the server is up. I could also access Che as the specified port on localhost.

I stopped all containers, disabled firewald sudo ufw disable and it fixed it. Also, you may allow a particular port - sudo ufw allow 7000

[l0rd]

@TylerJewell and @eivantsov I've tested on a CentOS VM and I confirm that the che-launcher cannot connect to the che-server using its external URL (e.g. 10.0.2.15:7070). But if I use the internal URL (e.g. 172.17.42.1:8080) I could connect successully. Can you please try it on your VM:

docker run -ti --rm --entrypoint=curl codenvy/che-launcher:nightly -I http://$(docker inspect -f '{{.NetworkSettings.IPAddress}}' che-server):8080/api/

If that command works fine on your environment too we could fix the launcher accordingly.

[ghost]

@l0rd I was to quick to speak. With ufw enabled Che server could not ping ws agent:

2016-07-26 11:22:44,872[kspaceManager-0] [ERROR] [.c.a.m.s.w.WsAgentLauncherImpl 106] - Fail pinging ws agent. Workspace ID:workspacesqvr1venflpnhn4b. Url:http://172.19.20.16:32801/wsagent/ext/. Timestamp:{}

When I disable firewald, the problem's gone.

[TylerJewell]

@l0rd - Your solution worked for me. I turned on firewalld. I was able to run your command with the internal IP + port 8080 and was able to get a successful ping. However, if I attempted to try the external IP and port, then it was host rejected.

Will you add this fix into one of your commits for another issue? I am not sure of the solution that you want to pursue. Are you thinking that the curl command will always test the internal IP + port, or is it situational?

[l0rd]

I would always use the internal IP/port for the curl command. It should work for every Linux distribution. No problem to commit that change myself.

[TylerJewell]

@eivantsov - I have included a fix for the curl issue in this pull request.
#1957