connections inside a container shown as going between containers

[monowai]

The container view shows an incorrect link between Rabbit and Riak.

Both services do use Erlang but they are independent containers that are otherwise unaware of each other.

[rade]

Would you mind attaching the report ("</>" button in bottom right corner) for this?

[monowai]

Sure thing
https://gist.github.com/monowai/8c85dd5a1796de5ab7188eebb5dd5036

[rade]

Thanks. I've managed to reproduce this with

docker run -d rabbitmq
docker run -d lapax/riak

Both of these run a separate epmd process inside the container, and the main erlang beam process should connect to that. Based on my quick investigation I reckon the connection from the riak beam is mis-attributed as going to the rabbitmq epmd instead of the riak epmd.

And I've just reproduced the same kind of mis-attribution with two alpine containers that each run

nc -l -p 1122 &
nc 127.0.0.1 1122

Thanks for reporting this.

[rade]

@paulbellamy's and my theory here is that connection endpoints in containers are generally just identified by IP and port. That is fine (and indeed required) for overlay networks, but clearly wrong for localhost connections inside a container.

So a (hopefully) quick fix would be to include the container id in the identity of localhost connection endpoints.

[2opremio]

@paulbellamy's and my theory here is that connection endpoints in containers are generally just identified by IP and port. That is fine (and indeed required) for overlay networks, but clearly wrong for localhost connections inside a container.

We also use PIDs for persistent connections (like this ones). proc-based tracking (which provides PIDs) and conntrack-based tracking run in parallel, and it could be that we don't correctly prioritize proc-tracked connections.

Also, even if the proc-tracked connections were used, we would probably be able to reproduce this with short-lived (conntrack-based) connections, so we need to handle the loopback interface specially.

[2opremio]

Actually, host-scoping aside, the problem adheres to the theory from @rade and @paulbellamy

After reproducing the problem in the way suggested by @rade :

[...] two alpine containers that each run

nc -l -p 1122 &
nc 127.0.0.1 1122

I inspected the Endpoint topology and found:

vagrant-ubuntu-wily-64 is the hostname of my (docker host) machine. So, connections are keyed with the hostname/ip/port, which, for loopback container connections is not good enough.

This causes a key-clash between the processes listening on 127.0.0.1:1122 making the containers fight for the PID entry in the LatestMap and always causing a connection across the containers (both clients are identified as talking to a single server).

Two possible solutions are:

• Append the net namespace inode to Endpoint node key
• Append the PID to the Endpoint node keys (the namespace inode is only available when we know the PID anyways).

Also, in case it's not done already, we should make sure that loopback interface IPs are discarded when tracking short-live connections since they cannot be uniquely attributed to a container (related #1260)

[2opremio]

Two possible solutions are:

• Append the net namespace inode to Endpoint node key
• Append the PID to the Endpoint node keys (the namespace inode is only available when we know the PID anyways).

@rade pointed out offline that we can only do it for loopback connections (since the PID/namespace scope won't match for connections across hosts)