docker is not mapping dynamic ports

[mmlb]

Following the getting started guide I expected to see the dynamic redis ports mapped, but docker port returns empty. Possibly related to fsouza/go-dockerclient#220

[markpollack]

I was not able to connect to the redis server using the redis-cli (tried the obvious port numbers) when following the getting started guide - how can I determine the port? netstat didn't help much. (using ubuntu 14.04)

[purehyperbole]

I'm also hitting this issue, netstat returns nothing on the docker host.

[purehyperbole]

Just leaving this here for reference:

It turns out the container port is accessible, it's just not mapped back to loopback on the docker host. You can find the correct IP by inspecting the docker container using this command (you have to run this on the host your container is running on).

`$ docker inspect CONTAINER_ID | grep IPAddress``

You should then be able to connect using that IP/Port combination.

It would be nice to be able to get this information from the nomad CLI :)

[ghost]

+1 This option is not functioning properly, setting dynamic_ports["redis"] or dynamic_ports["6379"] would expect to map 6370 from the docker host through to the redis container however this is not happening on either localhost or the public ip of the docker/nomad host.

Tested on CentOS 6, standard 'nomad init' example with either dynamic_ports setting on a 3 host nomad cluster with a server/client and 2 client slaves:

[core@bastion ~]$ ansible 10.42.0.60 -m shell -a "netstat -pan | grep LISTEN | grep tcp"
10.42.0.60 | success | rc=0 >>
tcp        0      0 10.42.0.60:4646             0.0.0.0:*                   LISTEN      1494/nomad
tcp        0      0 10.42.0.60:4647             0.0.0.0:*                   LISTEN      1494/nomad
tcp        0      0 10.42.0.60:4648             0.0.0.0:*                   LISTEN      1494/nomad
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1254/sshd
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1333/master
tcp        0      0 :::22                       :::*                        LISTEN      1254/sshd
tcp        0      0 ::1:25                      :::*                        LISTEN      1333/master


 [core@bastion ~]$ ansible 10.42.0.60 -m shell -a "docker ps"
10.42.0.60 | success | rc=0 >>
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
a7210ff4470a        redis:latest        "/entrypoint.sh redi   2 minutes ago       Up 2 minutes        6379/tcp            angry_pasteur

I would expect to see something like this in the port field:

0.0.0.0:6379->6379/tcp

nomad output looks like this:

10.42.0.60 | success | rc=0 >>
            Client: true
         Log Level: INFO
            Region: global (DC: dc1)
            Server: true

==> Nomad agent started! Log data will stream in below:

2015/10/01 01:06:58 [INFO] serf: EventMemberJoin: nomad.example.org.global 10.42.0.60
2015/10/01 01:06:58 [INFO] nomad: starting 2 scheduling worker(s) for [batch service _core]
2015/10/01 01:06:58 [INFO] client: using state directory /tmp/nomad/client
2015/10/01 01:06:58 [INFO] client: using alloc directory /tmp/nomad/alloc
2015/10/01 01:06:58 [INFO] raft: Node at 10.42.0.60:4647 [Follower] entering Follower state
2015/10/01 01:06:58 [WARN] serf: Failed to re-join any previously known node
2015/10/01 01:06:58 [INFO] nomad: adding server nomad.example.org.global (Addr: 10.42.0.60:4647) (DC: dc1)
2015/10/01 01:06:59 [WARN] raft: Heartbeat timeout reached, starting election
2015/10/01 01:06:59 [INFO] raft: Node at 10.42.0.60:4647 [Candidate] entering Candidate state
2015/10/01 01:06:59 [INFO] raft: Election won. Tally: 1
2015/10/01 01:06:59 [INFO] raft: Node at 10.42.0.60:4647 [Leader] entering Leader state
2015/10/01 01:06:59 [INFO] nomad: cluster leadership acquired
2015/10/01 01:06:59 [INFO] raft: Disabling EnableSingleNode (bootstrap)
2015/10/01 01:08:11 [INFO] driver.docker: identified image redis:latest as 2f2578ff984f013c9a5d6cbb6fe061ed3f73a17380a4c9b53b76d4b8da3eda7d
2015/10/01 01:08:11 [INFO] driver.docker: created container ec9f5b1855722a8070f7e217df8f48066104bbfc2937096023696d279bb35f48
2015/10/01 01:08:11 [INFO] driver.docker: started container ec9f5b1855722a8070f7e217df8f48066104bbfc2937096023696d279bb35f48
2015/10/01 01:18:43 [INFO] driver.docker: stopped container ec9f5b1855722a8070f7e217df8f48066104bbfc2937096023696d279bb35f48
2015/10/01 01:18:43 [INFO] driver.docker: removed container ec9f5b1855722a8070f7e217df8f48066104bbfc2937096023696d279bb35f48
2015/10/01 01:18:48 [INFO] driver.docker: removed image 2f2578ff984f013c9a5d6cbb6fe061ed3f73a17380a4c9b53b76d4b8da3eda7d
2015/10/01 01:18:48 [INFO] client: completed task 'redis' for alloc '43d42dd5-4335-e5d0-595b-6d805a8c2349'
2015/10/01 13:41:29 [INFO] driver.docker: identified image redis:latest as 2f2578ff984f013c9a5d6cbb6fe061ed3f73a17380a4c9b53b76d4b8da3eda7d
2015/10/01 13:41:29 [INFO] driver.docker: created container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:41:29 [INFO] driver.docker: started container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:48:37 [INFO] driver.docker: stopped container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:48:38 [INFO] driver.docker: removed container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:48:44 [INFO] driver.docker: removed image 2f2578ff984f013c9a5d6cbb6fe061ed3f73a17380a4c9b53b76d4b8da3eda7d
2015/10/01 13:48:44 [ERR] driver.docker: failed stopping container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:48:44 [ERR] client: failed to kill task 'redis' for alloc '1ccec690-70ee-abf3-15d5-f207f66be77a': Failed to stop container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc: No such container: 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:48:44 [ERR] driver.docker: failed stopping container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:48:44 [ERR] client: failed to kill task 'redis' for alloc '1ccec690-70ee-abf3-15d5-f207f66be77a': Failed to stop container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc: No such container: 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:48:44 [ERR] driver.docker: failed stopping container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:48:44 [ERR] client: failed to kill task 'redis' for alloc '1ccec690-70ee-abf3-15d5-f207f66be77a': Failed to stop container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc: No such container: 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:48:44 [ERR] driver.docker: failed stopping container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:48:44 [ERR] client: failed to kill task 'redis' for alloc '1ccec690-70ee-abf3-15d5-f207f66be77a': Failed to stop container 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc: No such container: 3f4a2748282ee7e85e93da0955f23ddb2ee86e1006e0c7a7727333c9c1fbafcc
2015/10/01 13:48:44 [INFO] client: completed task 'redis' for alloc '1ccec690-70ee-abf3-15d5-f207f66be77a'
2015/10/01 13:50:11 [INFO] driver.docker: identified image redis:latest as 2f2578ff984f013c9a5d6cbb6fe061ed3f73a17380a4c9b53b76d4b8da3eda7d
2015/10/01 13:50:11 [INFO] driver.docker: created container a7210ff4470a42788fc909bce81844844d8c98b47422c031462d8eaea7ea4315
2015/10/01 13:50:11 [INFO] driver.docker: started container a7210ff4470a42788fc909bce81844844d8c98b47422c031462d8eaea7ea4315
2015/10/01 13:53:31 [INFO] driver.docker: stopped container a7210ff4470a42788fc909bce81844844d8c98b47422c031462d8eaea7ea4315
2015/10/01 13:53:31 [INFO] driver.docker: removed container a7210ff4470a42788fc909bce81844844d8c98b47422c031462d8eaea7ea4315
2015/10/01 13:53:38 [INFO] driver.docker: removed image 2f2578ff984f013c9a5d6cbb6fe061ed3f73a17380a4c9b53b76d4b8da3eda7d
2015/10/01 13:53:38 [INFO] client: completed task 'redis' for alloc '39daa1b1-f964-d61e-eeea-23c2f3d95ae0'
2015/10/01 13:53:38 [ERR] client: dropping update to alloc '43d42dd5-4335-e5d0-595b-6d805a8c2349'

[markpollack]

Thanks @purehyperbole !

[dadgar]

I think there might be some confusion on what dynamic ports means. When the job asks for a dynamic ports, the scheduler will pick any available port on the host. So asking for dynamic port REDIS will pick a random port. That is why you can't ping redis on its default host. Nomad sets an environment variable for each port so you can determine which port you were assigned as you can see in docker inspect.

Please see: https://www.nomadproject.io/docs/jobspec/index.html#dynamic_ports

I am going to close this issue as it is working as intended.

[mmlb]

@dadgar the problem I originally encountered was with the docker driver. Reading the Manual Port Mapping section of the docs seems to indicate that the driver will pickup on the env var, then map the randomly selected port 1:1 to the container, then pass the env var down to the container so that it can bind to said port and/or advertise the correct port to others.

I might have mis-read that (if so the docs could probably be a little more explicit IMO), but I think this is a scenario that makes a lot of sense and should be supported.

[mmlb]

I'd like to expand on my possible mis-reading of the docker driver docs.

The docs states

In this case, Nomad doesn't know which container port to map to, so it maps 1:1 with the host port.
For example, 1.2.3.4:22333 will map to 22333 inside the container.

dynamic_ports = ["http"]

Your process will need to read the NOMAD_PORT_HTTP environment variable to determine which port to bind to.

Where did 1.2.3.4:223333 come from? Is it the tuple randomly selected by higher layer? How do I map it to lower? I read further down and see http. Will that correspond to the 22333 from above?

Having the code block almost looks like the start of a new subsection since it does not seem to be introduced by the previous paragraph.

If the 223333 port does in fact map to the http config then dynamic_ports section should come before the 223333 section in the docs. And there should be wording like

In this case higher layers randomly picked port 223333 to satisfy the dynamic_ports=["http"] directive, the container will have it's port 223333 published as 223333 on the host. The selected port can be discovered by reading the value of the NOMAD_PORT_HTTP environment variable (and this is where the docker driver is not behaving as documented).

[dadgar]

@mmlb sorry I totally misread this. I am opening it back up.

[vtuson]

I still have this issue. Whatever combination of IP/Port I select the binding does not take effect.

"HostConfig": {
"PortBindings": {
"80/tcp": [
{
"HostPort": "8001"
}
]
},
"RestartPolicy": {},
"LogConfig": {}
}

[diptanu]

@vtuson Can you post your entire job config and which version of Nomad you're using?

[vtuson]

Sorry, I posted in the wrong thread. My problem is while using directly go-dockerclient lib

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.