Docker-Compose + Swarm : Problem when trying to deploy twice the same stack on two different Node using different constraint.

[djsly]

Hello, I am trying to use Swarm + Docker-Compose to deploy a stack of services meant to be running and talking to each other on the same physical host.

I have a A/B type of environment, where I would like to deploy version X on Node1 while deploying version Y of the stack on Node2.

What I have discovered so far is that I can safely deploy the stack on either of the Nodes (1|2) as long as the other Node isn't running the same stack.

Before I start into greater details here are some details about the tools I am using.

Docker Compose Version

[root@<hostname> big-knowledge]# docker-compose -v
docker-compose version: 1.3.0rc3
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013

Docker Swarm Version

[root@<hostname> big-knowledge]# docker ps
CONTAINER ID        IMAGE                                            COMMAND                CREATED             STATUS              PORTS                                         NAMES
5b0404b7b35b        swarm:0.3.0-rc3                                  "/swarm manage token   11 minutes ago      Up 11 minutes       0.0.0.0:2375->2375/tcp                        romantic_pasteur       
56fec2504f08        swarm:0.3.0-rc3                                  "/swarm join --addr=   12 minutes ago      Up 12 minutes       2375/tcp                                      grave_yalow            
373a8353af24        swarm:0.3.0-rc3                                  "/swarm join --addr=   12 minutes ago      Up 12 minutes       2375/tcp                                      suspicious_curie  

Here are the details of my two nodes with their respective labels.
Node1: pod=B & service=srbk
Node2: pod=A & service=srbk

[root@<hostname> big-knowledge]# DOCKER_HOST=tcp://localhost:2375 bash -c 'docker info'
Containers: 0
Images: 0
Storage Driver: 
Role: primary
Strategy: spread
Filters: affinity, health, constraint, port, dependency
Nodes: 2
 Node1: <ip>:2375
  └ Containers: 0
  └ Reserved CPUs: 0 / 42
  └ Reserved Memory: 0 B / 131.9 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.8.13-68.1.2.el6uek.x86_64, operatingsystem=CentOS Linux 7 (Core), pod=B, service=srbk, storagedriver=btrfs
 Node2: <ip>:2375
  └ Containers: 0
  └ Reserved CPUs: 0 / 42
  └ Reserved Memory: 0 B / 131.9 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.8.13-68.1.2.el6uek.x86_64, operatingsystem=CentOS Linux 7 (Core), pod=A, service=srbk, storagedriver=btrfs
Execution Driver: 
Kernel Version: 
Operating System: 
CPUs: 84
Total Memory: 263.8 GiB
Name: 
ID: 
Http Proxy: 
Https Proxy: 
No Proxy:

Starting fresh

[root@<hostname> big-knowledge]# DOCKER_HOST=tcp://localhost:2375 bash -c 'docker images'
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE

[root@<hostname> big-knowledge]# DOCKER_HOST=tcp://localhost:2375 bash -c 'docker ps -a'
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

I ran the following docker-compose file

cat srbk-poda.yml 
virtuoso:
        image: <private-registry>/virtuoso
        ports: 
                - "1111:1111"
                - "8890:8890"
        volumes:
                - /localdata/big-knowledge/data:/localdata/big-knowledge/data
        env_file:
                - ./poda.env
bkr:
        image: <private-registry>/virtuoso
        command: /bin/bash -c "source /etc/sysconfig/crond && exec /usr/sbin/crond -n -s $CRONDARGS"
        links:
                - virtuoso
                - canonicalizer
        volumes:
                - /localdata/big-knowledge/data:/localdata/big-knowledge/data
                - /localdata/bigkr/canon/solr_home/bigkr/data:/localdata/bigkr/canon/solr_home/bigkr/data
        env_file:
                - ./poda.env
canonicalizer:
        image: <private-registry>/canonicalizer
        links:
                - virtuoso
        ports:
                - "8983:8983"
        volumes:
                - /localdata/big-knowledge/data:/localdata/big-knowledge/data
                - /localdata/bigkr/canon/solr_home/bigkr/data:/localdata/bigkr/canon/solr_home/bigkr/data
        env_file:
                - ./poda.env

where the env_file looks like this

cat poda.env 
constraint:pod==A
constraint:service==srbk

Now when running docker-compose I end up with the following (so far so good, everything gets downloaded and started started!)

[root@<hostname> big-knowledge]# DOCKER_HOST=tcp://localhost:2375 bash -c 'docker-compose -f srbk-poda.yml up -d --allow-insecure-ssl'
Pulling virtuoso (<private-registry>/virtuoso:latest)...
Node1: Pulling <private-registry>/virtuoso:latest... : downloaded
Node2: Pulling <private-registry>/virtuoso:latest... : downloaded
Creating bigknowledge_virtuoso_1...
Pulling canonicalizer (<private-registry>/canonicalizer:latest)...
Node1: Pulling <private-registry>/canonicalizer:latest... : downloaded
Node2: Pulling <private-registry>/canonicalizer:latest... : downloaded
Creating bigknowledge_canonicalizer_1...
Creating bigknowledge_bkr_1...

Question #1:
Why is the images downloaded on all nodes even though I have a valid constraint that applies to Node2 only ?

Question #2:
Why is the 2nd container ends up with 4 different container names while
the 3rd container ends up with 7 different container names.

[root@<hostname> big-knowledge]# DOCKER_HOST=tcp://localhost:2375 bash -c 'docker ps'
CONTAINER ID        IMAGE                                    COMMAND                CREATED             STATUS              PORTS                                                          NAMES
e81dc5a169bd        <private-registry>/virtuoso:latest        "/bin/bash -c 'sourc   3 minutes ago       Up 3 minutes        1111/tcp, 8890/tcp                                         Node2/bigknowledge_bkr_1                                                                                                                                                                                                                                                                                                                                                                                                                        
11013f63c97b        <private-registry>/canonicalizer:latest   "/opt/apache-tomcat-   3 minutes ago       Up 3 minutes       <Node2 ip>:8983->8983/tcp                                   Node2/bigknowledge_bkr_1/bigknowledge_canonicalizer_1,Node2/bigknowledge_bkr_1/canonicalizer,Node2/bigknowledge_bkr_1/canonicalizer_1,Node2/bigknowledge_canonicalizer_1                                                                                                                                                                                                      
0d8dbab50e33        <private-registry>/virtuoso:latest        "/bin/sh -c 'supervi   5 minutes ago       Up 5 minutes       <Node2 ip>:1111->1111/tcp,<node1 ip>:8890->8890/tcp         Node2/bigknowledge_bkr_1/bigknowledge_virtuoso_1,Node2/bigknowledge_bkr_1/virtuoso,Node2/bigknowledge_bkr_1/virtuoso_1,Node2/bigknowledge_canonicalizer_1/bigknowledge_virtuoso_1,Node2/bigknowledge_canonicalizer_1/virtuoso,Node2/bigknowledge_canonicalizer_1/virtuoso_1,Node2/bigknowledge_virtuoso_1  

Now if I look at the PS for the other docker-compose file (that one that should deploy on Node1), I get the list of running container on Node2

[root@<hostname> big-knowledge]# DOCKER_HOST=tcp://localhost:2375 bash -c 'docker-compose -f srbk-podb.yml ps'
            Name                          Command               State                              Ports                             
------------------------------------------------------------------------------------------------------------------------------------
bigknowledge_bkr_1             /bin/bash -c source /etc/s ...   Up      1111/tcp, 8890/tcp                                           
bigknowledge_canonicalizer_1   /opt/apache-tomcat-7.0.59- ...   Up      <Node2 ip>:8983->8983/tcp                                
bigknowledge_virtuoso_1        /bin/sh -c supervisord -c  ...   Up      <Node2 ip>:1111->1111/tcp, <Node2 ip>:8890->8890/tcp

Trying to run the POD B docker-compose file I get the following error.

[root@<hostname> big-knowledge]# DOCKER_HOST=tcp://localhost:2375 bash -c 'docker-compose --verbose -f srbk-podb.yml up -d'
Recreating bigknowledge_virtuoso_1...
unable to find a node that satisfies pod==B

Full output

[root@<hostname> big-knowledge]# DOCKER_HOST=tcp://localhost:2375 bash -c 'docker-compose --verbose -f srbk-podb.yml up -d'
Compose version 1.3.0rc3
Docker base_url: http://localhost:2375
Docker version: KernelVersion=, Arch=amd64, ApiVersion=1.16, Version=swarm/0.3.0, GitCommit=28cd51b, Os=linux, GoVersion=go1.3.3
docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=bigknowledge', u'com.docker.compose.service=virtuoso', u'com.docker.compose.oneoff=False']})
docker containers -> (list with 1 items)
docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=bigknowledge', u'com.docker.compose.service=canonicalizer', u'com.docker.compose.oneoff=False']})
docker containers -> (list with 1 items)
docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=bigknowledge', u'com.docker.compose.service=bkr', u'com.docker.compose.oneoff=False']})
docker containers -> (list with 1 items)
Recreating bigknowledge_virtuoso_1...
docker stop <- (u'0d8dbab50e33558f406597b6a9a1b00064ed5814bd05688e3b7647358819e7ef')
docker stop -> None
docker rename <- (u'0d8dbab50e33558f406597b6a9a1b00064ed5814bd05688e3b7647358819e7ef', u'0d8dbab50e_bigknowledge_virtuoso_1')
docker rename -> None
docker inspect_container <- (u'0d8dbab50e33558f406597b6a9a1b00064ed5814bd05688e3b7647358819e7ef')
docker inspect_container -> {u'AppArmorProfile': u'',
 u'Args': [u'-c', u'supervisord -c /etc/supervisor/supervisord.conf'],
 u'Config': {u'AttachStderr': False,
             u'AttachStdin': False,
             u'AttachStdout': False,
             u'Cmd': [u'/bin/sh',
                      u'-c',
                      u'supervisord -c /etc/supervisor/supervisord.conf'],
             u'CpuShares': 0,
             u'Cpuset': u'',
...
docker inspect_image <- ('<private-registry>/virtuoso')
docker inspect_image -> {u'Architecture': u'amd64',
 u'Author': u'Sylvain_Boily',
 u'Comment': u'',
 u'Config': {u'AttachStderr': False,
             u'AttachStdin': False,
             u'AttachStdout': False,
             u'Cmd': [u'/bin/sh',
                      u'-c',
                      u'supervisord -c /etc/supervisor/supervisord.conf'],
             u'CpuShares': 0,
...
docker inspect_image <- ('<private-registry>/virtuoso')
docker inspect_image -> {u'Architecture': u'amd64',
 u'Author': u'Sylvain_Boily',
 u'Comment': u'',
 u'Config': {u'AttachStderr': False,
             u'AttachStdin': False,
             u'AttachStdout': False,
             u'Cmd': [u'/bin/sh',
                      u'-c',
                      u'supervisord -c /etc/supervisor/supervisord.conf'],
             u'CpuShares': 0,
...
docker inspect_image <- (u'7ba417c20ccae287c4d8ee84f824f6226ff1ceb731be20bf6f2e12361b4d503a')
docker inspect_image -> {u'Architecture': u'amd64',
 u'Author': u'Sylvain_Boily',
 u'Comment': u'',
 u'Config': {u'AttachStderr': False,
             u'AttachStdin': False,
             u'AttachStdout': False,
             u'Cmd': [u'/bin/sh',
                      u'-c',
                      u'supervisord -c /etc/supervisor/supervisord.conf'],
             u'CpuShares': 0,
...
docker create_container <- (name=u'bigknowledge_virtuoso_1', image='<private-registry>/virtuoso', labels={u'com.docker.compose.service': u'virtuoso', 'com.docker.compose.config-hash': 'e1d9643836c73ce9edc909e73a190bbf71e33cb95188acf9361e577ad65ff8e1', u'com.docker.compose.project': u'bigknowledge', 'com.docker.compose.version': u'1.3.0rc3', u'com.docker.compose.oneoff': u'False', 'com.docker.compose.container-number': '1'}, host_config={'NetworkMode': u'bridge', 'Links': [], 'PortBindings': {'8890/tcp': [{'HostPort': '8890', 'HostIp': ''}], '1111/tcp': [{'HostPort': '1111', 'HostIp': ''}]}, 'Binds': [u'/var/lib/docker/vfs/dir/d10a81bd2a02afaae4cdea7d2f986b1a68506d54c3d8fedc4b06f8de716899ec:/localdata/bigkr/canon/solr_home/bigkr/data:rw', u'/var/lib/docker/localdata/big-knowledge/data:/localdata/big-knowledge/data:rw'], 'ExtraHosts': [], 'LogConfig': {'Type': u'json-file', 'Config': {}}, 'VolumesFrom': []}, environment={'constraint:service': '=srbk', 'constraint:pod': '=B', u'affinity:container': u'=0d8dbab50e33558f406597b6a9a1b00064ed5814bd05688e3b7647358819e7ef'}, volumes={u'/localdata/big-knowledge/data': {}}, detach=True, ports=[u'1111', u'8890'])
unable to find a node that satisfies pod==B

Also everytime I rerun the command the name of the container that it is trying to recreated gets prepended an ID

[root@<hostname> big-knowledge]# DOCKER_HOST=tcp://localhost:2375 bash -c 'docker-compose -f srbk-podb.yml up -d'
Recreating 0d8dbab50e_bigknowledge_virtuoso_1...
unable to find a node that satisfies pod==B

double ID (etc)

[root@<hostname> big-knowledge]# DOCKER_HOST=tcp://localhost:2375 bash -c 'docker-compose -f srbk-podb.yml up -d'
Recreating 0d8dbab50e_0d8dbab50e_bigknowledge_virtuoso_1...
unable to find a node that satisfies pod==B

From what I can see, docker-compose is trying to keep the existing previously lined container as part of the affinity hence forcing the newly recreated container to be on the original Node2 (BUT since the new docker-compose file is specifying a different env_file, we get conflicting label constraint.)

Let me know if there is something (hacky or not) that I could do to prevent this from occurring ! :)

PS: Sorry for the double post: docker-archive/classicswarm#970

[dnephin]

The prepended id happend when we try to recreate a conatiner. We rename it with this prepended id, then copy the volumes over, then remove it. Sometimes the remove fails, which leaves this weirdly named container around.

[timgriffiths]

I having the same problem

# docker-compose up -d test
Recreating compose_test_1
ERROR: unable to find a node that satisfies container==28aaef3e61413657d3849627c35647602ee995cae7823a1cebf2b5ae3310332f

Any work around?

[dweomer]

I appear to be experiencing but without the complication of an environment file.

[navinSing]

@dnephin
same problem.. any fix for this

[dnephin]

I'm going to close this issue as a duplicate of #2866 since the problem desciption in that issue reflects the problem more directly. There is a discussion in the issue about the problem.