Overriding the set of values when extending the "ports" option.

[ghost]

The default behavior when extending a service or overriding a whole docker-compose.yml file is to concatenate the sets of values on the multi-value options: ports, expose, external_links, dns and dns_search.

The problem appears when for example you have two production hosts both serving on the port 80 and a single machine for development, there is no way of extending or overriding the services for a development environment that avoids the port collision on the host.

An example: lets assume there are two servers one is a REST API and the other is a webpage that uses the API, both listen on port 80 on the production host

• compose.yml on host A:

restserver:
  image: node
  ports:
    - "80:80"

• compose.yml on host B:

webserver:
  image: django
  ports:
    - "80:80"

On a development environment where you want to test this setup you would define:

• compose.override.yml on dev for A:

restserver:
  ports:
    - "127.0.0.1:3030:80"

• compose.override.yml on dev for B:

webserver:
  external_links:
    - "service_restserver_1:api.restserver.domain.com"
  ports:
    - "127.0.0.1:8080:80"

Of course the "ports" option is concatenated with the production and there is no way of running both containers on the same host because both try to bind to port 80.

Is there any workaround or yml specific syntax for this use case?

[dnephin]

I would do something like this:

docker-compose.yml (no ports)

restserver:
  image: node

webserver:
  image: django

For dev you can setup some host ports

docker-compose.override.yml

restserver:
  ports: ['3030:80']

webserver:
  ports: ['8080:80']

and for prod you can set them to a different port:

docker-compose.prod.yml

restserver:
  ports: ['80:80']

webserver:
  ports: ['80:80']

In prod you would run docker-compose -f docker-compose.yml -f docker-compose.prod.yml up

[ghost]

It seems like a good, clean approach. Only cons would be the verbose command on production but that one is automatic.

Thanks @dnephin

[gregmartyn]

Another downside is that it no longer "just works". The user has to either create an override file or specify ports manually.

It would be nice if the override file could actually override array values like it does for scalar values. Instead, the override file appends array values and only actually overrides scalar values.

[mkurzeja]

I agree with @gregmartyn . A feature that would allow to override ports array instead of merging them would be awesome.

In my case we use it to override a port when one of the developers is not able to work on the default one.

Maybe docker-compose could just override the ports that are already forwarded: If port 80 is specified in both files, it uses just the one from docker-compose.override.yml?

[udhayakumaran]

@gregmartyn @mkurzeja Docker is already having that feature. I tried that to override with simple docker-compose, it is working.

For example,

docker-compose.yml with content,

my-httpd:
  image: httpd:latest
  ports:
    - "1110:80"  

And docker-compose.override.yml with content,

my-httpd:
  image: httpd:2.4

This is the docker ps info,

12286c55b76b httpd:2.4 "httpd-foreground" 5 seconds ago Up 4 seconds 0.0.0.0:1110->80/tcp dockercomposetest_my-httpd_1

It is taking override's image and port from the compose file.

[gregmartyn]

@udhayakumaran see above. "the override file appends array values and only actually overrides scalar values."

The image in your example is a scalar value and did get overridden. The port you specified is part of an array and was not overridden. You didn't specify any ports in your override file, so they were not overridden. Had you specified ports in the override file, they would have been combined with the port specified in the main config file.

[udhayakumaran]

@gregmartyn Okay, I understand now. Thanks for correcting me..

[datacarl]

@gregmartyn If you are using the .env file introduced in 1.7 you can define the compose files there. You'll have to ensure you have the correct .env file (a prod version for prod, dev version for dev) ofc but that's the only manual change required.

For dev environments you can set
COMPOSE_FILE=docker-compose.yml:docker-compose.dev.yml
and for production
COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml

Together with what @dnephin suggested it works fine.

[kevin-cantwell]

@datacarl Do you have a link to documentation on the DOCKER_FILE env var? It's not clear to me how it's used by Docker Compose.

[datacarl]

@kevin-cantwell https://docs.docker.com/compose/env-file/

(should be COMPOSE_FILE, my bad. Edited my reply above)

[kevin-cantwell]

@datacarl Ah, COMPOSE_FILE. Found it now, thanks: https://docs.docker.com/compose/reference/envvars/#compose-file

[CarstenHoyer]

I have added a preliminary pull request, that introduces a way to add an overwrite strategy other than the default append/union. The overwrite config currently only supports "ports" but it would be a smallish thing to support the other array options too.

Your compose file would then look like either:

version: "2"
services:
  web:
    override_strategy: overwrite

or

version: "2"
services:
  web:
    overwrite:
      - ports

And the result could be that if docker-compose.yml has ports: 80, 443, and your override file has 8080 and 8443 fx, then only the 8080 and 8443 ports would be in the final config.

[dimaqq]

@CarstenHoyer did your overwrite: change make it to any release?

[CarstenHoyer]

@dimaqq - I am afraid not. I have not kept the PR up to date with the upstream, and it also doesnt contain any automated tests.

Since I never got any official feedback, I figured it was not in scope.

[keithy]

Solution available see http:gitHub.com/keithy/sprankle-podOn 3 May 2019 11:27 am, port22 <[email protected]> wrote:This problem is even bigger, when you have to use multiple files. YAML merging/anchors are not sufficient. I need services for developers, and the same service with different definitions for developers of other departments, quality assurance, testing ... I need to override volumes. There should be a possibility to merge definitions. Or can you put an example how to use an anchor of another docker-compose.yml file? Even if that would be possible, it would make the additional file senseless and specific persons would have to modify the docker-compose.yml or an .override. file themselves, which has to be avoided. I want to serve a main yml and different override yamls for different persons —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[nyetwurk]

Does anybody else see why I am suggesting a dictionary rather than an array?

[mike-code]

I do but the compose project looks semi abandoned. I wouldn't recommend it using in production.

[im-n1]

@mike-code thats nonsense this project is quite active.

[vectorjohn]

I don't think the dictionary suggestion is a good one, docker-compose shouldn't have some new half baked scheme for merging files when YAML already has a standard that does all that is needed (other than @keithy 's use case, which sounds so complex it's out of scope).

[nyetwurk]

Portmapping is fundamentally a dictionary, not an array, and merging comes for free.

[im-n1]

@vectorjohn is there any official yaml lib that can do the merging?

[keithy]

My use case and solution is very simple actually:

docker-compose allows you to supply .env as an environment file. The environment variable COMPOSE_FILE provides a list of compose_files to be assembled.

In those individual files I place environment variables where there are ports mentioned, so that the .env file can supply those values also.

The missing piece of the puzzle, as with so much of the docker stuff, is to make the .env file readable. I do this by creating the .env file from a readable version using an all-to-simple pre-processor docker-compose-use <my.env>

[vectorjohn]

@vectorjohn is there any official yaml lib that can do the merging?

Oddly enough, docker-compose already supports this it looks like:
https://medium.com/@kinghuang/docker-compose-anchors-aliases-extensions-a1e4105d70bd

[typoworx-de]

How does the override feature work for version: '3.6' in yaml head? Trying to override some settings in DDEV-Project (https://github.com/drud/ddev) also using docker-compose (version 1.24.1 which seems to be latest release already).

Adding override_strategy: overwrite won't work and breaks docker-compose:

Unsupported config option for services.web: 'override_strategy'

[mcabrams]

@typoworx-de That was a proposed solution with a corresponding pull request that did not get merged. It is not available in version 3.6, or any other version for that matter.

[noahtallen]

One reason this is very important (didn't see it mentioned yet) is that we don't always have access to change the base docker-compose.yml file. In fact, I'm using a dockerfile provided by a different service that I cannot control. I probably ought not to update that file since it is in the source control of that other service.

If I was able to create env files and remove ports from the base config, then sure, I'd accept that there's a workaround. But you aren't always in control of the base config file, and modifying it directly generally isn't the best idea.

[nyetwurk]

I'm also unclear how you could use yaml array merge to remove an entry from the list, whereas this is trivial if it is a dict

[ebuildy]

What about adding a excludes option in extends clause?

service_A:
   extends:
     file: docker-compose.yaml
     service: service_A
     excludes: ["image", "ports"]
   build: .

[kkom]

@ebuildy - looks quite promising to me, but note that extends is not available in Compose file versions 2.1 and up: moby/moby#31101

[azimuthdeveloper]

This should be reopened as the existing functionality is inconsistent with scalar values being overwritten by default. Or people should be able to make a choice - it makes no sense for them to be appended by default.

[adrianhelvik]

It makes no sense, but changing it would break a lot of configs. I've proposed another non-breaking solution. #8276

[gostega]

I'd really like to have the override functionally suggested by CarstenHoyer on 13 Sep 2016. I have a situation where by default I want the application exposed on a port for easy testing. But also for testing, and in prod, staging etc, I want to deploy the app behind nginx, and therefore not expose the ports. So the default docker-compose.yml file would have the ports specified, then the docker-compose-withnginx.yml which just defines the nginx service, should override the ports.

[wivaku]

@adrianhelvik 's proposal #8276 is clean and does not require modifying the config / original docker-compose file.

[luke-hill]

Why don't people copy what rubocop do. They have this exact same problem.

For those who don't know rubocop is a ruby gem that provides stylistic configuration through yml files. Often people may have 2/3/4 on their repo at any time.

There is a top level config option that specifies how to combine (In rubocop's example you often want to merge them, but you can have both behaviours).

https://docs.rubocop.org/rubocop/configuration.html#merging-arrays-using-inherit_mode

[oizik]

Perhaps this is an old discussion but I needed to do this quickly today, my solution was pretty simple:

In docker-compose.yml define the ports as follows:

web:
  ...
  ports:
      - ${APP_PORTS:-8000:8000}		
      - ${DEBUG_PORTS:-5678:5678}	

Default ports used are 8000:8000 for the app and 5678:5678 for debug attachment.

Now I needed to start another web container to run a command within that container, so simply prepended APP_PORTS=8001:8000 and DEBUG_PORTS=5679:5678 to the docker compose run. This overrides the host ports, hence no port conflicts.

The full command as follows:

(export APP_PORTS=8001:8000; export DEBUG_PORTS=5679:5678;  docker compose run --service-ports --name debug_django_command_container  django python -m debugpy --listen 0.0.0.0:5678 -m manage my_django_command)

Notice the use of brackets around the whole command. This is so the exports do not remain in the current running shell session when the command finishes.

This works on my Linux machine, should be the same on Mac, not 100% sure on Windows, but should have similar options to pass environment variables to a command, maybe: this or this.

Reference: Use an Environment File

Hope this helps someone.

[zba]

The problem is with vendor's source, nowdays ot os essy to distribute your systems using docker and not all vendors makes variables or overridable ports, so for example if I have

ports: 
  - 8080:8080

in vendor's compose file, I can't change it anyway without changing the source, but it means that on update by git pull I will have to deal with this and moving things in and out or had to remeber that in the project i have to use some startup script and need to watch my composer up2date or so on. would be nice to have a way to reset lists in overrides