Proposal: Types, imports and external services

[aanand]

This proposal describes an enhancement to docker-compose.yml that will enable the importing of selected bits of configuration from other Compose configuration files, and the definition of ‘external’ or ‘dummy’ services. It is intended to serve three use cases:

1. I deploy my app in multiple environments - development, staging, production. Some configuration is different in each environment; some services only need to exist in some environments. I want to maintain a configuration file for each environment without repeating myself.
2. I have multiple apps which perform different tasks but re-use some services. I don’t want to repeat the definitions of those services in each app’s configuration.
3. I have a defined dependency, such as a database, in my app. When I’m developing the app, I want Compose to spin up a local database container to link to from my other services. When running the app in production, I instead want to point my other services at an already-running, separately-managed database service, which may not be managed with Compose or even Docker.

It is related to #495/#845, in that use case 1 is partially served by parameterisation of configuration values. It does not replace that functionality, but complements it.

It is distinct from #318/#758, in that it does not serve this use case:

• I have two apps. App A has link dependencies on app B. When I type docker-compose up in app A’s directory, I want Compose to first spin up app B and then app A, with cross-app links in place.

However, that use case could be served with an implementation that builds on the enhancements proposed here.

Service types

A new configuration key, type, can be specified on a service defined in docker-compose.yml. It’s optional, and its value defaults to "container". There are three possible types of value.

"container"

Denotes that this service consists of one or more homogenous Docker containers, configured using the options specified here. This is exactly how docker-compose.yml services are defined today.

web:
  type: container # optional
  build: .
  ports:
    - 80:8000
  links:
    - db

Path to another Compose file

Denotes that this service is defined in another file. The file’s path, and the name of the service within that file, are supplied here.

db:
  type: common.yml#db

Assuming common.yml defines a db service, this is equivalent to copying and pasting its configuration here.

Configuration can also be overridden:

db:
  type: common.yml#db
  environment:
    POSTGRES_USER: devpass
    POSTGRES_PASSWORD: devuser

"external"

Denotes an externally-defined service. Its location, and any other configuration, are supplied here.

db:
  type: external
  host: 1.2.3.4
  port: 5432
  environment:
    POSTGRES_USER: produser
    POSTGRES_PASSWORD: prodpass

This results in services which link to this service being furnished with hostnames and environment variables in exactly the same way as if they were linked to a Docker container:

$ docker-compose run web cat /etc/hosts
127.0.0.1  localhost
1.2.3.4    db

$ docker-compose run web env
DB_PORT=tcp://1.2.3.4:5432
DB_PORT_5432_TCP=tcp://1.2.3.4:5432
DB_PORT_5432_TCP_ADDR=1.2.3.4
DB_PORT_5432_TCP_PORT=5432
DB_PORT_5432_TCP_PROTO=tcp
DB_ENV_POSTGRES_USER=produser
DB_ENV_POSTGRES_PASSWORD=prodpass

In this way, the external keyword defines a “dummy” service.

Usage example

Pulling it all together, here’s an example 3-file setup:

common.yml

web:
  build: .
  ports:
    - 80:8000
  links:
    - db
db:
  image: postgres

development.yml

web:
  type: common.yml#web
  environment:
    - APP_ENV=development
db:
  type: common.yml#db

production.yml

web:
  type: common.yml#web
  environment:
    - APP_ENV=production
db:
  type: external
  host: 1.2.3.4
  port: 5432
  environment:
    POSTGRES_USER: produser
    POSTGRES_PASSWORD: prodpass

Discussion: including transitive dependencies

It would be useful to be able to pull in an externally-defined service and its own dependencies - for example, if db got its volumes from a dbdata container, it would be valuable from an encapsulation and DRY perspective for that to implicitly come along with it.

However, it’s also important that transitive dependencies can be overwritten, such as in the example case above where web’s dependency on db is swapped out in production.

The exact semantics of imports, and how to serve both use cases, need to be carefully worked out.

[dnephin]

I like the direction this is going. As you mention it doesn't fully cover my use case, but it does make progress in that direction. I really think the inclusion of transitive dependencies is the critical bit that makes a feature like this really valuable. This proposal also drops including from remote sources. In the simple case (where only a single level of files is being included) I agree that piece could be external to fig, and would remove a lot of the complexity from my earlier PR.

Path to another Compose file

It seems a little strange to me that the type is a path to a file. Something like this seems more intuitive:

db:
  type: include
  path: common.yml#db

including transitive dependencies

I've been experimenting with a setup that uses #758 and one of the rough parts was the project name. All the services end up sharing the same project name (because otherwise fig can't really keep track of it), so a service with the same name in two different fig.yml would cause a conflict. I was forced to create really verbose names for all the services.

I think docker labels might be the missing bit that makes this all work. With labels the container could be named based on the file it came from, and compose could identify that the container/image was being included to this project with a label.

The syntax might look something like this:

servicea:
  type: include
  path: servicea-compose.yml#webapp
  include_dependencies: True

Of course without support for including remote urls, it means you still have to know ahead of time all of the files to fetch to make the dependency graph complete. When the graph is depth > 2 that becomes difficult.

[girvo]

A feature that I'd really love to see in the next-gen docker-compose.yml is the ability to pass a Dockerfile in, taking advantage of Docker 1.5's new feature that allows you to specify a Dockerfile when you run commands.

The reason this will be useful, is to have a Dockerfile.dev, Dockerfile.stage, Dockerfile.prod, etc. That feature in the docker-compose.yml will allow for docker-compose to really be taken to another level, especially with the coming support for Swarm. You'd be able to completely manage the entire project through Docker, Docker Compose and Docker Swarm -- nothing else needed!

[noisy]

@girvo syntax could be as simple as adding optional filename to build: option

build: . would be equivalent of build: ./Dockerfile or build: Dockerfile, but user could also use:
build: Dockerfile.dev or build: dockerfiles/prod

[aanand]

@girvo @noisy This is being discussed in #834.

[aanand]

It seems a little strange to me that the type is a path to a file.

Agreed. Let’s do as you suggest and have an include type, and specify the path separately. We could even separate path and service, though I do like the concision of the hash syntax.

I think docker labels might be the missing bit that makes this all work. With labels the container could be named based on the file it came from, and compose could identify that the container/image was being included to this project with a label.

Yep. There’s a fun problem here where you could include the same service multiple times under different names, and we should allow that. So it seems to me that the fully-qualified name of an included service should have the following components (whatever the actual syntax):

• filename + service name of the include service
• service name of the included service, or the transitive dependency

So if I import a db service from another file under the name mydb, the fully-qualified name components are:

• /path/to/myapp/docker-compose.yml
• mydb
• db

If the included db service depends on a dbdata container, its fully-qualified name components are:

• /path/to/myapp/docker-compose.yml
• mydb
• dbdata

[aanand]

Here’s how I see transitive dependencies working. First, here’s our external config file:

common.yml

web:
  build: .
  links: ["db"]
db:
  image: postgres
  volumes_from: ["dbdata"]
dbdata:
  image: busybox
  command: /bin/true
  volumes: ["/var/lib/postgresql"]

Now, what we want to do from our docker-compose.yml is:

• Include the db service as mydb and have dbdata brought along as a volumes_from dependency without having to specify it.
• Include the web as myweb, but rather than have it bring along its own db (and, transitively, dbdata), point it at mydb.

Here’s how I see that syntax working:

docker-compose.yml

myweb:
  type: include
  path: common.yml
  service: web
  links: ["mydb:db"]
mydb:
  type: include
  path: common.yml
  service: db

The include type sees the links option and knows to:

• ignore the link to db in common.yml
• link our locally-defined mydb service in under the name db

If I docker-compose ps, I see something like:

Name          Command          State  Ports
------------------------------------------------
myweb:web     python app.py    Up     5000->5000
mydb:db       postgres         Up     5432
mydb:dbdata   /bin/true        Up

Does that make sense?

[dnephin]

Yup, I think that all looks good and makes sense.

There’s a fun problem here where you could include the same service multiple times under different names, and we should allow that.

Along these lines, there is a question about instance re-use and sharing. In the current implementation of #758 if you have a dependency graph where service A and service B both link to a service C, they both get a link to the same container. There is only every 1 instance of service C. I think this is correct.

With the above example (where you can override parts of a service definition) I think that gets a bit more complicated. By including web and giving it a new name, it is now a new instance of that service. In this case we're changing the definition of the service (its links), so that is correct.

It would be great if there was some way to say "link to an included service" where that service can be linked to from other included files without it creating a new instance of that service. Something like this:

common.yml

web:
  build: .
  links: ["db"]
db:
  image: postgres
  volumes_from: ["dbdata"]
dbdata:
  image: busybox
  command: /bin/true
  volumes: ["/var/lib/postgresql"]

docker-compose.yml

frontend:
  build: .
  links: ["db", "web"]

db:
  type: include
  path: common.yml
  service: db

web:
  type: include
  path: common.yml
  service: web

I would expect that only 1 db container would be launched, and both web and frontend would link to that same instance.

I think this conflicts with the previous example, but I think it's an important usecase. Maybe there needs to be two different types for this? include vs share ?

include would mean, re-use the service definition, but treat it as a separate instance. where as share (or some better term) would mean, actually use the instance from the other file, don't create a new container for it. share would not allow you to customize the definition at all.

share is basically what is supported by #758 where as include is for supporting the point 1 from the OP.

[aanand]

Yep! This is exactly what I was driving at when I said this proposal doesn't cover the use case served by #758.

I agree that there should be two separate types. I see the user story for share being:

1. I have a separate app, with its own docker-compose.yml.
2. In my current app, I define a service of type share, with the path to the other YAML file and the name of a particular service. I can refer to it from any other service in my YAML file with links, volumes_from or net.
3. When I start the current app, the service in the other app I referred to is also created/started (along with any dependencies).
4. When I cd to the directory the other app is in and type docker-compose ps, I see the service running, exactly as if I'd typed docker-compose up <servicename> in that directory. I can manage it normally from there using regular docker-compose commands.

As far as naming goes, type: share feels a bit odd. Maybe external is a better fit for this use case, and we should come up with another name for the use case currently covered by external (maybe remote?).

[dnephin]

Cool, that all makes sense and sounds great.

To me external makes sense as it's described in the OP, and remote would be a good name for this "share" type, but I don't have a strong opinion about the naming. They do feel a bit similar though.

[tianon]

What about cases where you don't want to overwrite the shared key? For example, if I want a bunch of common environment variables, but I want to tweak/add one or two after the fact, how would that be accomplished?

[aanand]

Yep: environment, volumes, links etc should have "merge semantics" (which is more complex than simple list concatenation, unfortunately).

[hadim]

Definitively a very usefull feature !

[dmp1ce]

Yes please! I needed a way to do simple volume overrides to switch my project from being totally 'static' to 'dynamic' in some areas. To do this I created a Fig wrapper but I would prefer overrides being supported natively by Fig.

Here is my wrapper example project.

[funkyfuture]

(disclaimer: this may be fully thought through, i urgently need a nap.)

i'd propose a syntax like this, which seems to me more matching to the existing:

db:
  include: common.yml#mysql
  …

where include is an alternative to image and build. i don't see any further use to introduce that type-key atm.

i'm not completely sure if that would also fit to @dnephin's thoughts of shares/externals. but if i got it right, i suppose it would since that include-statement would mean to merge the remaining dictionary of db into common.yml#mysql.

but one thing comes to mind that may be missing in the concept: how could an included configuration reference services in its ultimate neighbourhood and what happens when a pattern is included twice? maybe so:

common.yml

…
debug:
  …
  links: !imports
  volumes_from: !imports

mysql:
  …

sftp:
  …
  volumes_from: !imports

…

docker-compose.yml

web:
  links:
    - db
  …

db:
  include: common.yml#mysql

sftp:
  include: common.yml#sftp
  exports:
    - web
## eventually does:
#  …
# volumes_from:
#  - web

debug:
  include: common.yml#debug
  exports:
    - db
    - web
## eventually does:
#  …
# volumes_from:
#  - db
#  - web
# links:
#  - db
#  - web

one should keep in mind here that variables in the config are also requested when thinking about the placeholder proposed here as !imports (a list, btw).

and one question, @aanand: what's planned on the location of included configs? to restrict it to the directory containing the docker-compose.yml makes no sense, imo.

i could go along with only relative paths and a default where is looked for lastly, e.g. /etc/docker-compose/includes.

---

edited and extended

[aanand]

After some discussion in #1080, we're trying a different tack: see #1088.