Setting container hostname to service_name-xx instead of container-id

[sapvs]

For this sample compose file

services:
  my-app:
    image: my-image:0.1

  his-app:
    image: his-image:0.1

Compose creates my-app-1 container for my-app with host-name as container-id say rndmcntrid

This container is reachable from other services like his-app-1 with my-app, my-app-1 or rndmcntrid.

Can we have compose set container hostname as service-1? e.g. in case of my-app above, hostname be set to my-app-1 instead of rndmcntrid

Docker Compose version v2.10.2

[milas]

Set the hostname field. See the docs for details: https://docs.docker.com/compose/compose-file/#hostname

services:
  my-app:
    image: my-image:0.1
    hostname: my-app-1

  his-app:
    image: his-image:0.1

[sapvs]

Oops! The description I gave made it look as 'question: how to set hostname for container' while what wanted was:

Is it possible to enhance compose to set hostname to my-app-1 for containers without explicit hostname configuration in compose file, instead of container-id.

And in case of multiple replicas set as my-app-1, my-app-2 etc. for the replicas.

[milas]

Ah, ok, I'll re-open and mark this as a feature request.

Is there a specific reason that the built-in name resolution is inadequate and you want the hostname explicitly set? Trying to understand the use case here.

[sapvs]

Thanks @milas.

Explanation follows, but before that would like to mention that this may be a specific use case, as I have not much idea if there are other systems, that rely on hostname like rabbitmq.

Came across this when trying to cluster rabbitmq using its classic peer discovery, using compose.

RabbitMQ configuration

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
cluster_formation.classic_config.nodes.1 = rabbit@rabbit_1  
cluster_formation.classic_config.nodes.2 = rabbit@rabbit_2

rabbit_1, rabbit_2 should be hostnames of the rabbit instances. Doesn't work with alias app-rabbit-1 or service name if hostname is not explicitly set.

compose file
2 services with explicit hostname.

# compose project name = app
  rabbit-1:
    image: rabbitmq:management-alpine
    hostname: "rabbit_1"
  rabbit-2:
    image: rabbitmq:management-alpine
    hostname: "rabbit_2"

For 2 instances 2 services, for 5 instances 5 service definitions. No benefit of scaling can be taken here.

---

If compose could set hostname app-rabbit-1, app-rabbit-2 etc. then rabbit service could be declared as

# compose project name = app
rabbit:
    image: rabbitmq:management-alpine
    deploy:
      replicas: 3

And peer discovery configured as

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
cluster_formation.classic_config.nodes.1 = rabbit@app-rabbit-1  
cluster_formation.classic_config.nodes.2 = rabbit@app-rabbit-2
cluster_formation.classic_config.nodes.3 = rabbit@app-rabbit-3

We can use app-rabbit-X as hostnames in config in this case, because we know compose will set hostname for containers as that.
Also this opens up option for scaling.

[milas]

Awesome, thanks so much for the detailed explanation!

I can't guarantee we will take this feature, but having well-defined use cases helps us prioritize requests.

For anyone else who has reached this issue and is interested, please use the 👍 reaction on the main issue. If you have other use cases not covered, please feel free to comment as well.

[milas]

Some possible implementation notes:

As of now, this is what the spec has to say on hostname:

hostname
hostname declares a custom host name to use for the service container. MUST be a valid RFC 1123 hostname.

Arguably, this is already underspecified for usage with replicas.

At a first glance, I'd say we should he appending the replica # (if > 1) to the custom hostname (if provided) to prevent duplicates.

Then you could set something like hostname: foo- to get foo-1, foo-2, etc.

Any bigger change (e.g. to allow a templating variable, maybe similar to mktemp?) would likely necessitate a bigger spec update/change rather than clarification/amendment.

[sapvs]

@milas the issue is closed, could you please re-open it.

[tony-sol]

Sorry for interrupt, but maybe ability to set a pattern for hostname may solve the problem?
Thats actually a question which i goggled for and found this issue:

• is there a way to generalize hostname value in templates, e.g.

x-templates:
    common: &x-common
        env_file: .env
        hostname: 'my-{{ .Service.Name }}'
        # etc.

services:
    service1:
        <<: *x-common
        # etc.
    service2:
        <<: *x-common
        # etc.
    service3:
        <<: *x-common
        # etc.

and get as a result service1 with my-service1 hostname, service2 with my-service2, etc. ?

Some kind of "helmy" templating, i would say

[ndeloof]

We really, really don't want to introduce a templating language inside the compose file format

[ndeloof]

@milas we indeed could compute hostname as container is created with some custom, dedicated logic for this use-case
Hostname: strings.Replace(service.Hostname,"XXX", number)

I wonder which character to use to prevent any side effet, and make it clear a replacement will take place here (as I can guess we will get requests to support this in a few other places)

[ndeloof]

I ran a quick test:

services:
    base:
        image: nginx
        scale: 2
    master:
        image: alpine
        command: ping localhost

ran the stack then exec into my "master" node:

docker compose exec master ash
/ # ping chose-base-1
PING chose-base-1 (172.18.0.4): 56 data bytes
64 bytes from 172.18.0.4: seq=0 ttl=64 time=0.258 ms
64 bytes from 172.18.0.4: seq=1 ttl=64 time=0.425 ms
^C
--- chose-base-1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.258/0.341/0.425 ms
/ # ping chose-base-2
PING chose-base-2 (172.18.0.5): 56 data bytes
64 bytes from 172.18.0.5: seq=0 ttl=64 time=0.573 ms
64 bytes from 172.18.0.5: seq=1 ttl=64 time=0.419 ms
^C
--- chose-base-2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.419/0.496/0.573 ms

service container is registered on network with alias {{project-service-replica}} you can use for service discovery as requested, without any additional configuration or explicit hostname

[ndeloof]

just a quick note that container index might not start by 1. If for some reason you delete service_replica_1 and run another docker compose up, a new container will be created to match the requested scale but as service_replica_n+1

[dbwodlf3]

not now it is supported natrually?

[ndeloof]

Do you have a use case for this feature you could describe here ?
As container is connected to network with service alias, I wonder why setting hostname as requested would be useful

[dbwodlf3]

Here's the scenario:
We have a Master Node communicating with Slave Nodes (Host).

In most cases, cluster software is designed as a single process. Sometimes it utilizes all hardware resources, but often there are times when it's blocked(Networking, IO, Sync.... etc).

Here's the structure: Master Node -> [Docker Compose (Host Machine)] -> Slave Nodes (Container).

In this setup, Docker Compose acts as an intermediate level manager node. However, there's no information about the original software's host machine. So, I'd like to add metadata to the slave nodes regarding their host machine.

For example:

Host1-1
Host1-2
Host1-3
Host1-4

Host2-1
Host2-2
Host2-3
Host2-4

Of course, this can be achieved through scripting, but it's a bit complex.

(I have already solved this problem... just using scripting.)

I think if Docker Compose supported more systemable environment variables(like service name, replica index, etc...), it would simplify the problem. (Almost the way is good. of course, more complex things have to use theres way)

[ndeloof]

IIUC you'd like to get some metadata in container to let you know about the service and replica number being ran. Can you please describe how you would use this, and why this can't just be achieved by external monitoring (which has easy access to container labels)?

[thaJeztah]

Perhaps this could use the templating option as they're supported by swarm services https://docs.docker.com/engine/reference/commandline/service_create/#create-services-using-templates

[ndeloof]

before considering feasability and syntax to support this, I'd like to understand the benefits having such a feature. Actually I'd prefer we get a generic introspection mechanism for a container to know about it's usage context, I would typically suggest we rely on the legacy /.dockerenv file created at all containers root (for legacy reason IIUC) or something comparable

[thaJeztah]

Yeah, introspection is a topic that returns frequently, but still needs a lot of thinking. The templating options mean that the user remains in control over what information "leaks" into the container. (There were some (probably similar) use-cases for users running swarm-services).

I agree that we should have a good understanding of use-cases before committing to anything (but thought I'd comment the suggestion so that IF we would consider, we could re-use existing principles, instead of creating something new "from scratch').

[dbwodlf3]

Actually I am trying to use Azure Pipeline Agent on Host machine, the agent is designed to perform 1 task at 1 time. So I run this with docker compose. 1 host machine, 4 agent docker containers. If some agent dies, then I have to check the agent is where on it.

[ndeloof]

@dbwodlf3 are you running those agents as replicas for a single service? How would hostname help you managing a died agent? Don't you just restart it with compose up?

[ndeloof]

Closing this feature request as "not planed" as there's no clear use-case requiring such a change. Feel free to open a follow-up issue if you have one you can share

[matifali]

@ndeloof, please close again as not planned. The purple colors indicate that it is closed as done.

[womblep]

@ndeloof could you please reopen this?
The author put a very good description of the use case he (and I) are trying to solve with rabbitmq as an example.
I think the discussion got a little distracted.

IMHO it could be solved with a setting in the deploy section something like "set-hostname: true" which set the hostname of each node.

[ndeloof]

@womblep as commented on #9858 (comment) replicas can be accessed as <service>-<replica> and setting host name wouldn't help from this use-case

[womblep]

@ndeloof yes setting the hostname does help. RabbitMQ expects that the reverse-dns name is the same as the hostname to be able to resolve cluster members when setting up the cluster using the DNS peer discovery protocol.
The process is:

1. Create the node name using the hostname
2. get the A record used to discover peers (with docker compose the setting deploy->endpoint_mode= dnsrr works)
3. With each IP address contact the peer and get it's node name then do a reverse-DNS on the IP address and see if the 2 match (with docker they dont)
4. Only if they match will the peer be used

[ndeloof]

@womblep that's a weird requirement by rabbitmq, is this documented anywhere ?

According to https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-dns there's no "see if the 2 match" constraint you describe
I also can see there are alternative peer discovery plugins for cloud infrastructures (kubernetes, aws ..) which could also be useful for a compose deployment (https://github.com/rabbitmq/rabbitmq-server/tree/main/deps/rabbitmq_peer_discovery_consul)

[mfontana-elem]

I also have an usecase for this.

I am running a local slurm cluster using docker compose. For this to work, the compute node which is starting the daemon needs to be aware of its hostname, which I cannot do if the compute-node service has a scale argument (otherwise hostname: suffices).

When running with podman I don't have this issue, as the name of the service compute-node-1 gets added to /etc/hosts as an alias for my $CONTAINER_ID. This is not the case for docker and there is not an easy fix short of the hack of querying DNS for the pattern of compute nodes names and comparing for the interface's IP.

This year Slurm added a kind of "autodiscovery" service as mentioned here, but unfortunately I am stuck on an older version for compatibility with other systems. In any case, these kinds of issues seems recurrent when building different kinds of clusters, as previously exemplified in this thread.

[ndeloof]

@mfontana-elem you're basically asking for moby/moby#2335 which has been rejected by maintainers.

[mfontana-elem]

Thanks for the link. That is one possibility. The other one being hostname attribute in the compose file specification being compatible with scale (i.e. hostname: compute-node resolving into compute-node-1 for the first member in the ensamble if scale was provided).

[mfontana-elem]

Thanks for the link. That is one possibility. The other one being hostname attribute in the compose file specification being compatible with scale (i.e. hostname: compute-node resolving into compute-node-1 for the first member in the ensamble if scale was provided).

For the time being, just in case it helps someone in the same circustances, I resorted to adding this very ugly line in my entrypoint with which I wouldn't be able to sleep if this wasn't for a local development environment:

for IP in $(hostname -I); do
    echo -e "$IP\t$(hostname)$(dig -x $IP +short | cut -d'.' -f1 | xargs)" >> /etc/hosts
done

(This also force me into adding bind-utils to my image, but this being already a fat image -- HPC packages are large and required-- it isn't a big deal)

[yurenchen000]

It is useful feature when deal with --scale
cause there is no way to set a pattern for hostname(or other fields) other than const value.

docker swarm seems has '{{.Service.Name}}-{{.Task.Slot}}' things