docker containers cannot access consul DNS out of the box (not sure if bug or feature request)

[josh-m-sharpe]

Nomad version

1.2.3

Operating system and Environment details

nomad / consul / amazonlinux2
consul DNS works on the host machine seemingly a-ok. Both of these work:

• ping traefik.service.dc1.consul
• dig a traefik.service.dc1.consul

...but they do not work inside any docker container - which includes nomad docker managed containers.

Reproduce:

On a nomad client (with nomad/consul agent running)

$ ps aux|grep "consul agent"
consul   20460  0.4  0.9 781612 73572 ?        Ssl  Apr20 104:26 /usr/bin/consul agent -config-dir=/etc/consul.d/
$ ps aux|grep "nomad agent"
nomad    20496  0.9  1.1 1458444 93492 ?       Ssl  Apr20 196:43 /usr/bin/nomad agent -config /etc/nomad.d

$ dig statsd-exporter.service.dc1.consul | grep "10.20"
statsd-exporter.service.dc1.consul. 0 IN A      10.20.11.131

$ docker run -it --rm amazonlinux:2
bash-4.2# yum install -y bind-utils > /dev/null
bash-4.2# dig statsd-exporter.service.dc1.consul | grep "10.20"
bash-4.2#

Issue

nomad docs (https://www.nomadproject.io/docs/job-specification/network#dns) read: Sets the DNS configuration for the allocations. By default all DNS configuration is inherited from the client host. -- but this doesn't seem to apply to docker containers.

It does also say: DNS configuration is only supported on Linux clients at this time. - does this mean not "docker clients" ?

Is this documentation wrong or should maybe include an asterisk for docker?

If it isn't possible out of the box - what changes can I make to allow docker containers to access their host DNS?

[josh-m-sharpe]

I guess this is somewhat redundant of this:
#7279

But those answers maybe aren't comprehensive enough? Maybe the answer lies in here somewhere - but i'm not really sure at this point

[ghost]

@josh-m-sharpe,
This probably needs an asterisk for docker, because docker inherits all non-loopback entries as mentioned in the linked ticket. Essentially docker strips out anything with 127.0.0.1. The docs should be updated to clarify this, since I think it is a common stumbling block.

The solution boils down to setting consul dns at a private IP instead of 127.0.0.1. Coming soon in Nomad 1.3 is support for variable interpolation of the network.dns values (well it currently doesn't work, but I've got a PR up to fix it). This means that if you set consul to use 0.0.0.0, you can then set the dns to attr.unique.network.ip-address. You may still need to ensure that consul dns is available on port 53, but that's the solution we're going to attempt once Nomad 1.3 is released.

[shantanugadgil]

@josh-m-sharpe If your host machine already resolves foo.service.consul correctly, then for the docker containers what might be needed is passing the first DNS server as the Docker bridge ip, typically 172.17.0.1.

I have a similar setup using DNSMasq:

file: /etc/dnsmasq.d/10-consul

port=53
bind-dynamic
interface=*
server=/consul/127.0.0.1#8600

The following line is APPENDED to /etc/dhcp/dhclient.conf

prepend domain-name-servers 127.0.0.1;

and my Nomad Docker jobs have this:

    network {
      dns {
        servers = ["172.17.0.1", "8.8.8.8", "8.8.4.4"]
      }

[DerekStrickland]

Hi @josh-m-sharpe

Did these (very helpful) posts from the community help you get to a working configuration?

thx @shantanugadgil && @twunderlich-grapl !!!

[pikeas]

I'm in the same boat, running Nomad 1.3.1. DNS works on the host (after following https://learn.hashicorp.com/tutorials/consul/dns-forwarding#systemd-resolved-setup) and not in containers.

This runs counter to https://www.nomadproject.io/docs/job-specification/network#dns:

By default all DNS configuration is inherited from the client host. DNS configuration is only supported on Linux clients at this time.

Also relevant: https://github.com/docker/docker-ce/blob/master/components/engine/libnetwork/resolvconf/resolvconf.go#L35-L39:

// When /etc/resolv.conf contains 127.0.0.53 as the only nameserver, then
// it is assumed systemd-resolved manages DNS. Because inside the container 127.0.0.53
// is not a valid DNS server, Path() returns /run/systemd/resolve/resolv.conf
// which is the resolv.conf that systemd-resolved generates and manages.
// Otherwise Path() returns /etc/resolv.conf.

Regarding Nomad 1.3:

The solution boils down to setting consul dns at a private IP instead of 127.0.0.1. Coming soon in Nomad 1.3 is support for variable interpolation of the network.dns values (well it currently doesn't work, but I've got a PR up to fix it). This means that if you set consul to use 0.0.0.0, you can then set the dns to attr.unique.network.ip-address. You may still need to ensure that consul dns is available on port 53, but that's the solution we're going to attempt once Nomad 1.3 is released.

I''ll try this shortly, it should work as an interim solution. However, this involves:
a) updating Consul to listen on at least one non-localhost IP
b) updating Consul to listen on port 53 instead of 8600, which may break other tools that depend on Consul's default ports
c) updating every Docker task to explicitly set network.dns

This feels pretty clunky and error-prone. Are there other ways to insert Consul DNS lookups into Docker's DNS resolution? For example, running DNS on the Docker bridge IP (which is used by containers to find each other on non-default bridges)?

Nomad + Consul + Docker tasks is the most common use case, ideally this would Just Work!

[ghost]

FYI, the PR I mentioned was #12817, presumably it'll actually be released as part of 1.4.0 but 🤷
I think I may take a look at how much work it would be to allow setting dns settings in the Nomad config for bridge networking instead of just host networking.

[pikeas]

Chewing on this a bit, docker containers are already routable to the host's private interfaces, and we do have a way to insert ourselves onto that path, iptables!

I haven't found the right invocation yet. This doesn't work:

sysctl -w net.ipv4.conf.all.route_localnet=1
iptables -A PREROUTING -d 10.x.y.z -p udp --dport 53 -j DNAT --to 127.0.0.1:8600 -t nat

[shantanugadgil]

@pikeas does this not help?

#12894 (comment)

[pikeas]

I ended up swapping out systemd-resolved for CoreDNS and it's been working well so far. I also realized that Consul Connect is a better solution for 90% of my needs, as it provides for inter-service routing.

[xinau]

My current solution is to setup systemd-resolved (version 247+) as described in the tutorial, configure a extra stub listener for the default Docker bridge interface, e.g. docker0 and configure the docker daemon to use the bridge interface ip as dns server.

/etc/systemd/resolved.conf.d/docker.conf

[Resolve]
DNSStubListener=yes
DNSStubListenerExtra=172.17.0.1

/etc/docker/daemon.json

{
  "dns": ["172.17.0.1"]
}

I've described this approach with a bit more detail here

[ibuamod-tc]

#12894 (comment)

no it didn't work for me

[oxycash]

Is this issue similar to my issue?

https://discuss.hashicorp.com/t/consul-agent-not-resolving-inside-ubuntu-docker-container/51106

[Lord-Y]

Following #12894 (comment) answer, If you guys are using google cloud, /etc/docker/daemon.json you need to be "dns": ["172.17.0.1", "169.254.169.254"] and add in /etc/resolvconf/resolv.conf.d/head this:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "resolvectl status" to see details about the actual nameservers.
nameserver 127.0.0.53

From your docker container, dig command to your services will work and you will also be able to install any OS packages.

[EugenKon]

I checked this solution and it works as expected. It would be nice if this will be added to official documentation here