Service registration for IPv6 docker addresses

[42wim]

Hi, we're using nomad / consul in almost IPv6-only environments and are still using glidelabs/registrator to make our consul registrations. (just the basic docker bridge, not with any network overlays)

Since @schmichael #2709 PR has been merged, it's now also possible to have nomad register IPv6 container addresses with not much code change.

I made a simple patch 42wim@f991a0a and it seems to work fine. (easy to test in our env because only impacts nomad clients not servers)

This would also fix the ipv6 issues in #511, #1124 and #646 ;-)

I guess a better way is to add a DockerDriverConfig key to https://github.com/hashicorp/nomad/blob/master/client/driver/docker.go#L164, something like UseIPv6Address ?

Comments ?

[schmichael]

Hi @42wim,

Thanks for the patch! I'm curious whether you're aware of additional address keys we might want to support? From the Docker docs it appears there's also secondary IP addresses and the link-local IPv6 address. I can't imagine anyone wanting to advertise the link-local IPv6 address, but perhaps the secondary addresses are useful?

Either way a new top-level DockerDriverConfig field is the way to go. If we determine the IPv6 address is the only other useful address to advertise, a boolean like your label is probably ideal. privileged is a good example.

Otherwise we'll probably want some sort of enum address: ipv4 (default), ipv6, linklocal_ipv6, secondary_ipv4, or secondary_ipv6

That would look something like the NetworkMode field but force a known enum value instead of allowing for arbitrary values.

[42wim]

Personally I doubt anyone is using the SecondaryIP(v6)Addresses
(even docker/moby isn't sure anyone uses this https://github.com/moby/moby/blob/master/api/swagger.yaml#L920-L934)

The link-locals, like you already said, aren't very useful for being registered to services.

So I think the UseIPv6Address boolean would be good enough.

If you agree with this conclusion, I'll make a PR :-)

[schmichael]

Thanks for digging into that @42wim! A boolean sounds good to me then.

[42wim]

PR #3790 is added, also updated some documentation and just rebased again to master.

[tsujp]

This isn't working for me, when I view the addresses

$ sudo docker inspect --format '{{ .NetworkSettings.GlobalIPv6Address }}' server-250d454d-46f6-8fc4-26a1-3b4a38527733
2a01:4f8:c2c:bafc::3

Which shows that the container started by nomad has IP address 2a01:4f8:c2c:bafc::3

Yet when I inspect the job allocation it shows empty Addresses

ID                  = 250d454d
Eval ID             = 597d08e7
Name                = ip6-echo.example[0]
Node ID             = 1e1c3ab8
Node Name           = centos-4gb-nbg1-1
Job ID              = ip6-echo
Job Version         = 824636069392
Client Status       = running
Client Description  = Tasks are running
Desired Status      = run
Desired Description = <none>
Created             = 47s ago
Modified            = 36s ago

Task "server" is "running"
Task Resources
CPU        Memory           Disk     Addresses
0/500 MHz  4.3 MiB/256 MiB  300 MiB  

Task Events:
Started At     = 2019-10-01T08:57:49Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2019-10-01T10:57:49+02:00  Started     Task started by client
2019-10-01T10:57:48+02:00  Driver      Downloading image
2019-10-01T10:57:48+02:00  Task Setup  Building Task Directory
2019-10-01T10:57:48+02:00  Received    Task received by client

Here's the jobfile for this

job "ip6-echo" {
  datacenters = ["dc1"]

  group "example" {
    task "server" {
      driver = "docker"

      config {
        image = "localhost:5000/express-test"
        advertise_ipv6_address = true
      }

      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
        network {
          mbits = 10
        }
      }

      service {
        name = "ip6-echo-service"
        port = 5555
        address_mode = "driver"
      }
    }
  }
}

[schmichael]

Yet when I inspect the job allocation it shows empty Addresses
-- @tsujp

Since you're using advertise_ipv6_address in your Docker config and address_mode = "driver" in your Consul service stanza, Docker will assign your container an ipv6 address and Consul will advertise that address in the service registration. That part should be working properly I hope.

Unfortunately the only addresses in the CLI's status output are the ones assigned by Nomad's scheduler, not necessarily the address Docker assigns to the container. Now I'm not sure why the address is empty in your CLI output.

Could you open a new issue with all the information in this comment as well as the interfaces and IP addresses on the host machine? The node status and network_interface configuration for the client is useful as well.

Thanks!

[tsujp]

@schmichael roger that, I've made a very comprehensive issue here: #6412

Please check it out

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.