address_mode = driver & ${NOMAD_IP_myservice} not working as expected.

[james-masson]

Nomad version

v1.1.0

Operating system and Environment details

Debian 10 or RHEL 7

Issue

Variable interpolation around IPs/networks are broken with service using address_mode = "driver"

eg. ${NOMAD_IP_myservice} report the IP of the host, rather than the IP of the container.

This seems related to similar fixed bugs like #8615

What I'm looking for is the ability to use the container IP as a var somewhere else in the Job definition - in this case in Consul service tags.

Reproduction steps

Create a docker service with address_mode = "driver", and a static network def "myservice"

Try to use ${NOMAD_IP_myservice}

Expected Result

the container IP is provided

Actual Result

the Nomad host IP is provided.

[DerekStrickland]

Hi @james-masson,

Thanks for using Nomad!

Could you try setting address_mode = "alloc" and see if that resolves your problem? Looking at these docs, I'm wondering if that would be the more appropriate mode for your scenario.

Thanks!

@DerekStrickland and the Nomad Team

[james-masson]

Thanks @DerekStrickland - but this doesn't seem to play well with named docker networks.

eg. network_mode = "my_network"

cannot use address_mode = "alloc" - no allocation network status reported

I presume I'd have to swich to CNI before this worked?

[DerekStrickland]

Yes, that or bridge. It should support either bridge or cni.

[DerekStrickland]

Hi @james-masson,

Just wanted to follow up and see if a CNI switch is an option for you.

• Derek

[james-masson]

Hi,

thanks for following up...

While it is a technical solution, it involves changing a lot of existing applications and servers in Production.

Logistically, it's a couple of months worth of work to get there, and while valuable in itself - it's not really a solution to the immediate problem.

[DerekStrickland]

I understand. Would you mind posting your actual config files here without any secrets removed so I can replicate and see if there is anything that comes to mind?

[james-masson]

Difficult - air-gapped environmnent - but here's a more fully formed but untested job def.

job "example" {
  datacenters = ["dc1"]

  group "cache" {

    network {
      port "redis" {
        static = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis:3.2"
        network_mode = "my_net"
      }

      resources {
        cpu    = 100
        memory = 256
      }

      service {
        name = "redis"
        port = 6379
        address_mode = "driver"
        meta {
          scrape_url = "http://${NOMAD_ADDR_redis}/foobar"
        }
        check {
          name     = "redis-check"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
          port     = 6379
          address_mode = "driver"
        }
      }
    }
  }

[DerekStrickland]

Can you also share your config file where you define my_net?

[tgross]

Hi @james-masson! I wanted to follow-up on this issue...

It looks like you're using a userspace networking plugin for Docker here, right? But you have a group.network block that's using the default "host" mode, so that implies the IP address should come from the host. The group.network block configures the network namespace and that's where Nomad gets all the variables from, so typically I'd expect to see cni/whatever in the network mode here, depending on what kind of networking plugin you're using.

Can you explain in a bit more detail how your networking is configured here? If we knew a bit more about what you're doing that'll help us give you some workarounds.

Also, btw it doesn't looks like the scrape_url metadata value you have there will be interpolated, so I don't think this specific approach would work even if the variable was populated the way you expect.

[tgross]

This has been open for a while waiting on response, so I'm going to close it out. Please feel free to reopen if you have more info.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.