networking model

[gurvindersingh]

Thanks for putting out a new project in resource manager space. I am interested to learn more about Nomad and specifically its networking model. Currently it seems that containers/jobs share the host network and expose the ports based on that. I am wondering if having each job/container having its own address make sense in nomad and if yes can you put up some design documentation how you plan to achieve that.

Currently it seems nomad leaves networking part to runtime e.g. docker, so if you user docker libnetwork, calico etc each container can have its own IP and be route able but does nomad has plan to enable such use case.

Having some design documentation about networking, storage etc would be really nice, I do understand it quite early but nomad has peaked interest so would like to learn more about it :)

[cbednarski]

@gurvindersingh Thanks for opening an issue.

Networking is pretty complicated and networking scenarios vary widely between deployments. For example, deploying Nomad in AWS comes with a certain set of networking assumptions which are different than those you would find in a physical datacenter with IPv6 capabilities, or an overlay network like Weave.

Our goal with Nomad is to be compatible with all of these. Nomad's core support is based on host interfaces. Our expectation is that IPv6 will provide each Nomad node a range of IPs to allocate to different jobs so each job can have its own IP.

Additional networking scenarios like overlay networks, etc. will be supported through a plugin system (which is not yet implemented). We'd like to support the most common patterns in Nomad core but so far there isn't a clear winner in this space.

It's helpful for us if you share your use case(s) so we can make sure the plugin system supports them.

[nixoz]

Great job as usual guys! Thank you for another great tool.
@cbednarski, can i ask a question?

We are now under consideration of Nomad as a container orchestration, but our scenario depends on overlay networking, which is not supported at the moment. Is it possible to know the plans for this feature implementation? If needed we can try to collaborate on that subject.

Please give me know.
Sergey.

[far-blue]

Hi guys - so very happy I stumbled on this project while looking into some Consul details! Like many I'm trying to wrangle a multi-host docker cluster but find mesosphere and kubernetes don't 'feel' right. I love Consul and already use it for service discovery - and Nomad seems almost exactly what I'm looking for.

But, like others, I really need to make use of Docker's new overlay network facilities. I'm also very keen on the way docker-compose is doing things with a 'job' (to use nomad terms) having its own private network space with then a limited number of ports from a limited number of tasks (maybe even only 1 port on one task) being available on a global network. Those globally available host/port combinations would be the only ones registered as the 'service' in Consul.

I really would like to see such a capability added to nomad soon so I can use it :)

[teulaert]

We have been experimenting using Nomad and Docker configured with IPv6, which seems to work very well. What would make it perfect in our case is that Nomad would register the services in Consul using the IPv6 address of the container instead of the container host IP (or we should have the option to).

This way we can use Consul for retrieving the actual IPv6 address of the container and configure our infrastructure with it. We have been testing this by monitoring container start actions on our hosts and then updating the records (Nomad put in Consul) using the GlobalIPv6Address Docker gave us. And guess what everything seems to be working fine :-) Service monitoring works, and the DNS nicely returns AAAA records.

For us the functionality shouldn't be more complex than this. We like it that Nomand doesn't bother the network part, cause Docker does it for us. Ultimately we would like to choose the MAC adres of the docker container, so we can even determine what IPv6 address the container is going to get. But futhermore, i currently don't see what Nomad needs on the IPv6 part to be functional.

What do you guys think?

[wido]

Seems like a good idea @teulaert !

With IPv6 there is no need for any host proxy or NAT. You want the container address so you can directly talk to it.

[camerondavison]

Has there been any more work on plugins to get something like weave to work, for the AWS use case which cannot use ipv6?

[far-blue]

I think the IPv6 suggestion above is a specific example of a good general philosophy that Nomad should give the option to register the container's IP (v4 or v6) with Consul rather than the host's IP address. This way, whichever networking model is used things should work. If jobs or task groups were to have their own subnets then an additional option could be the subnet's gateway ip.

Separate to this, Docker does support plugins for networking and for volumes (and, of course, with release 1.11, soon for alternatives to runc). If nomad doesn't at the very least support the basics needed to work with docker plugins then I think over time it is going to become marginalised in the container scheduling space. As it is, Kubernetes, Mesos and Swarm all have mechanisms for supporting docker plugins.

[wido]

Indeed @far-blue. That would be the best option. I want to know the IPv4 and IPv6 address of the container, that's all which is required.

In the IPv6 case it is routable space and the host's IP or name is never involved.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.