[feature] Expose contents of DriverNetwork via API

[dalegaard]

Both the Docker and rkt(with #3256) expose a DriverNetwork structure to Nomad. The DriverNetwork structure can be used to get the actual network address of the container, which is very useful when used in container networking solutions.

Currently this data is only exposed when Nomad performs Consul service registration, but the data is potentially relevant to any allocation.

I therefore propose that the DriverNetwork contents be exposed via the /v1/allocation/:id endpoint and potentially the /v1/allocations endpoint as well. This would allow external systems to monitor the DriverNetwork IP adresses of all active allocations in the Nomad cluster via blocking queries. My personal use case is automatically configuring firewalls and network policies between containers running under the rkt driver. The Consul service discovery mechanism is not really useful in this case, because I need to know about all endpoints at both ends of all connections, even if a given job exposes no services itself(e.g. it updates a database).

I would be up for implementing this, but I would need some pointers as to where to best place the data.

BR.

[schmichael]

I'm afraid - but uncertain - the complexity of this implementation will outweigh its benefits vs its workaround.

Use case / Workaround

As far as I can tell - and please correct me I'm wrong - this is to avoid having to register services in Consul for things which aren't actually services. For example if webapp connects to db, webapp would have to register its IP in Consul simply so the firewall manager could grant it access to db? Seems like only a minor annoyance and Consul's blocking queries and tagging lead to a pretty natural event-based implementation for the firewall manager.

Future workaround: lifecycle plugin

Instead of exposing this data for consumption we could pass it to lifecycle plugins ... once those exist.

That seems like the ideal place to manage firewall rules if the plugins can acquire credentials external from the task invoking them -- as I would assume allow tasks to manipulate firewall rules directly kind of defeats the point.

Implementation

The complexity I'm worried about is that we really expose very little task runtime information from allocations. I think this is it:

• stats via /v1/client/allocation/$alloc/stats
• task events on the allocation

/v1/client/allocation/$alloc/net

Adding a /net endpoint to /v1/client/allocation/.../ wouldn't be too hard. It could just call through to an exported method on the TaskRunner like stats does. However, it's a bit of a weird one-off and requires communicating directly with the client node (nomad alloc-status already attempts to do this, so it's not that weird).

Like Task events

Task events are currently the only metadata I can think of that populate from the task back up to the server. We could add more, but that's a lot of plumbing and more metadata adds more overhead to the servers which must be done carefully.

I'll see what others think... the /net approach seems reasonable, but it's a bit of a weird one-off when if there's an existing workaround until lifecycle plugins exist.

[dalegaard]

I'm afraid the workaround doesn't actually work though, atleast on rkt. To register a service we must, from what I can tell, specify a service port name, which must match a port in the driver port map config. I'm unfamiliar with how docker handles this, but rkt will only let you bind to ports that are registered in the image - and we are talking about images with no ports :-)

So to actually do this, you would have to rebuild all container images to expose atleast a single port. Not so much a big deal for internal images, but problematic if you want to use publicly available images.

I'll just add that I had missed the /v1/client/allocation/... endpoints, having the data under those would obviously work just as well for my purposes. I'm still quite new to Nomad, so remembering the client/server distinction isn't yet automatic :-)

[schmichael]

you would have to rebuild all container images to expose atleast a single port.

Aha! That makes sense and is a big deal.

the /v1/client/allocation/... endpoints, having the data under those would obviously work just as well for my purposes.

After discussing this with @dadgar I think we should hold off on this for now. We'd like to expose all of the metadata generated by drivers somehow, and it will be easiest to pull that off when we refactor the client<->driver interactions prior to adding plugin support.

I'm afraid adding a one off for networks now would complicate maintenance and backward compat.

[dalegaard]

That makes sense @schmichael . I'll close this for now, since it will be subsumed by the more general mechanism later on. Thanks for considering it.

[schmichael]

#6412 and #5862 are related to this, and we're tentatively considering a fix in the 0.11 time frame.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.