Document exact network patterns between nomad agents

[sheerun]

Hey,

I'm trying to understand the networking between nomad nodes, and its impact on security and communication for nodes behind a NAT. I saw some documentation about how nodes communicate with each other, but there's no single page with exhaustive reference.. Could you create such page and confirm / add something to what I found?

1. Nomad agents can communicate only in 3 ways: http api (default tcp port 4646), rpc api (default tcp port 4647), and serf (default tcp and udp port 4648)
2. http, rpc and serf communication is unencrypted and unauthenticated by default. It is possible to enable TLS encryption for http and rpc, but not authentication. Serf traffic can be encrypted and authenticated with a shared token.
3. Clients don't communicate with servers using gossip. They use only http (for what?) and rpc (for what?)
4. How clients receive tasks from servers? How hey send stats to servers? From what I found some commands, like nomad logs, require servers being able to dial clients. But seems that in general its clients that dial servers.
5. What are exact dial requirements? Should servers be able to dial clients on http api? For what commands? What about dialing rpc? For what commands?
6. Can one limit communication between servers and clients to only http(s)?

[dadgar]

Hey @sheerun,

Agree that there should be more detail. You can see some here: https://www.nomadproject.io/docs/internals/architecture.html

Client/servers communicate exclusively via RPC. The HTTP endpoints are used by the CLI. The client has an HTTP endpoint to serve the CLI command and to server logs.

[schmichael]

Hopefully the new TLS guide in #2923 clears up point 2. It's live on the site: https://www.nomadproject.io/guides/securing-nomad.html

[stale]

Hey there

Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.

Thanks!

[skluck]

I would still very much like documentation in what scenarios server -> client communication is needed. I believe this was made (more?) optional in a recent version but the docs do not mention it (and in fact do not imply any server -> client connection is ever required).

[schmichael]

@skluck server -> client connections are not made. Servers route RPC requests to clients (such as for log streaming) via the connection clients make to servers. Migrating allocation directories is the only operation I can think of that only works if two clients are able to directly communicate (to avoid saturating server's bandwidth copying GBs or TBs of data).

The CLI can communicate with servers and the requests will be routed to the proper client via the client's connection to the server. Clients still listen on the configured http port by default (4646), but this can be firewalled without hurting the client's functionality.

[skluck]

Ah yes that is what I was thinking of - log streaming, thank you. I was going spelunking and #4482 was the issue I was remembering when searching for clarity of which advertise ports are critical vs noncritical.

[stale]

Hey there

Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.

Thanks!

[stale]

This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem 👍

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.