Nomad fs and related doesn't work with NATed client nodes.

[ninoles]

Nomad version

Nomad v0.5.1

Operating system and Environment details

Windows 10 Pro (but unrelated)

Issue

Accessing nomad through an NATed address (like in the AWS environment) used the WAN address of the node.

Reproduction steps

Start a node in AWS and exposed the public address to your local node. Now, try to fs an allocation

$ bin/nomad fs -address http://52.23.211.160:4646 -f d4ac3fb9-05f1-b1eb-183b-a7a1fe97ae1f alloc/logs/update.stdout .0
Get http://10.0.2.82:4646/v1/client/fs/stat/d4ac3fb9-05f1-b1eb-183b-a7a1fe97ae1f?path=alloc%2Flogs%2Fupdate.stdout.0: dial tcp 10.0.2.82:4646: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

[ninoles]

I'm not sure how this should work, since the wan address is usually configured through consul (in the case of AWS it is available however through a node attribute, but I think that's too specific as a solution).

May be a special "used consul translated addressed" option could be used ?

[diptanu]

@ninoles I guess you want Nomad CLI to use the public or EIP of nodes? If so please use the advertise address client option to specify the public IP when configuring Nomad client. That should make the CLI use the EIP instead of the private IP.

[ninoles]

The tricky part here, is that doing so will make the other node start using the external network for accessing each other (with charged bandwidth, etc.). At least, that's how I understand this is working.

Right now, since there is no way in nomad to advertise wan address in additional to internal address, I think the best way to achieve what I need is through a bastion host. In any way, it's probably also more secure this way.

[ketzacoatl]

One possibility: if the client can communicate with the nomad servers, have the servers relay log/fs information from the target remote agent.

[sheerun]

Related: #1640

[dadgar]

Thanks @sheerun. Lets close this in favor of that issue.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.