client network_interface config doesn't parse sockaddr templates #3675
Comments
|
Same behavior on 0.6.0 and 0.7.1 |
|
Having a similar problem on DigitalOcean. They add a Would be nice to be able to blacklist an IP or have more granular controls over interfaces/IPs. |
|
Yes, this bug makes it hard to rollout Nomad on tier 2 cloud providers like Digital Ocean and Linode. These providers use network interface alias on their VMs. |
|
Error comes from net library here: https://golang.org/src/net/interface.go?s=4532:4585#L153 falls through to line 169 |
|
Looking through net library and I don't see a way to reference a network alias as its own interface. eth1:0 is the eth1 interface according to the net library. nomad/client/fingerprint/network.go Lines 100 to 109 in 167c81a I think we might need to add an additional option network_interface = eth1 Knowing the alias number perhaps we can ask for it in |
|
Having same issue with Linode. Does anyone have a workaround by chance? |
|
@dmitrif You can run your config through |
|
I've been bitten by this as well, is there any update? |
|
I've been researching similar issue, related to this. I have same setup. On adapter eth0, I have public ip and aliased internal one on eth:100. Network fingerprinter pickes this as 2 separate network resources. I would be cool to have some control over this. Because almounst all the time I want to bind everything to internal network. Because all the services are made public by ReverseProxy. But currently everything binds to public IP because its first. So I need to listen on 0.0.0.0 as workaround, so the service is available over internal network. And block everyhing public on firewall. But this also causes issue, that logically I can have two allocations with same port on same machine. One with Public IP and one with Private IP. The second one logically fails to start because of the 0.0.0.0. The simple possisble solution add some part of network resouces to blacklist. Simillarly as https://www.nomadproject.io/docs/configuration/client.html#quot-fingerprint-network-disallow_link_local-quot- |
|
I've run into this when on Linode, too. My node has My service then binds to |
|
Since Linode does not offer a separate network interface device with their private networking setup, the private IP, by default, is added as an alias to public network device, eth0. > cat /etc/network/interfaces
# Generated by Linode Network Helper
# Fri Jun 21 12:44:06 2019 UTC
#
# This file is automatically generated on each boot with your Linode's
# current network configuration. If you need to modify this file, please
# first disable the 'Auto-configure Networking' setting within your Linode's
# configuration profile:
# - https://manager.linode.com/linodes/config/workerpool1-node2?id=15820662
#
# For more information on Network Helper:
# - https://www.linode.com/docs/platform/network-helper
#
# A backup of the previous config is at /etc/network/.interfaces.linode-last
# A backup of the original config is at /etc/network/.interfaces.linode-orig
#
# /etc/network/interfaces
auto lo
iface lo inet loopback
auto eth0
allow-hotplug eth0
iface eth0 inet6 auto
iface eth0 inet static
address 72.14.190.210/24
gateway 72.14.190.1
up ip addr add 192.168.167.74/17 dev eth0 label eth0:1
down ip addr del 192.168.167.74/17 dev eth0 label eth0:1This means that Nomad (+ private ip for services) is not usable on Linode as is (cc @angrycub), unless we're able to tell nomad to use aliased ip based on either their label, like Until such a facility is built into nomad, as a workaround, the whole cluster would require a new dummy interface for nomad to pick up the private address from, which is undesirable. Is there another workaround? |
|
Here's what I ended up doing to get this working. Requires that you know the IP you want to assign for your nomad scheduled tasks. Add a dummy interface with private ip cidr> ip link add dummy10 type dummy
> ip addr add 192.168.x.x/17 dev dummy10 # Linode uses /17 for private networkEdit nomad config to have the scheduler use ip from dummy10 for allocating tasks> vim /opt/nomad/config/default.hcl
# ...
log_level = "DEBUG"
client {
enabled = true
network_interface = "dummy10"
}
# ...Read the debug logs to ensure expected behaviourLooks like we got it right. Our nomad scheduled job (haproxy) is serving on private ip set in dummy10 interface. This was easier than I thought it would be. Lovely. |
|
In order to persist the above mentioned dummy interface across restarts, etc., I used ansible to create a systemd managed network configuration across all of the nomad client nodes. The result was something equivalent of this on each client node: > cat /etc/systemd/network/10-dummy10.netdev
[NetDev]
Name=dummy10
Kind=dummy> cat /etc/systemd/network/20-dummy10.network
[Match]
Name=dummy10
[Network]
Address=192.168.x.x/17> systemctl daemon-reload
> systemctl restart systemd-networkd |
|
@Gurpartap Seeing as there is only one private IP, and dummy interfaces drop all packets sent to them, this doesn't work as networkd then disables the eth0:1 alias. |
|
@sean- Not sure I follow.. Which param would we be using in this case? |
Works as intended on my consul nomad cluster on Linode. Nomad does not send any packets on the Lines 188 to 189 in ee7803d nomad/client/fingerprint/network.go Lines 63 to 67 in 33f550f |
|
For sure. But do you have anything else bound on the private IP such as
consul?
On Sun, Jun 23, 2019 at 18:13 Gurpartap Singh ***@***.***> wrote:
@Gurpartap <https://github.com/Gurpartap> Seeing as there is only one
private IP, and dummy interfaces drop all packets sent to them, this
doesn't work as networkd then disables the eth0:1 alias.
Nomad does not send any packets on the network_interface. Afaict, this
config is only used for determining the nomad client's IP addresses (which
is also assigned to tasks).
Works as intended on my consul nomad cluster on Linode.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3675?email_source=notifications&email_token=AAFADOHBEDG32LTAUYVLT7LP37YO3A5CNFSM4EI776Z2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYLIIMI#issuecomment-504792113>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAFADOHSKLCET2KRYP3WNRDP37YO3ANCNFSM4EI776ZQ>
.
--
…_________________
Dmitri Farkov
647.898.5054
|
|
Hey there Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this. Thanks! |
I have to If someone knows a way to ensure |
|
Hey there Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this. Thanks! |
|
This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem 👍 |
tgross
added
the
stage/accepted
|
I wanted to do some follow-up on this to clarify the issue. As others have noted, Nomad's network fingerprinting relies on the golang stdlib to parse the network interfaces. At We can see the results of this if we spin up a DO droplet with private networking and IPv6 enabled. Networking configuration on the host: A simple golang program to read out the interfaces: package main
import (
"fmt"
"net"
)
func main() {
ifaces, err := net.Interfaces()
if err != nil {
panic(err)
}
for _, iface := range ifaces {
fmt.Printf("%#v\n", iface)
}
}And the results: Using a log_level = "DEBUG"
data_dir = "/var/nomad"
# this will bind correctly
bind_addr = "{{ GetAllInterfaces | include \"name\" \"eth0\" | exclude \"type\" \"IPv6\" | sort \"-p\
rivate\" | limit 1 | attr \"address\" }}"
server {
enabled = true
bootstrap_expect = 1
}
client {
# this will not... see below
network_interface = "{{ GetAllInterfaces | limit 1 }}"
enabled = true
}Unfortunately although For clarity I'm going to rename this issue title so it can be properly triaged for future work. cc @galeep |
tgross
changed the title
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
0.7.1
Operating system and Environment details
Ubuntu 16.04
Issue
Unable to specify network_interface option for an alias interface, eth0:1
Reproduction steps
Assign an interface eth0:1 (Linode uses these for private address space)
The text was updated successfully, but these errors were encountered: