Why is NetworkManager disabled?

[cgwalters]

Can the doc link to a rationale?

One answer might be:
https://github.com/openshift/openshift-ansible/blob/master/playbooks/libvirt/openshift-cluster/templates/user-data#L22

But this is not too hard to express via NM. Is there anything else?

The reason I bring this up is current RHEL 7 defaults to NetworkManager on bare metal installs, and Atomic uses it even in cloud for consistency.

[dcbw]

Also note that with Fedora 21 and later, the legacy initscripts are not enabled by default. This means that if you use a Fedora VM image and a default Vagrant you will be left with no networking on one interface because Vagrant adds NM_CONTROLLED=no in templates/guests/fedora/network_static.erb and templates/guests/redhat/network_static.erb.

With RHEL7+ and Fedora 21+ (and likely earlier versions) it is not necessary to set NM_CONTROLLED=no because NetworkManager will not interfere with external interfaces, and NM is also quite capable of setting up static and DHCP connections from ifcfg files.

[jorgemoralespou]

Just faced same problem, and In any case, instructions that say:

Remove NetworkManager: "yum -y remove NetworkManager*"

Should be removed in any case, and replaced by disable NetworkManager, if at all there is a reason.

systemctl stop NetworkManager
systemctl disable NetworkManager

But I agree that you can add DNS configuration for NM to control resolv.conf and avoid any other problems.

[dcbw]

I'm pretty sure there isn't a great reason to disable NM anymore inside a VM. If anyone can come up with one, we'll fix it in NetworkManager.

[thoraxe]

I'll try to do some testing with NM enabled to see if I can find any problems. Most of the avoidance of NM has to do with what it does with DNS and etc, but I think that's just me not understanding the config-time options.

[dcbw]

For DNS, resolv.conf is a shared resource that on many systems changes depending on what interfaces are active. That's what NM has to update it, and why often it cannot just be written once and left alone. Of course, in cases where you know the DNS info will never change and are static, you have a couple choices:

1. the static DNS info can be put into the connection config files themselves, eg into /etc/sysconfig/network-scripts/ifcfg-* or /etc/NetworkManager/system-connections/* in which case the DNS information from all connections gets combined into /etc/resolv.conf by NM. This is preferred since NM can combine automatic (DHCP, SLAAC, etc) DNS information with adminstrator overrides.

2. set dns=none in /etc/NetworkManager/NetworkManager.conf in the [main] section, restart NM, and then write /etc/resolv.conf all you want. NM will no longer touch resolv.conf. In the upcoming NM 1.2, a simple SIGHUP will suffice and NM does not need to be restarted, but that's a couple months off and will take a while to filter down to distros.

[thoraxe]

beta4 docs will remove this "requirement" and link to @dcbw comment.

[jorgemoralespou]

@thoraxe
As I haven't seen the the comment yet in beta4 tag, I recommend you to explain different configurations that should be needed in case dnsmasq is needed or not.

• When dnsmasq is needed, resolv.conf in master and node 1 should be dnsmasq server ip (assuming installed on node 2), and node 2 should have DNS entry for the public DNS server.
• When dnsmaq is not needed. What should be recommended here?

@dcbw What should be the configuration in when creating the demo in environments where 2 NICs are present? (e.g. a Vagrant setup where NIC1 is a NATed mgmt interface and NIC2 is the host only networking)

I think this configuration description could simplify the life of many. :-D

[dcbw]

@jorgemoralespou in the case of 2 NICs you create two ifcfg files, one for each NIC and you can choose which NIC gets the default route by setting DEFROUTE=<yes/no> in each ifcfg file, just like standard old Sys V initscripts. If you have DNS information for either of these interfaces, or if that DNS comes from DHCP, then the NIC that has DEFROUTE=yes is preferred for the DNS information. Also, PEERDNS=no suppresses any DNS information for that interface.

So for a normal OpenShift "vagrant up" where you have two networks, the first a private management network for the provider (be that VirtualBox or libvirt) and the second defined by the Vagrantfile for OpenShift's private use, you'd want:

ifcfg-eth0 (provider management network)
BOOTPROTO=dhcp
DEVICE=eth0
ONBOOT=yes

ifcfg-eth1 (private network from Vagrantfile)
BOOTPROTO=none
ONBOOT=yes
IPADDR=xxxx
NETMASK=xx
DEVICE=eth1
PEERDNS=no
DEFROUTE=no

In this case, since eth1 has suppressed DNS (with PEERDNS=no), all DNS information will come from eth0's DHCP, and eth0 will receive the default route because it has been denied to eth1.

This is all standard Sys V initscripts stuff; NetworkManager doesn't change any of the interpretations of these ifcfg key/value pairs. NM tries to remain 99.5% compatible with the old initscripts network service.

(aside: if some ethernet interface doesn't have an ifcfg file, then NetworkManager will by default run DHCP on that interface at startup unless it has been told not to through /etc/NetworkManager/conf.d configuration with the "no-auto-default" option. This behavior can be easily suppressed by 'yum install NetworkManager-config-server" or by dropping that option into a conf.d snipped in a [main] section. So, you don't even need to write out ifcfg-eth0 if all you want there is DHCP, because NM will just do that for you)

[dcbw]

Also note that DEFROUTE and PEERDNS default to "yes" (like initscripts), so DEFROUTE=yes and PEERDNS=yes are really redundant and the only time you'd see them in an ifcfg file is with "no" to suppress their respective behaviors.

[jorgemoralespou]

@dcbw Thanks so much for the explanation. Now it is quite clear.
Am I wrong to saying that when using a dnsmaq service (let's say in node 2) we have to reverse that configuration for master and node1?

Something like this:

• master:
  eth0(management) PEERDNS=no
  eth1 PEERDNS=yes, DEFROUTE=yes
• node1:
  eth0(management) PEERDNS=no
  eth1 PEERDNS=yes, DEFROUTE=yes
• node2:
  eth0(management) PEERDNS=yes, DEFROUTE=yes
  eth1 PEERDNS=no

Otherwise, I always end up with DNS server set by management interface before the dnsmasq entry set by the host only interface.

Thanks again for your time.

[cgwalters]

Yeah, looks like we need a way to tell NM to order the connection's DNS before the DHCP-retrieved DNS.

[dcbw]

DNS follows the interface with the default route at the moment. And remember that DEFROUTE defaults to 'yes', so in all your exmaples, both interfaces can receive the default route and perferred DNS.

If you ensure that only one interface has DEFROUTE=yes (by setting all others to DEFROUTE=no) then that interface will also get preferred DNS.

What's the goal though? Is the goal to have the non-default-route (host only) connection's DNS before the management (default-route) connection's DNS, but still have all DNS available?

[jorgemoralespou]

I guess that setting DEFROUTE=no on eth0 should be enough.
The goal is to have only one interface providing DNS. But in the host with dnsmasq, the DNS should be provided by the eth0 instead of eth1, as in the other hosts.

At the end, I've set up another host just for dnsmasq, as it simplifies the automation of installation that I'm doing, and it's more close to a real external dns server.

[juddmaltin-dell]

Or be evil and:

chattr +i /etc/resolv.conf

No processes will be able to edit the file.

[cgwalters]

@dcbw There's a number of things going on in this thread - in my case I only have one network interface - AFAICS I need the DNS order reversed to prefer the connection specified before DHCP.

Of course one could simply attempt to control the DHCP response, but that's hard in a cloud environment.

Or alternatively, I found that an easier solution is to configure the host to only use the local dnsmasq, and tell dnsmasq about the DHCP servers. See #274

So...maybe this isn't worth any changes in NetworkManager, particularly given the major change pending for DNSSEC/unbound.

[dcbw]

@juddmaltin-dell at least for NM chattr +i hasn't been required for a long time. You can tell NM to stop touching DNS by setting 'dns=none' in /etc/NetworkManager/NetworkManager.conf in the [main] section. Recent versions of NM will still write out a private resolv.conf to /var

[dcbw]

@cgwalters if there's only one network interface, then in NM-land that means there's only one connection and thus only one place for DNS information. In that case, is your intent that you'd like your manually specified DNS servers to be pre-pended to the DHCP ones? If so, valid point and we should make that possible in NM without hackery.

Also, would dns=dnsmasq help you out there at all?

[cgwalters]

For reference I decided that the conclusively Right Thing is to:

• configure a custom dnsmasq server (similar to the training)
• Set dns=none in NetworkManager.conf and systemctl restart NetworkManager
• Set /etc/resolv.conf on each node to point just to the dnsmasq server

[dcbw]

@cgwalters instead of munging NetworkManager.conf, you can drop a file into /etc/NetworkManager/conf.d named something like "90-dont-touch-resolv.conf" that contains:

[main]
dns=none

and that'll override whatever's in NM.conf (if anything).