Switching board causes network not to work anymore

[ffes]

Yesterday I installed a new image on a pi. Network works as expected.

After that I put the sd-card in another pi and eth0 was not working anymore, so no network access was possible.

Searching in the forums resulted in two post that describe (and fix) this problem:

http://www.raspberrypi.org/forums/viewtopic.php?f=28&t=82467
http://www.raspberrypi.org/forums/viewtopic.php?f=6&t=5442

It turns out that the /etc/udev/rules.d/70-persistent-net.rules added a new rules where eth1 is linked to the mac of the second pi and therefore a mismatch with /etc/network/interfaces

Moving the rules-file out of the way causes that file to be recreated again on reboot with the proper mac address.

In my original installation (based on an official foundation image from the early days) this file is not created at all. Switching boards always worked. I haven't tried a recent image.

With this behavior (adding eth1 to the new mac) switching a sd card between different pi's is sort of impossible.

[goranche]

that's "predictable" network device naming at it's best... it's a feature (some, including me, would call it a bug) added to the udev package by the developers of the systemd system management daemon... they have assimilated udev (and many other packages as well) and are introducing stuff like this...

debian has adopted systemd as the SysV-style init replacement for the jessie release, so I'm afraid it's something we will have to get used to

or switch to another (non-debian derived, obviously) distribution / system (which is what I'm exploring)

I'm not sure if this functionality will be changed, given that this is standard debian behaviour (yes, the official foundation raspbian does many things differently), let's see what @diederikdehaas has to say about it

full disclosure: I'm very much against systemd and support the boycott systemd movement

[ffes]

A feature, I see... Never knew about it. But on a pi it doesn't make sense to me. I have several boards and swap cards often.

I'm sure you have seen this section ;-)
Just tried the 4th option (seems the best one) by adding "net.ifnames=0" to /boot/cmdline.txt but that didn't work.

[goranche]

to me it doesn't make sense period 👼

the problem with those options is that they are intended for systemd, not udev, that's why they don't work... I'll try something out on my boards (just got a brand new B+, so I can test stuff again 😁 ) and write about the results here... soon 😉

[goranche]

oh... it seems that using "net.ifnames=0" should work with udev as well, sorry

[diederikdehaas]

Just a quick remark, since I'm REALLY busy:
It is because of udev, but it has nothing to do with systemd

[diederikdehaas]

OT: @goranche and because of that, I won't have time to review your changes this week. It'll have to wait till next week

[goranche]

I won't have time to review your changes this week

no problem 😉 I was REALLY busy a few days back, that's why the changes came in so late ;)

and I know this is a problem with udev, but it was introduced because of the "hijack" of udev by the systemd developers

[goranche]

@ffes the only reliable workaround I found was adding a rm command to /etc/rc.local:

rm -f /etc/udev/rules.d/70-persistent-net.rules

I can't say I like it, but still, it's a workaround

[diederikdehaas]

Then this 'hack' is probably 'better' (add it to post-install.txt):

  if [ -f /rootfs/lib/udev/write_net_rules ] ; then
    echo -n "Modifying /lib/udev/write_net_rules so net rules don't get written ..."
    sed -i 's/\(write_rule "\$match" "\$INTERFACE" "\$COMMENT"\)/#\1/' /rootfs/lib/udev/write_net_rules
    echo "OK"
  fi

All it does is comment out the line which is responsible for creating the /etc/udev/rules.d/70-persistent-net.rules.
Will take a look into it next week and then decide if I want to change it and if so, how.
I do realize that in this case the RPi is used differently then a normal debian system.

[goranche]

this will probably be overwritten when upgrading udev

in this case the RPi is used differently then a normal debian system

well, I guess you know I'm not going to complain about something not being "the debian way" 👼

[diederikdehaas]

true, but I don't think that will happen very often on wheezy. On jessie, that's a different story and that's also when systemd gets relevant.

EDIT: I'm not so sure since it may very well be against debian policy (ie don't touch locally modified files)

[ffes]

Thanks for the work around. I will keep that in mind for the next image. The current image isn't likely gonna change hardware any time soon.

Do I understand correctly the wheezy doesn't really use systemd yet? Does that explain why adding net.ifnames=0 to cmdline.txt doesn't work?

[diederikdehaas]

That is correct. Wheezy does not use systemd and udev's version is 175 and the net.ifnames=0 parameter was added in version 199, thus it does not work in wheezy.

[goranche]

no problem... I'll be using it myself, because I'm building a SD card image that will be used by a client on many boards, and I'll have to make it as robust as possible... this is definitely something I wasn't really thinking about, so thanks for the "heads up" 👍

oh, and I see @diederikdehaas answered about the udev version already 😄

[hleitzell]

I've been using this snippet to fix this problem without issues or complications with upgrades:

# sed -i -e 's/KERNEL\!=\"eth\*|/KERNEL\!=\"/' /lib/udev/rules.d/75-persistent-net-generator.rules
# dpkg-divert --add --local /lib/udev/rules.d/75-persistent-net-generator.rules

Don't know if this is sufficiently the "debian-way" but has been working for me.

[diederikdehaas]

@hleitzell Thank you for that dpkg-divert statement, that's pretty awesome 👍

I'll likely use a different sed statement though, so that not only eth* is exempted but all network interfaces, including wireless. So something like this:

sed -i -e 's/KERNEL\!=\"eth\*|/KERNEL==\"eth\*|/' /lib/udev/rules.d/75-persistent-net-generator.rules

(and change the comment accordingly)
And I feel that a config option disable_persistent_network_naming (or sth like that) is needed since it would change the default behavior, thus it will be for version v1.1.x.

Comments/remarks/etc on this would be appreciated 😉

[diederikdehaas]

On the image from the Raspberry Pi Foundation (Raspbian d.d. 2015-01-31) they removed eth0 and wlan*[0-9] in the device name whitelist section from /lib/udev/rules.d/75-persistent-net-generator.rules and dpkg-divert-ed the file like @hleitzell did to tackle this issue.

[cherylquirion]

We are putting the same image on a large number of Raspberry Pis. Therefore we had to ensure that the /etc/udev/rules.d/70-persistent-net.rules is recreated every time a device boots up. We also move memory cards from one device to another for testing and in the field. Therefore, in the /rootfs/lib/udev/write_net_rules script, we added a line to delete /etc/udev/rules.d/70-persistent-net.rules so that it is recreated every time.

[Mausy5043]

Good idea. I assume you're using sed or echo in post-install.txt to modify the file?

[cherylquirion]

We can't do it as part of the post-install.txt because we swap memory cards between devices in the field if/when required - no copying involved... same memory card in a different device. The 70-persistent-net.rules file has to be deleted at boot. So we had to change the write_net_rules script by adding a line to delete the 70-persistent-net.rules file when the script starts. We override the write_net_rules script in our build process.

[kpfleming]

Instead of modifying that script, you can just remove /lib/udev/rules.d/75-persistent-net-generator.rules. Then the write_net_rules script will never be called.

[Mausy5043]

Is this still an issue?
I ask because I have four Pi's running v1.1.x (cloned on 05MAR2016) and there is no clue in /lib/udev/rules.d/75-persistent-net-generator.rules for the MAC of the device.

[diederikdehaas]

Is this still an issue?

Yes. Just put a sd card from one Pi into another and the network didn't come up.
You should look in /etc/udev/rules.d/70-persistent-net.rules and there you will find the MAC address(es) 😉

[cherylquirion]

It is still a problem. I tested with the latest version and published a workaround that we needed to implement.

[Mausy5043]

And the issue is with the installed system being moved to another Pi, right? Or are you trying to run the installer (again) on a different board?
Sorry I ask, but its not entirely clear to me.

[cherylquirion]

This is on an installed system. We create a release image that is then used on hundred(s) of different Raspberry devices installed in different vehicles.

[Mausy5043]

🆗 thanks

[Mausy5043]

Oh wait
/etc/udev/rules.d/70-persistent-net.rules
NOT /lib/udev/rules.d/75-persistent-net-generator.rules
I've been staring at the wrong file all this time 😁

[diederikdehaas]

I've been staring at the wrong file all this time

Maybe it's time to call it a day? 😜

[goranche]

@Mausy5043 (and others, of course)
it's a (as so many things these days are) a systemd issue (well, udev, really; same difference)... it's called Predictable Network Interface Names and it should be easy to find additional info on this... (just google predictable network interface names 😇 )
there is a section on how to disable it in the document... (and in systemd fashion it mentions 4 methods of doing so, but lists 3... nope, I'm not a systemd fan)

but IMHO we should leave disabling this to the end user... document it, yes, of course, but leave the default state as is in the default installer...

[kpfleming]

I agree, predictable interface naming is very useful. It doesn't matter for the Pi's built-in Ethernet of course, but for USB-attached WiFi interfaces it could be very important, as the interface name could change as a result of moving the interface to a different USB port (when MAC-based naming is not in use).

[dicer]

Maybe it could just be disabled for ethernet then? It is pretty annoying and took me quite a while to find back then.

[goranche]

predictable interface naming is very useful

you must have been sent from the devil... 😁
(but yes... I agree that for many "normal" people it might be useful... I hate it)

[richard-scott]

Here is a way to fix the problem. Udev records the MAC address of a network interface in this file:

/etc/udev/rules.d/70-persistent-net.rules

You need to blank out the contents of this file with this command:

# >/etc/udev/rules.d/70-persistent-net.rules

And then make it unwritable by anyone (including root) with this command:

# chattr +i /etc/udev/rules.d/70-persistent-net.rules

As long as you don't move your network interfaces (i.e. wifi) to different USB slots you should get the same interface name after reboot even after moving SD Card to different hardware.

It is not a fault of the Operating system tho, as this is how it was designed to work.

[Mausy5043]

PR#391 was merged in an attempt to fix this issue.
Closing the issue.
Please feel free to re-open if you find the issue is not resolved.

[goranche]

@cherylquirion I think you'll find that the latest changes will be better suited to what you're trying to do (I have a "large scale" deployment as well, same image, ...)

also, if you'd like, I could give you part of my post-install.txt (and a script or two) that makes the system readonly (optionally having a partition read write), and thus highly improving the reliability of the installed device...

[cherylquirion]

@goranche would love to see the scripts - thanks! Sounds like we have a similar deployment (we currently hundreds of devices, and by the end of the year - thousands). Look forward to seeing those updates.

[goranche]

@cherylquirion sure... you can find the main part here...

if you need to add a third partition, that stays read write (this is my case), then I recommend using an installer built from the v1.1.x branch, copying the scripts/etc/init.d/rcS (from the repository) file to config/installer/rcS on the SD card (this will cause the installer to run this file instead of the original on in the cpio archive), and applying this patch... (this will probably not become part of the installer, BTW)
this will allow you to add a post-fdisk.txt file to the SD card, where you can manipulate the partition table during the install...

(as is, the installer doesn't allow for a hook early enough in the process, that would allow altering the partition table without the need for a reboot of the kernel, to read the new partition table... that's why I added that patch)

you only need to make sure you set bootsize and rootsize in installer-config.txt to some values that leave some space on the SD card... the content of post-fdisk.txt I'm using (this could be made more robust, but it works for me 😊 ):

# Create additional partition
fdisk "${bootdev}" &>/dev/null <<EOF
n
p
3


w
EOF

the resulting system will boot with both /boot and /root read-only, which means that a sudden power loss will not result in a corrupt filesystem, so you can be relatively certain, that it will boot up again...
if needed, one could add a boot-up script that checks if the read-write partition is ok (do a fsck), and if needed, recreate the partition...

that's a quick overview... let me know if you need anything more (my current post-install.txt is around 450 lines long, and includes about 30 additional files... much functionality for headless deploys was added 😎 )

[goranche]

one fortunate side effect (of the read-only system) is that persistent naming doesn't work, as udev can't write out the calculated names 😈