Kernel upgrades on existing DietPi installations went wrong

[maartenlangeveld]

Just to inform I had to re-install dietpi from scratch on both my rpi-zero-w (buster) and nanopi-neo2 (buster) machines after on those machines new kernel versions were installed. Both machines did not boot thereafter.

When freshly installing dietpi on both machines the kernel upgrade went fine.

Lots of work so maybe there's a fix that will prevent other users from havong to rebuild their setups from start when doing apt update and apt upgrade.

UPDATE:
To be more precise: at least there was no networking connection anymore after reboot. Since my NanoPi NEO2 is headless by nature and I had configured my RPi-Zero-W as a headless system, the only way out was to do fresh installs as I was not able to investigate if only networking failed or there were more issues after kernel upgrade.

[MichaIng]

Many thanks for your report.

Hmm, I guess you don't have logs from the kernel upgrade anymore, right? Carefully check the output in case any error occurs during the process. Especially on RPi, the kernel upgrades should be very stable, I never had issues like that. On Armbian devices indeed I faced many reports about specific kernel versions introducing certain issues. Armbian seems to be not too careful or testing enough when pushing new kernel versions, sadly.

Since SDcard corruption is the by far most source for boot issues, after doing a kernel upgrade, check if any file system errors occurred, e.g.: dmesg -l err,crit,alert,emerg
Since on RPi (and Odroids), the boot partition is FAT, when it fails to boot you can plug it into a Windows PC and do an error scan there + check that all files exist as expected. When having errors on the ext4 partition, I was usually able to recover by mounting the SDcard into a DietPi VM and run fsck there, or of course any other Linux system.

Btw, it is also possible to make journalctl system logs boot persistent and review them from an external Linux/systemd system, and, a serial console is as well nice to check early boot output on headless systems and even earliest bootloader output when no video output is started yet. Just to mention a few ways to debug such issues 😉.

[Joulinar]

just a side note. I did a couple of kernel updated the last days as I rolled back my RPi3b+ (for testing) quite often. All of these kernel updated went without issue and system went fine after rebooting.

[maartenlangeveld]

@MichaIng
Thanks for your advice. Maybe I will take precautious measures in the future.

Today, there was another kernel update for armbiam-firmware on my NanoPi NEO2.
Everything went fine, only one error during install ("Failed to enable unit: Unit file /etc/systemd/system/armbian-ramlog.service is masked."). See output below.

The machine did boot however this time :-)

root@DietPi-DNS:~# G_AGUP
[ INFO ] APT update, please wait...
Hit:2 https://deb.debian.org/debian buster InRelease
Get:3 https://deb.debian.org/debian buster-updates InRelease [49.3 kB]
Get:4 https://deb.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:5 https://deb.debian.org/debian buster-backports InRelease [46.7 kB]
Get:1 https://apt.armbian.com buster InRelease [18.3 kB]
Get:6 https://deb.debian.org/debian-security buster/updates/main arm64 Packages [200 kB]
Get:7 https://deb.debian.org/debian-security buster/updates/main armhf Packages [199 kB]
Get:8 https://apt.armbian.com buster/main armhf Packages [237 kB]
Get:9 https://apt.armbian.com buster/main arm64 Packages [245 kB]
Fetched 1060 kB in 4s (295 kB/s)
Reading package lists...
[  OK  ] APT update
root@DietPi-DNS:~# G_AGUG
[ INFO ] APT upgrade, please wait...
(Reading database ... 29223 files and directories currently installed.)
Preparing to unpack .../armbian-firmware_20.05.2_all.deb ...
Unpacking armbian-firmware (20.05.2) over (20.05.1) ...
Preparing to unpack .../linux-buster-root-current-nanopineo2_20.05.2_arm64.deb ...
Unpacking linux-buster-root-current-nanopineo2 (20.05.2) over (20.05.1) ...
Preparing to unpack .../linux-dtb-current-sunxi64_20.05.2_arm64.deb ...
Unpacking linux-dtb-current-sunxi64 (20.05.2) over (20.05.1) ...
Preparing to unpack .../linux-image-current-sunxi64_20.05.2_arm64.deb ...
Unpacking linux-image-current-sunxi64 (20.05.2) over (20.05.1) ...
Preparing to unpack .../linux-u-boot-nanopineo2-current_20.05.2_arm64.deb ...
Unpacking linux-u-boot-nanopineo2-current (20.05.2) over (20.05.1) ...
Setting up linux-image-current-sunxi64 (20.05.2) ...
update-initramfs: Generating /boot/initrd.img-5.4.43-sunxi64
update-initramfs: Converting to u-boot format
Setting up linux-dtb-current-sunxi64 (20.05.2) ...
Setting up linux-u-boot-nanopineo2-current (20.05.2) ...
Setting up linux-buster-root-current-nanopineo2 (20.05.2) ...
Failed to enable unit: Unit file /etc/systemd/system/armbian-ramlog.service is masked.
Setting up armbian-firmware (20.05.2) ...
Processing triggers for initramfs-tools (0.133+deb10u1) ...
update-initramfs: Generating /boot/initrd.img-5.4.43-sunxi64
update-initramfs: Converting to u-boot format

[MichaIng]

G_AGUG

Ah I was not 100% sure but now it's clear that Armbian kernel packages are not provided as meta packages anymore, like it is for the (generic) Debian kernel packages, the ones of Meverics Odroid repo and Radxa e.g.:

• linux-image-amd64 (meta package) depends on linux-image-4.19.0-8-amd64 (actual versioned kernel package)

A regular apt upgrade/G_AGUG only upgrades packages which do not require either the install of new or the removal of conflicting packages due to changed dependencies. So when the kernel has been installed via meta package, regular apt upgrades does not upgrade it, since it would need to pull a new actual kernel package, e.g. linux-image-4.19.0-9-amd64. For this apt full-upgrade/G_AGDUG is required, which we never do automated (only on very first boot) but via dietpi-config > Advanced Options > Firmware upgrade instead.

You can see it as benefit since kernel/dtb/bootloader is kept up-to-date on a more regular basis, or you can see it as regression since you loose certain control when those critical upgrades are done as every dietpi-update and dietpi-software install calls G_AGUG. RPi never shipped their kernel as meta package, but I never had issues with it so don't see a problem, but in case of Armbian I see things differently, especially on rare used devices or such which are EOL, like the Odroid C1, NanoPi M1, NEO Plus2 and some others. A new kernel did by times break boot indeed, and not because of SDcard corruption or similar 😉.

Finally I have to think about this. Probably we have to set those packages on hold (apt-mark hold <pkg>) and take care that a regular G_AGUG does not override this, while G_AGDUG does.

Failed to enable unit: Unit file /etc/systemd/system/armbian-ramlog.service is masked.

Yes that is expected. Armbian RAM log implementation conflicts with ours, and it creates a zRam mount in /var/log which caused file corruptions at least with the parallel (or nested) tmpfs mount that DietPi-RAMlog places there. Blocking those tools from being installed while keeping the files from linux-buster-root-current-* installed which are required for kernel updates, is kinda ugly workaround currently. I have already the plan to ban this package completely and create an own solution for the kernel upgrade, from what I remember it is actually only a postinst script to convert the Linux image file to uImage u-boot format.

[maartenlangeveld]

@MichaIng
Thanks a lot. Since new kernel is installed :

Failed to enable unit: Unit file /etc/systemd/system/armbian-ramlog.service is masked.

Do I have to worry about it? Do I need to take additional steps to reactive DeitPi-RAMlog ? Or can I leave it as it is. I am running Pi-hole on my NanoPi NEO2.

Thanks again, Maarten

[MichaIng]

armbian-ramlog.service is masked as expected, hence is not started (even if it would try, the binary has not been installed), so nothing do to 🙂.

[maartenlangeveld]

Great ! Thx

[StephanStS]

@maartenlangeveld: I updated the kernel on my NEO2 manually to linux-image-dev-sunxi64 which leads to a Linux kernel, version 5.6.15-sunxi64.
uname -a gives Linux Node-RED 5.6.12-sunxi64 #trunk.133 SMP Wed May 13 18:09:20 CEST 2020 aarch64 GNU/Linux (I am only running Node-RED on it).
I had no problems with the manual update to the dev kernel.

Also an apt update/upgrade today did not change anything towards a worse state.

Regarding the network reboot issue at the NEO2: I had similar issues at different NanoPi types (e.g. M3, M4v2, NEO4,...). Sometimes further reboots fixed it until the system comes to a "stable state" where further reboots did not show problems any more. I.e. just power cycle the NanoPi at these network issues after changing the network configuration.
Also at my NanoPi versions an update to the newest DietPi software helped to get it more stable (-> dietpi-update).

[MichaIng]

I mark this as closed. Feel free to reopen if required.