Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

linux-image-5.10.0-8-amd64 upgrade fails on x86_64 Bullseye image #4788

Closed
dynobot opened this issue Sep 29, 2021 · 20 comments
Closed

linux-image-5.10.0-8-amd64 upgrade fails on x86_64 Bullseye image #4788

dynobot opened this issue Sep 29, 2021 · 20 comments
Milestone

Comments

@dynobot
Copy link

dynobot commented Sep 29, 2021

ADMIN EDIT

Solution

mkdir -p /etc/kernel/preinst.d
cat << '_EOF_' > /etc/kernel/preinst.d/dietpi
#!/bin/sh -e
# Remove old kernel files if existing: https://github.com/MichaIng/DietPi/issues/4788
{
# Fail if the package name was not passed, which is done when being invoked by dpkg
if [ -z "$DPKG_MAINTSCRIPT_PACKAGE" ]
then
        echo 'DPKG_MAINTSCRIPT_PACKAGE was not set, this script must be invoked by dpkg.'
        exit 1
fi

# Loop through files in /boot, shipped by the package, and remove them, if existing
for file in $(dpkg -L "$DPKG_MAINTSCRIPT_PACKAGE" | grep '^/boot/')
do
        [ ! -f "$file" ] || rm "$file"
done
}
_EOF_
chmod +x /etc/kernel/preinst.d/dietpi

Details:

  • Date | Wed Sep 29 14:12:03 BST 2021
  • DietPi version | v7.4.2 (MichaIng/master)
  • Image creator | DietPi Core Team
  • Pre-image | from scratch
  • Hardware | Native PC (x86_64) (ID=21)
  • Kernel version | Linux DietPi 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU/Linux
  • Distro | bullseye (ID=6)
  • Command | apt-get -qq upgrade
  • Exit code | 100
  • Software title | DietPi-Update

Steps to reproduce:

  1. ...Install dietpi to ssd, run first boot
  2. ...

Expected behaviour:

  • ...system should upgrade automatically

Actual behaviour:

  • ...system failed to upgrade

Extra details:

  • ...
root@DietPi:~# apt-get -qq upgrade
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
root@DietPi:~# apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  linux-image-5.10.0-8-amd64
Suggested packages:
  linux-doc-5.10 debian-kernel-handbook
Recommended packages:
  apparmor
The following packages will be upgraded:
  linux-image-5.10.0-8-amd64
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
Need to get 53.6 MB of archives.
After this operation, 2048 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://deb.debian.org/debian-security bullseye-security/main amd64 linux-image-5.10.0-8-amd64 amd64 5.10.46-5 [53.6 MB]
Fetched 53.6 MB in 8s (6994 kB/s)                                                                                                                                                       
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 19949 files and directories currently installed.)
Preparing to unpack .../linux-image-5.10.0-8-amd64_5.10.46-5_amd64.deb ...
Unpacking linux-image-5.10.0-8-amd64 (5.10.46-5) over (5.10.46-4) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-5.10.0-8-amd64_5.10.46-5_amd64.deb (--unpack):
 unable to make backup link of './boot/System.map-5.10.0-8-amd64' before installing new version: Operation not permitted
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/linux-image-5.10.0-8-amd64_5.10.46-5_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@DietPi:~# 

Additional logs:

E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

@MichaIng
Copy link
Owner

Oh no, it is trying to create a link or symlink on upgrade it seems, which is not possible as it is a FAT partition. I haven't seen this before, so checking the latest changelog.

@dynobot
Copy link
Author

dynobot commented Sep 29, 2021

Hi Michalng

If I can do anything on my end to help let me know.

Thank you
Ed

@MichaIng
Copy link
Owner

MichaIng commented Sep 29, 2021

I cannot find the part of the code which does this. Can you try the following:

/var/lib/dpkg/info/linux-image-5.10.0-8-amd64.preinst configure
/var/lib/dpkg/info/linux-image-5.10.0-8-amd64.postinst configure

It cannot be part of dpkg itself, else on RPi and Odroids it would fail the same way, which ship with a /boot FAT partition since ever.

@dynobot
Copy link
Author

dynobot commented Sep 29, 2021

root@DietPi:~# /var/lib/dpkg/info/linux-image-5.10.0-8-amd64.preinst configure
root@DietPi:~# /var/lib/dpkg/info/linux-image-5.10.0-8-amd64.postinst configure
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.10.0-8-amd64
/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.10.0-8-amd64
Found initrd image: /boot/initrd.img-5.10.0-8-amd64
done
root@DietPi:~# 

Okay would you like me to reboot or anything now?

@dynobot
Copy link
Author

dynobot commented Sep 29, 2021

root@DietPi:~# apt-get -qq upgrade
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
root@DietPi:~# apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  linux-image-5.10.0-8-amd64
Suggested packages:
  linux-doc-5.10 debian-kernel-handbook
Recommended packages:
  apparmor
The following packages will be upgraded:
  linux-image-5.10.0-8-amd64
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
Need to get 0 B/53.6 MB of archives.
After this operation, 2048 B of additional disk space will be used.
Do you want to continue? [Y/n] y
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 19949 files and directories currently installed.)
Preparing to unpack .../linux-image-5.10.0-8-amd64_5.10.46-5_amd64.deb ...
Unpacking linux-image-5.10.0-8-amd64 (5.10.46-5) over (5.10.46-4) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-5.10.0-8-amd64_5.10.46-5_amd64.deb (--unpack):
 unable to make backup link of './boot/System.map-5.10.0-8-amd64' before installing new version: Operation not permitted
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/linux-image-5.10.0-8-amd64_5.10.46-5_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@DietPi:~# 

@MichaIng
Copy link
Owner

MichaIng commented Sep 29, 2021

No please do not reboot at this stage. Okay it seems to be the mv command which causes this, probably in combination with /proc/sys/fs/protected_hardlinks set, but this shouldn't happen as FAT does not support hard links, which mv should know (and also it doesn't fail when trying it manually with other files).

It is a regular physical x86_64 machine?

Can you try the following:

echo 0 > /proc/sys/fs/protected_hardlinks
apt upgrade

If it does not work, revert and work around the issue by removing this file:

echo 1 > /proc/sys/fs/protected_hardlinks
rm /boot/System.map-5.10.0-8-amd64
apt upgrade

Likely it is required for two other files (kernel config and kernel image) in /boot. I need to run some testing here.

@MichaIng MichaIng changed the title First Run amd64 upgrade/update issue linux-image-5.10.0-8-amd64 upgrade fails on x86_64 Bullseye image Sep 29, 2021
@dynobot
Copy link
Author

dynobot commented Sep 29, 2021

Yes its a regular physical x86_64 machine

root@DietPi:~# echo 0 > /proc/sys/fs/protected_hardlinks
root@DietPi:~# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 linux-image-amd64 : Depends: linux-image-5.10.0-8-amd64 (= 5.10.46-5) but 5.10.46-4 is installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
root@DietPi:~# 
root@DietPi:~# echo 1 > /proc/sys/fs/protected_hardlinks
root@DietPi:~# rm /boot/System.map-5.10.0-8-amd64
root@DietPi:~# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 linux-image-amd64 : Depends: linux-image-5.10.0-8-amd64 (= 5.10.46-5) but 5.10.46-4 is installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
root@DietPi:~# 

@MichaIng
Copy link
Owner

apt -f install
apt upgrade

Due to the unconfigured new package APT navigated itself into a broken dependency situation.

@dynobot
Copy link
Author

dynobot commented Sep 29, 2021

root@DietPi: ~ # apt -f install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  linux-image-5.10.0-8-amd64
Suggested packages:
  linux-doc-5.10 debian-kernel-handbook
Recommended packages:
  apparmor
The following packages will be upgraded:
  linux-image-5.10.0-8-amd64
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
Need to get 0 B/53.6 MB of archives.
After this operation, 2048 B of additional disk space will be used.
Do you want to continue? [Y/n] y
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 19949 files and directories currently installed.)
Preparing to unpack .../linux-image-5.10.0-8-amd64_5.10.46-5_amd64.deb ...
Unpacking linux-image-5.10.0-8-amd64 (5.10.46-5) over (5.10.46-4) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-5.10.0-8-amd64_5.10.46-5_amd64.deb (--unpack):
 unable to make backup link of './boot/config-5.10.0-8-amd64' before installing new version: Operation not permitted
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/linux-image-5.10.0-8-amd64_5.10.46-5_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@DietPi:~ # apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 linux-image-amd64 : Depends: linux-image-5.10.0-8-amd64 (= 5.10.46-5) but 5.10.46-4 is installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
root@DietPi:~# 

@MichaIng
Copy link
Owner

Yes as said, there are the other two files which fail the same way and hence need to be removed as well:

rm -f /boot/{config,System.map,vmlinuz}-5.10.0-8-amd64
apt -f install
apt upgrade

@dynobot
Copy link
Author

dynobot commented Sep 29, 2021

RESOLVED

root@DietPi:~# rm -f /boot/{config,System.map,vmlinuz}-5.10.0-8-amd64
root@DietPi:~# apt -f install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  linux-image-5.10.0-8-amd64
Suggested packages:
  linux-doc-5.10 debian-kernel-handbook
Recommended packages:
  apparmor
The following packages will be upgraded:
  linux-image-5.10.0-8-amd64
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
Need to get 0 B/53.6 MB of archives.
After this operation, 2048 B of additional disk space will be used.
Do you want to continue? [Y/n] y
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 19949 files and directories currently installed.)
Preparing to unpack .../linux-image-5.10.0-8-amd64_5.10.46-5_amd64.deb ...
Unpacking linux-image-5.10.0-8-amd64 (5.10.46-5) over (5.10.46-4) ...
Setting up linux-image-5.10.0-8-amd64 (5.10.46-5) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.10.0-8-amd64
/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.10.0-8-amd64
Found initrd image: /boot/initrd.img-5.10.0-8-amd64
done
Setting up linux-image-amd64 (5.10.46-5) ...
Setting up tzdata (2021a-1+deb11u1) ...

Current default time zone: 'Europe/London'
Local time is now:      Wed Sep 29 15:36:27 BST 2021.
Universal Time is now:  Wed Sep 29 14:36:27 UTC 2021.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

Setting up openssl (1.1.1k-1+deb11u1) ...
Processing triggers for libc-bin (2.31-13) ...
root@DietPi:~# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

@Joulinar Joulinar added the Workaround available 🆗 Workaround is available/has been implemented, but a definite solution should be found when possible. label Sep 29, 2021
@dynobot
Copy link
Author

dynobot commented Sep 29, 2021

Thank you

@MichaIng
Copy link
Owner

Okay great so far. So it is dpkg which internally calls mv to install new files and seems to try to create a backup if the file exists already. For unknown reasons it believes the existing file is a link, which is impossible as it's on FAT, and then tries to create a link as backup accordingly, which of course then fails as well as FAT does not support any kind of links.

I'll try to replicate this behaviour, as it strangely is not an issue on any other image with a /boot FAT partition.

One last thing before you reboot:

dmesg -l emerg,alert,crit,err,warn

@dynobot
Copy link
Author

dynobot commented Sep 29, 2021

root@DietPi:~ # dmesg -l emerg,alert,crit,err,warn
[    0.022007] [Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0x22 (or later)
[    0.186262] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[    0.186262]  #5 #6 #7
[    0.213057] pmd_set_huge: Cannot satisfy [mem 0xf8000000-0xf8200000] with a huge-page mapping due to MTRR override.
[    0.878512] ACPI Warning: SystemIO range 0x0000000000001828-0x000000000000182F conflicts with OpRegion 0x0000000000001800-0x000000000000187F (\PMIO) (20200925/utaddress-204)
[    0.878520] ACPI Warning: SystemIO range 0x0000000000001C40-0x0000000000001C4F conflicts with OpRegion 0x0000000000001C00-0x0000000000001FFF (\GPR2) (20200925/utaddress-204)
[    0.878523] ACPI Warning: SystemIO range 0x0000000000001C40-0x0000000000001C4F conflicts with OpRegion 0x0000000000001C00-0x0000000000001FFF (\GPR) (20200925/utaddress-204)
[    0.878528] ACPI Warning: SystemIO range 0x0000000000001C30-0x0000000000001C3F conflicts with OpRegion 0x0000000000001C00-0x0000000000001FFF (\GPR2) (20200925/utaddress-204)
[    0.878531] ACPI Warning: SystemIO range 0x0000000000001C30-0x0000000000001C3F conflicts with OpRegion 0x0000000000001C00-0x0000000000001C3F (\GPRL) (20200925/utaddress-204)
[    0.878534] ACPI Warning: SystemIO range 0x0000000000001C30-0x0000000000001C3F conflicts with OpRegion 0x0000000000001C00-0x0000000000001FFF (\GPR) (20200925/utaddress-204)
[    0.878538] ACPI Warning: SystemIO range 0x0000000000001C00-0x0000000000001C2F conflicts with OpRegion 0x0000000000001C00-0x0000000000001FFF (\GPR2) (20200925/utaddress-204)
[    0.878541] ACPI Warning: SystemIO range 0x0000000000001C00-0x0000000000001C2F conflicts with OpRegion 0x0000000000001C00-0x0000000000001C3F (\GPRL) (20200925/utaddress-204)
[    0.878544] ACPI Warning: SystemIO range 0x0000000000001C00-0x0000000000001C2F conflicts with OpRegion 0x0000000000001C00-0x0000000000001FFF (\GPR) (20200925/utaddress-204)
[    0.878548] lpc_ich: Resource conflict(s) found affecting gpio_ich
[    2.263497] at24 0-0050: supply vcc not found, using dummy regulator

@MichaIng
Copy link
Owner

The ACPI warnings seem to be common on some Intel boxes:

Not sure about the initial firmware bug, as intel-microcode is upgraded together with all other APT packages. However, I do not believe it is related.

@dynobot
Copy link
Author

dynobot commented Sep 29, 2021

Okay great....everything seems to be running smoothly.

Thanks again...if you want me to do more testing etc. just let me know.

@MichaIng
Copy link
Owner

MichaIng commented Sep 29, 2021

Verified bug. I can replicate it with a little dummy package which contains an empty file in /boot. Installing it work, reinstalling it fails as the file exists already. Trying to replicate it on a Buster system.

echo 0 > /proc/sys/fs/protected_hardlinks has not effect on this.


Tested first on a Raspberry Pi image where the official one ships with FAT /boot partition where the bootloader ships packages below /boot. Hmm reinstalling it works fine, probably this is the reason for the rpikernelhack diversions back and forth 🤔.


Same on Buster. It seems to be expected. Luckily for the kernel package this can be solved quite easily.

@MichaIng MichaIng added this to the v7.7 milestone Sep 29, 2021
@MichaIng
Copy link
Owner

MichaIng commented Sep 29, 2021

This kernel install hook prevents the error in case of future kernel upgrades (without kernel version increment):

mkdir -p /etc/kernel/preinst.d
cat << '_EOF_' > /etc/kernel/preinst.d/dietpi
#!/bin/sh -e
# Remove old kernel files if existing: https://github.com/MichaIng/DietPi/issues/4788
{
# Fail if the package name was not passed, which is done when being invoked by dpkg
if [ -z "$DPKG_MAINTSCRIPT_PACKAGE" ]
then
        echo 'DPKG_MAINTSCRIPT_PACKAGE was not set, this script must be invoked by dpkg.'
        exit 1
fi

# Loop through files in /boot, shipped by the package, and remove them, if existing
for file in $(dpkg -L "$DPKG_MAINTSCRIPT_PACKAGE" | grep '^/boot/')
do
        [ ! -f "$file" ] || rm "$file"
done
}
_EOF_
chmod +x /etc/kernel/preinst.d/dietpi

I will install it into our images. However, it is somehow a bumper for the idea to ship images with a Windows/macOS accessible boot partition, as it feels like an unclean workaround. The behaviour seems to be expected by dpkg and indeed Odroids and RPi kernel packages work around it as well, but it still feels like a bug to me, respectively I don't understand why it is better to have dpkg failing compared to letting it handle FAT partitions gracefully. If a backup is wanted, then there are other ways to achieve it when the filesystem does not support links.

@MichaIng
Copy link
Owner

MichaIng commented Sep 29, 2021

Added to PREP script: ee8ad81
Pre-patch for v7.7: 1e8143f

And I decided to update pre-patches in master branch to solve it for new installs and when users update DietPi from v7.4 or v7.5. This is too large for a live patch and next release is still too far away. So that way we minimise inconvenience and only need to catch users which manually call apt upgrade to apply the patch manually: 9653688

@MichaIng MichaIng added Solution available 🥂 Definite solution has been done and removed Workaround available 🆗 Workaround is available/has been implemented, but a definite solution should be found when possible. Image update 💾 Priority 🔆 labels Sep 29, 2021
@MichaIng
Copy link
Owner

I just asked about this matter on the Debian bug tracker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995332

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants