Add a new ostree admin instutil subcommand to copy EFI binaries to the ESP

[martinezjavier]

Currently the only file in the EFI System Partition (ESP) that is updated is grub.cfg. Every other file in the ESP is the one that was copied by Anaconda on installation. This means that shim, grub, firmware update, etc are never upgraded so users won't get any bugfixes or security updates for the bootloaders files, i.e:

$ rpm -qa | grep grub2-efi-x64
grub2-efi-x64-2.02-26.fc28.x86_64

$ md5sum /boot/efi/EFI/fedora/grubx64.efi 
be1b3601da78bab1483690276c6e47f1  /boot/efi/EFI/fedora/grubx64.efi

$ atomic host upgrade && systemctl reboot

$ rpm -qa | grep grub2-efi-x64
grub2-efi-x64-2.02-34.fc28.x86_64

$ md5sum /boot/efi/EFI/fedora/grubx64.efi 
be1b3601da78bab1483690276c6e47f1  /boot/efi/EFI/fedora/grubx64.efi

ESP is mounted on /boot/efi and the packages copy the EFI binaries to /boot/efi/EFI/fedora/. The rpm-ostree unpacker translates the /boot path to /usr/lib/ostree-boot. So the EFI binaries are in /usr/lib/ostree-boot/efi/EFI/fedora/. They just are never copied to the ESP as a part of a deploy.

This is because the ESP is a vfat partition and so it can't be part of the ostree transactional deployment. But still users should be have the option to update the files in the ESP outside of the normal ostree deployments in case they want to update their bootloaders.

For example there could be a ostree admin instutil esp-upgrade command that copies the EFI binaries from /usr/lib/ostree-boot/efi/EFI/fedora/ to /boot/efi/EFI/fedora/.

[cmurf]

How about a generic ESP updating service that does not depend on a. always keeping the ESP mounted; b. mdraid (because it's explicitly not recommended); c. assuming only one ESP; d. assuming traditional or ostree installation; e. users knowing a multitude of esoteric information, and documenters having to document and translate all of it.

A more reliable design would be to update the binaries in a location common to traditional and ostree based installs own. Maybe this is in /usr, or maybe /boot/efi (no longer the ESP). And a trigger for a sync service that syncs to all ESPs. The service would mount any ESP (by partition type GUID) e.g. /run/mount/ and then use cp -r or rsync -r from our static location to the path we own, EFI$DISTRO, then unmount them. Maybe it'd be done after switchroot, before all other services are running.

[martinezjavier]

@cmurf sounds good to me. I don't really have a strong opinion on how this should be implemented.

I just filed this because I think we should provide a way for users to update the ESP, specially since I guess may users are not aware that the bootloaders are not updated as a part of a ostree admin upgrade operation (for example I only realized this from the discussion in #717).

[cmurf]

Well if it's simple to do, it's certainly better than not being able to update the binaries. Someone taking on a more wholistic design and implementation may not happen anytime soon.

[martinezjavier]

@cmurf yes, having it as a ostree admin instutil subcommand should be pretty easy to implement I believe and I can take a look to that once we land all the BLS changes.

The sync service that you propose sounds like a much generic solution and I'll be happy if someone implements it, but I personally don't think that will have the time for that.

[AdrianVovk]

Let's revive this discussion... @cgwalters @martinezjavier

This is my previous comment on this:

---

I've been able to mess about with my distro and get some insight about the relationship between /boot, the ESP, and systemd

systemd, by default, really wants the ESP to be mounted to either /boot directly or /efi. If /efi exists, systemd will mount the ESP there. Otherwise, it will try to mount it to /boot. This is the behaviour of the systemd-gpt-auto-generator, which my distro will use to find and mount the rootfs. bootctl checks those two mounts points for an ESP also. It's safe to assume that any systemd-related tool will probably assume the EFI exists in either of these two directories (short of looking at /etc/fstab, which I don't intend to use on my distro)

I've been able to temporarily get OSTree working like so: I have a boot.mount unit which bind-mounts /sysroot/boot to /boot. OSTree can then operate on this directory normally. I created an /efi mount point (and systemd automagically mounts the ESP there) that I can copy files to as necessary. This, of course, is not an atomic layout. If something goes wrong while my distro's scripts are copying from /boot to /efi, I'm screwed. The system would no longer boot.

OSTree depends on symlinks in /boot, so it cannot operate directly on the ESP. If this dependency can be broken, however, it could be possible to work with the ESP in /boot. I think this can be done with the renameat2 syscall to atomically swap two folders, instead of using a symlink and rename. More info and initial patch. The new flow could go something like this: empty out /boot/loader.inactive/, generate the ostree config files, copy in non-ostree config files from /boot/loader, renameat2 to swap the two folders. This would give an atomic swap without relying on symlinks. If I'm not missing something, this would allow OSTree to work directly on the ESP. Steps like setting permissions on the files can be skipped or can error out silently, since vfat doesn't support permissions anyway AFAIK

If, for compatability, you need to keep the old behavior, you can translate all of this new behavior to the /efi directory. This new atomic swap would happen in /efi, and the old behavior would happen in /boot.

---

I've already borked the system by accidentally typing in ostree admin upgrade instead of running my script that also copies over from /boot to /efi. I'm sure that someone at some point will run that command and destroy their system's ability to boot. Making the user run a separate command to copy things to the EFI will lead to this very problem. OSTree should, on EFI and BLS bootloaders (like sd-boot), work with the EFI directly. I don't see a downside

What are your thoughts?

[wjt]

I think this can be done with the renameat2 syscall to atomically swap two folders, instead of using a symlink and rename.

I can't find my test program right now, but I'm pretty sure I tested and renameat2(..., RENAME_EXCHANGE) is not supported on vfat.

[AdrianVovk]

@wjt I tested it and you are correct, trying to do the exchange on the ESP returns EINVAL

OSTree can rely on creating/deleting files in the correct order instead

For example:

1. /efi/loader contains a.conf and b.conf
2. Create c+3.conf.staged
3. Write the contents into the file
4. Rename the file to c+3.conf
5. Delete a.conf

[bam80]

Any update? (didn't read all the conversation)

[martinezjavier]

@cgwalters I think this issue could be closed now that https://github.com/coreos/bootupd exists?

[cgwalters]

Yep, thanks!

To clarify, ostree doesn't depend on bootupd, and while bootupd currently only works with (rpm-)ostree based systems the idea is to support other systems too.

So using bootupd is one solution; others may want to just take a similar approach with other software.