Use rename of directories instead of symbolic links in boot partition.

[valentindavid]

This implements @AdrianVovk idea from #1719 (comment) to solve issue #1719.

This uses renameat2 to do atomic swap of the loader directory in the
boot partition. It fallsback to non-atomic rename. This stays atomic
on filesystems supporting links but also provide a non-atomic behavior
when filesystem does not provide any atomic alternative.

This is working with SystemD boot on EFI using boot loader
specifications.

There is still the issue of losing /loader/loader.conf with SystemD
boot. Maybe we should think about copying other files from previous loader directories.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: valentindavid
To complete the pull request process, please assign cgwalters
You can assign the PR to them by writing /assign @cgwalters in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

Hi @valentindavid. Thanks for your PR.

I'm waiting for a ostreedev member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rh-atomic-bot]

Can one of the admins verify this patch?
I understand the following commands:

• bot, add author to whitelist
• bot, test pull request
• bot, test pull request once

[AdrianVovk]

The challenge with this is that the ESP is vfat, and therefore rename will always be non-atomic. It's still better than what I'm doing now (manually copying files into /efi)

#1951
^ here's a proposed solution that I think could solve this same problem without needing the renameat2 call.

[rh-atomic-bot]

☔ The latest upstream changes (presumably #1767) made this pull request unmergeable. Please resolve the merge conflicts.

[bam80]

Is this PR still proposed or abandoned?

[valentindavid]

Is this PR still proposed or abandoned?

I can rebase and take care of it a bit. But since there was no interaction from maintainers, then I am not sure whether they are considering it.

[bam80]

@valentindavid have you been able to pay some attention to this issue again? The changes are highly anticipated here, in one form or another. Thanks.

[jjardon]

@mwleeds @rfairley any chance to take a look to this? We currently need this to make GNOME images work, see https://gitlab.gnome.org/GNOME/gnome-build-meta/-/commit/79fb62e0d243a21ab58dc1dda439c23db5d474ab

[jjardon]

@cgwalters Hey! Any chance to take a look to this?

[bam80]

Friendly ping.

[AdrianVovk]

@valentindavid This PR conflicts with upstream again. GNOME OS uses an ancient version of OSTree so they have yet to run into this issue, but on my OS I just updated to the latest version and I'm about to be downgrading again because this patch is broken.

@cgwalters Is there any chance for this patch to be reviewed or considered at all? Without this patch, I have to follow every single ostree admin deploy anywhere in the process of building, generating ISOs, installing the OS, and updating the OS with a bunch of ugly commands that mess with the contents of the ESP. With this patch, it just works. OSTree operates on BLS files, but has yet to support the bootloader that was actually built for the standard 🤷

[damianatorrpm]

updated.txt
Rebased patch file.

[kowalski7cc]

Any news on this?

[dbnicholson]

I'm still not totally sure what the best approach here is, but I think the fundamental change from replacing loader to exchanging loader is likely to have subtle effects that need more detailed analysis.

Since the replacing process needs to be supported in case RENAME_EXCHANGE is not available or there's no existing loader, I think the best approach would be to make it always work as if it's replacing. An approach I think would work is:

1. Use the existing symlink replacement, which involves creating a temporary loader.tmp symlink and the renaming it to loader. This is the simple tried and true approach.
2. If symlinking fails because of EPERM, then recursively copy the new loader directory to loader.tmp and then exchange it with loader using RENAME_EXCHANGE + the renaming fallbacks you already coded. When exchanging has occurred, recursively delete the exchanged loader.tmp directory.

[martinezjavier]

Since the replacing process needs to be supported in case RENAME_EXCHANGE is not available

@dbnicholson FYI, I worked on a patch series to add RENAME_EXCHANGE support to vfat, latest revision is: https://lkml.org/lkml/2022/6/1/1049

[dbnicholson]

Since the replacing process needs to be supported in case RENAME_EXCHANGE is not available

@dbnicholson FYI, I worked on a patch series to add RENAME_EXCHANGE support to vfat, latest revision is: https://lkml.org/lkml/2022/6/1/1049

That's great. Of course, you'd still need to support the fallback since it will be some time until that's commonly available (presuming it gets merged).

I've come around a bit on this idea in the sense that I think it's reasonable to assume RENAME_EXCHANGE as the primary mechanism since there will always be filesystems like vfat that can't do symlinks but presumably they can all gain support for RENAME_EXCHANGE. However, I still feel like any major changes in this area require exhaustive testing.

[ricardosalveti]

I'm currently playing with ricardosalveti@731b73f, which is based on the original patch from this PR, which keeps the current behavior for new systems and systems that are already deployed, but also supports loader as a directory (relying on RENAME_EXCHANGE).

That way I can create new deployments (using systemd-boot) with the extra kernel patches without breaking my current deployments that are ok with links (using grub / u-boot).

Still validating with my yocto integration, can post the final set after I'm done testing it.

[martinezjavier]

FYI the patch-set got merged by a subsystem maintainer and are heading towards the Linux v6.0 release.

[GrabbenD]

Is GNOME OS's patch still the best workaround until this is finished?
https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/45.beta/files/ostree/no-boot-symlink.patch

Edit: looks like it doesn't work with 2023.6 update

[GrabbenD]

I ended up working around this issue by making / a ext4 partition and then I mounted ESP on /boot/efi which makes /boot ext4 thus allowing symbolic links

Here's the commands if anyone else is stuck on Preparing final bootloader swap: symlinkat: Operation not permitted error:
M1cha/archlinux-ostree#1 (comment)

[bam80]

update?

[dbnicholson]

I've thought about this several times as I would really like to have this in Endless to support our systemd-boot systems. What always makes me anxious is trying to figure out how to handle compatibility with the vast majority of our systems that have symlink based deployments. There are 2 main issues I'm concerned with:

• The semantics of the deployment are different if loader is a symlink to a loader.N directory versus loader itself being a directory. What do the swapped loader.N directories mean in pure directory mode? I think the answer is that they don't really match the semantics of the symlink deployment. It's essentially a different algorithm even though it's seemingly only a small change from the symlink approach.
• What do you do with systems that already have symlink based deployments? Potentially you could write a migration to the directory scheme, but that means you can't roll back to a system with a libostree that doesn't understand the directory scheme.

So, to me this requires a couple additional pieces of implementation and policy.

• The directory deployment scheme is treated separately from the symlink deployment scheme. Your system can be in one scheme or the other, but the semantics are different and shouldn't be mixed.
• If your system is already using a symlink based deployment, it should stay that way. You can opt in to a migration to the directory scheme, but this means you may potentially lose roll back targets. At Endless we'd do that at some major version check point and probably have to delete some old deployments to ensure users didn't try to roll back to a system where they'd be screwed. Or maybe we'd never migrate anyone because the downsides are too great.
• If you're on a new deployment (i.e., no existing /boot/loader or /boot/ostree), then the directory scheme is preferred.

[bam80]

Status?