Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make ostree aware of live environments #1921

Open
bgilbert opened this issue Sep 15, 2019 · 7 comments · May be fixed by #1995
Open

Make ostree aware of live environments #1921

bgilbert opened this issue Sep 15, 2019 · 7 comments · May be fixed by #1995

Comments

@bgilbert
Copy link
Contributor

When Fedora CoreOS is booted as a live PXE system, ostree would ideally function in read-only mode, where it refuses to deploy updates (since that isn't a meaningful operation on a live system) but still provides read-only status info.

rpm-ostreed currently refuses to start on such systems because ostree is unhappy:

Error setting up sysroot: Unexpected state: /run/ostree-booted found and in / sysroot but not in a booted deployment

Live PXE systems don't need anything in /boot, since the kernel and initramfs are provided externally. Fedora CoreOS could generate a fake BLS entry if needed, though note that /proc/cmdline doesn't contain an ostree argument so we'd need to deal with that also (#1920).

What's the best way to proceed here?

@cgwalters
Copy link
Member

We could "standardize" /run/ostree-live here?

@cgwalters
Copy link
Member

#1995 is a WIP PR for this.

cgwalters added a commit to cgwalters/ostree that referenced this issue Mar 19, 2020
Support read-only queries, but deny any attempts to make
changes with a clear error.

Now...in the future, we probably do want to support making
"transient" overlays.  See:
coreos/fedora-coreos-tracker#354 (comment)

Closes: ostreedev#1921
cgwalters added a commit to cgwalters/ostree that referenced this issue Mar 21, 2020
Support read-only queries, but deny any attempts to make
changes with a clear error.

Now...in the future, we probably do want to support making
"transient" overlays.  See:
coreos/fedora-coreos-tracker#354 (comment)

Closes: ostreedev#1921
cgwalters added a commit to cgwalters/ostree that referenced this issue Mar 21, 2020
Support read-only queries, but deny any attempts to make
changes with a clear error.

Now...in the future, we probably do want to support making
"transient" overlays.  See:
coreos/fedora-coreos-tracker#354 (comment)

Closes: ostreedev#1921
cgwalters added a commit to cgwalters/ostree that referenced this issue Mar 23, 2020
Support read-only queries, but deny any attempts to make
changes with a clear error.

Now...in the future, we probably do want to support making
"transient" overlays.  See:
coreos/fedora-coreos-tracker#354 (comment)

Closes: ostreedev#1921
cgwalters added a commit to cgwalters/ostree that referenced this issue Mar 24, 2020
Support read-only queries, but deny any attempts to make
changes with a clear error.

Now...in the future, we probably do want to support making
"transient" overlays.  See:
coreos/fedora-coreos-tracker#354 (comment)

Closes: ostreedev#1921
@jlebon
Copy link
Member

jlebon commented May 26, 2020

This oddly works now in a locally built FCOS live ISO and PXE artifacts:

[root@localhost ~]# rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/testing-devel
                   Version: 31.20200526.dev.0 (2020-05-26T19:18:19Z)
                    Commit: f1bbe310d7ec3ac03e1dba2c819f43daf3c7f6c88c000b20892d7d4a25881f2e
              GPGSignature: (unsigned)
[root@localhost ~]# cat /proc/cmdline
BOOT_IMAGE=fedora-coreos-31.20200526.dev.0-live-kernel-x86_64 ip=dhcp rd.neednet=1 initrd=fedora-coreos-31.20200526.dev.0-live-initramfs.x86_64.img console=tty0 console=ttyS0 ignition.config.url=http://192.168.122.1:8000/config-fcos.ign ignition.firstboot ignition.platform.id=metal BOOTIF=01-52-54-00-44-96-c9

@jlebon jlebon changed the title ostree on live PXE systems Make ostree aware of live environments Jul 18, 2024
@jlebon
Copy link
Member

jlebon commented Jul 18, 2024

Repurposing this card now to just generalize it to having ostree be "live system-aware".

jbtrystram added a commit to jbtrystram/fedora-coreos-config that referenced this issue Aug 29, 2024
in the composeFS path, ostree-prepare-root want to
mount /etc/ and /var as writeable, which cannot in the live iso
environnement.
Overriding the kernel command line to disable composeFS in that case.

See ostreedev/ostree#1921
And coreos#3009 (comment)
jlebon pushed a commit to jbtrystram/fedora-coreos-config that referenced this issue Aug 29, 2024
In the composefs path, ostree-prepare-root want to mount /etc/ and /var
as writeable, which cannot in the live iso environnement.

Override the kernel command line to disable composefs in that case.

See ostreedev/ostree#1921
and coreos#3009 (comment)
jlebon pushed a commit to jbtrystram/fedora-coreos-config that referenced this issue Aug 30, 2024
In the composefs path, ostree-prepare-root want to mount /etc/ and /var
as writeable, which cannot in the live iso environnement.

Override the kernel command line to disable composefs in that case.

See ostreedev/ostree#1921
and coreos#3009 (comment)
@cgwalters
Copy link
Member

Moving this comment over coreos/fedora-coreos-config#3009 (comment)

I think this is basically another instance of #1921 (which I've just retitled); so then ostree-prepare-root would detect that this is a live environment and just not even try to set up composefs there.

Just for reference I don't think that's the right fix. I still think it makes sense to use composefs in this scenario...yes, there are "two layers" of read-only but IMO that's still fine. For example I would actually argue for example that we should enable fsverity by default for our ISOs...it obsoletes things like the anaconda "sha256 the whole filesystem before you run it" thing.

The messy thing here is the intersection between who sets up the filesystem writable portions. On the ostree side we already have the etc.transient flag - that covers /etc and is what we should use on ISOs. The messier thing is /var. Maybe we can also just add that to ostree and have it work the same way, pointing at a tmpfs.

The good thing about this is that from ostree's PoV then we could look at the block device underlying the target rootfs, and if that is physically ro then that tells us what we need to know, without having a special extar /run/ostree-live flag.

travier pushed a commit to coreos/fedora-coreos-config that referenced this issue Aug 30, 2024
In the composefs path, ostree-prepare-root want to mount /etc/ and /var
as writeable, which cannot in the live iso environnement.

Override the kernel command line to disable composefs in that case.

See ostreedev/ostree#1921
and #3009 (comment)
@jlebon
Copy link
Member

jlebon commented Aug 30, 2024

That sounds fine to me overall, but that seems like more of a follow-up to this? I'd be happy if we can just drop the /proc/cmdline hacks to start.

@jlebon
Copy link
Member

jlebon commented Aug 30, 2024

This oddly works now in a locally built FCOS live ISO and PXE artifacts:

Ahh, I think the reason this works now is that we're literally shipping a /boot in the squashfs. Not sure when that changed, but obviously it was a while ago now. It feels a bit awkward, but I guess we also need it now for the HMAC bits for FIPS (unless we also change the dracut module to look elsewhere). I think it'd still make sense to have ostree tolerate no /boot though, but also seems fine to close this issue for now until someone else shows up asking for it.

We have #1920 tracking the /proc/cmdline bit. And probably should have a separate issue for the "fold CoreOS live concepts into ostree"?

jlebon added a commit to jlebon/fedora-coreos-config that referenced this issue Nov 14, 2024
Currently, when building the squashfs, we drop the `sysroot.readonly`
flag from the ostree config because `ostree-prepare-root` doesn't know
how to handle it in the setup we have in our live environments.

But now inheriting from tier-x, we also inherited the move of that
knob to `/usr/lib/ostree/prepare-root.conf`, which is then included in
the initramfs. That's much harder to override during the build process
because we don't want to rebuild the initramfs. We could probably
instead just append a CPIO to the live initramfs that shadows it but the
real fix anyway is to adapt libostree to work in live environments.[^1]

So for now, just undo this bit to go back to how it was set up before
inheriting from tier-x, where the only sysroot.readonly knob lives in
the ostree repo config.

[^1]: ostreedev/ostree#1921
jlebon added a commit to jlebon/fedora-coreos-config that referenced this issue Nov 14, 2024
Currently, when building the squashfs, we drop the `sysroot.readonly`
flag from the ostree config because `ostree-prepare-root` doesn't know
how to handle it in the setup we have in our live environments.

But now inheriting from tier-x, we also inherited the move of that
knob to `/usr/lib/ostree/prepare-root.conf`, which is then included in
the initramfs. That's much harder to override during the build process
because we don't want to rebuild the initramfs. We could probably
instead just append a CPIO to the live initramfs that shadows it but the
real fix anyway is to adapt libostree to work in live environments.[^1]

So for now, just undo this bit to go back to how it was set up before
inheriting from tier-x, where the only sysroot.readonly knob lives in
the ostree repo config.

[^1]: ostreedev/ostree#1921
jlebon added a commit to coreos/fedora-coreos-config that referenced this issue Nov 18, 2024
Currently, when building the squashfs, we drop the `sysroot.readonly`
flag from the ostree config because `ostree-prepare-root` doesn't know
how to handle it in the setup we have in our live environments.

But now inheriting from tier-x, we also inherited the move of that
knob to `/usr/lib/ostree/prepare-root.conf`, which is then included in
the initramfs. That's much harder to override during the build process
because we don't want to rebuild the initramfs. We could probably
instead just append a CPIO to the live initramfs that shadows it but the
real fix anyway is to adapt libostree to work in live environments.[^1]

So for now, just undo this bit to go back to how it was set up before
inheriting from tier-x, where the only sysroot.readonly knob lives in
the ostree repo config.

[^1]: ostreedev/ostree#1921
aaradhak pushed a commit to aaradhak/fedora-coreos-config that referenced this issue Dec 4, 2024
Currently, when building the squashfs, we drop the `sysroot.readonly`
flag from the ostree config because `ostree-prepare-root` doesn't know
how to handle it in the setup we have in our live environments.

But now inheriting from tier-x, we also inherited the move of that
knob to `/usr/lib/ostree/prepare-root.conf`, which is then included in
the initramfs. That's much harder to override during the build process
because we don't want to rebuild the initramfs. We could probably
instead just append a CPIO to the live initramfs that shadows it but the
real fix anyway is to adapt libostree to work in live environments.[^1]

So for now, just undo this bit to go back to how it was set up before
inheriting from tier-x, where the only sysroot.readonly knob lives in
the ostree repo config.

[^1]: ostreedev/ostree#1921
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants