Add new `ex initramfs-etc` command #2170

jlebon · 2020-07-24T21:45:55Z

This command allows users to cheaply inject configuration files in the
initramfs stage without having to regenerate the whole initramfs (or
even a new OSTree commit). This will be useful for configuring services
involved in bringing up the root block device.

$ echo 'hello world' > /etc/foobar
$ rpm-ostree ex initramfs-etc --track /etc/foobar
Staging deployment... done
Run "systemctl reboot" to start a reboot
$ rpm-ostree status
State: idle
Deployments:
  ostree://fedora:fedora/x86_64/coreos/testing-devel
                   Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z)
                    Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b
              GPGSignature: (unsigned)
              InitramfsEtc: /etc/foobar
                  Unlocked: development

● ostree://fedora:fedora/x86_64/coreos/testing-devel
                   Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z)
                    Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b
              GPGSignature: (unsigned)
$ reboot
(boot into rd.break)
sh-5.0# cat /etc/foobar
hello world

See the libostree side of this at:
ostreedev/ostree#2155

Lots more discussions in:
coreos/fedora-coreos-tracker#94

openshift-ci-robot · 2020-07-24T21:45:57Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

jlebon · 2020-07-24T21:49:16Z

Requires: ostreedev/ostree#2155

Still need to write tests for this.

cgwalters · 2020-07-28T21:43:16Z

How often do we expect this stuff to change? Following up to ostreedev/ostree#2155 (comment) - what if all we did is support multiple initrds. Then the CoreOS code might generate a secondary initramfs (in the initramfs!) and install it as /boot/coreos-rootmap-initramfs.img or so, and update the bootloader configuration to use it...and that's it.

The experience for the administrator of "I want to update my rootmap" is certainly uglier but then it's clear - the "source of truth" is that file in /boot and we don't need any complex infrastructure to copy bits from /etc into images.

jlebon · 2020-08-13T16:02:23Z

Rebased!

Going to reply to this comment and ostreedev/ostree#2155 (comment) here.

Let's approach this from the other side. I.e. what do we want the UX to look like?

The major goals I'm trying to meet are the following:

Built into rpm-ostree: it should feel native to rpm-ostree; I think this command should replace rpm-ostree initramfs --enable whenever possible. Similarly, rpm-ostree status should show meaningful information about the state of the overlay.
The source of truth lives in /etc: users don't have to think about anything else, they can just edit the same files in /etc and it'll get updated on the next deployment. This is how rpm-ostree initramfs works as well. I think there might be use cases for having different configs in the real root and in the initrd, but we can extend the feature for that when we encounter it.
Generic: while the primary purpose is indeed for root device configuration, it shouldn't be designed with just that in mind. Configuring systemd and journald are good examples.
Rollback: config changes made to the overlay should be bound to a specific deployment so that users can rollback just as they would with rpm-ostree initramfs --enable. This argues for making it part of the bootcsum directory itself in /boot. (Edit: ended doing something a bit smarter here so we don't invalidate the bootcsum; see lib/deploy: Add support for overlay initrds ostreedev/ostree#2155).

Do you agree with the above?

Implementation-wise, I agree with you generating a new commit isn't ideal. Actually, my first approach at this was to extend the staging API to take additional initrds, so then e.g. rpm-ostree would create the initrd somewhere in /run and pass that to ostree_sysroot_{deploy,stage}_tree_with_options. This works, though the disconnect between the bits which handle staging and the bits which actually copy files into /boot made it a non-trivial patch in comparison to dffc850 (#2155). But maybe it's worth it in the end so we avoid having to create a new commit.

Re. the overlay not changing often, yeah it's tempting to try to make optimizations on that assumption. OTOH, the sizes we're talking about are minuscule in comparison to the initrd itself. One thought I had on this topic was it'd be interesting to have another OSTree repo in /boot so that we can share kernels and initrds across bootcsums.

I think if we're in agreement on the wanted UX, we could land this (once we add tests) and iterate on the implementation? Though there's no real rush either. I'll brush up the alternative libostree patch I had which will avoid having to generate a new commit.

jlebon · 2020-08-13T21:47:26Z

Updated as per ostreedev/ostree#2155 (comment)! Now with 100% less OSTree commit operations. (You can see in the updated sample output in #2170 (comment) that we just go straight to staging the deployment.)

cgwalters · 2020-08-18T15:27:00Z

We aren't doing change detection, right? IOW:

$ rpm-ostree ex initramfs --track /etc/systemd.conf
$ vi /etc/systemd.conf
...
$ rpm-ostree upgrade
No upgrade available.

And when I reboot, I'll have the old config until at some point something else changed, right?

I'd be OK if we added rpm-ostree ex initramfs update or something...but, a root problem here is having two copies of the files. What if instead the interface was:

$ cd $(mktemp -d)
$ cp /etc/systemd.conf .
$ vi systemd.conf
$ rpm-ostree ex initramfs --add .

Which would be a really thin wrapper around the libostree API (in fact, we should probably expose that via ostree admin deploy initramfs --add or so).

I am still skeptical that these initramfs overlays will need to be updated often; if we discover later that we need to make it more ergonomic to do so, we could add rpm-ostree ex initramfs --extract which would extract the initramfs into a temporary directory (but, one could also just use a small shell script to do so), then rerun -add.

(Implicit a bit in this is the idea there's at most one overlay, which seems fine to me; supporting multiple makes everything messy)

cgwalters · 2020-08-18T15:13:41Z

src/daemon/rpmostree-sysroot-upgrader.c

+}
+
+static gboolean
+generate_initramfs_overlay (GHashTable    *initramfs_etc_files,


Just a note/optional: Feels like we could move some of this into Rust.

jlebon · 2020-08-18T16:02:29Z

We aren't doing change detection, right?

Right, though that's a bug IMO. With dracut that's hard to do because we have no insight in what files get picked up. But for initramfs-etc we could create the cpio and compare checksums. (This PR doesn't do that yet... but I'll add it).

I am still skeptical that these initramfs overlays will need to be updated often

I don't disagree that in the common case they wouldn't be updated often. The new model in libostree reflects that (by not making it part of the bootcsum).

What if instead the interface was:

$ cd $(mktemp -d)
$ cp /etc/systemd.conf .
$ vi systemd.conf
$ rpm-ostree ex initramfs --add .

Hmm, perhaps. Though just directly pointing rpm-ostree at the files in /etc is a much nicer UX IMO. So with the change detection tweak mentioned above, you'd just have to edit /etc/systemd.conf then rpm-ostree initramfs-etc --sync (or just rpm-ostree upgrade). How does that sound?

lucab · 2020-09-17T13:39:47Z

One personal UX/design doubt on this point:

The source of truth lives in /etc: users don't have to think about anything else, they can just edit the same files in /etc and it'll get updated on the next deployment. This is how rpm-ostree initramfs works as well. I think there might be use cases for having different configs in the real root and in the initrd, but we can extend the feature for that when we encounter it.

I feel that the rest of the ecosystem is going in a direction where part of the source of truth (user overrides) lives in /etc. Those may not make sense taken in isolation without vendor defaults, i.e. the fragments in /usr.
Are we making the assumption that there is a symmetry in /usr between rootfs and initramfs? If so, is that assumption generally true in the current dracut+rpm world?

jlebon · 2020-09-28T19:12:30Z

Are we making the assumption that there is a symmetry in /usr between rootfs and initramfs? If so, is that assumption generally true in the current dracut+rpm world?

I think if an app wants to rely on its /usr configs in the initrd, the appropriate dracut module should already be pulling it in (as well as the /etc config really). IOW, this isn't unique to the initramfs-etc workflow, or even to OSTree-based systems (it's equally relevant on other dracut-based OSes, even if the initrd happens to be built client-side).

This is a basic `.packit.yaml` integration file which will allow us have continuous builds of OSTree in cosa and upstream CI. If things go well, we'll likely deploy this in other build tools like rpm-ostree. Prompted by wanting to get ostreedev#2155 out to unblock coreos/rpm-ostree#2170.

Get this in now so we get some early testing before releases next week. Also because it unblocks coreos/rpm-ostree#2170.

jlebon · 2020-10-19T18:24:15Z

Rebased this and restarted CI on it. Keeping as draft for now until I add tests, but the rest of the code should be ready for review!

jlebon · 2020-10-21T19:38:28Z

Now ready for review and with a test!

travier · 2020-10-22T13:03:16Z

It looks like this would fix the most common case for #1930.

This command allows users to cheaply inject configuration files in the initramfs stage without having to regenerate the whole initramfs (or even a new OSTree commit). This will be useful for configuring services involved in bringing up the root block device. ``` $ echo 'hello world' > /etc/foobar $ rpm-ostree ex initramfs-etc --track /etc/foobar Staging deployment... done Run "systemctl reboot" to start a reboot $ rpm-ostree status State: idle Deployments: ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) InitramfsEtc: /etc/foobar ● ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) $ reboot (boot into rd.break) sh-5.0# cat /etc/foobar hello world ``` See the libostree side of this at: ostreedev/ostree#2155 Lots more discussions in: coreos/fedora-coreos-tracker#94 Closes: coreos#1930

jlebon · 2020-10-23T14:26:51Z

Rebased on top of #2278!

It looks like this would fix the most common base for #1930.

Good catch, thanks! I added a Closes: #1930 footer to the commit message.

jlebon · 2020-10-23T19:46:24Z

✔️ continuous-integration/jenkins/pr-merge — This commit looks good

🎉

cgwalters · 2020-10-27T20:31:28Z

Two high level questions:

Is the idea that this will be ex to allow us to work on using it in FCOS for complex roots, and then we lift ex once that seems to be working well, but before we stabilize the FCOS usage of it?
I still don't much like the --force-sync thing - but presumably in most cases (as noted above) this will be for things that should very rarely change and so we don't need to worry about it?

jlebon · 2020-10-28T16:27:11Z

Is the idea that this will be ex to allow us to work on using it in FCOS for complex roots, and then we lift ex once that seems to be working well, but before we stabilize the FCOS usage of it?

Yes, exactly. We can refer users who need it to the new command and gather feedback there. Just playing it safe in case the UX shows clunkiness when used in more realistic situations.

I still don't much like the --force-sync thing - but presumably in most cases (as noted above) this will be for things that should very rarely change and so we don't need to worry about it?

Yeah, it's not great, but I think it fills a UX need. Let's say you use rpm-ostree ex initramfs-etc --track /etc/foobar.conf, reboot, and then realize you need to make a change to foobar.conf (I can imagine someone iterating on this multiple times as they try different versions of the config to test/debug something).

You'd have to do some artificial operation just to create a deployment (e.g. doing rpm-ostree deploy for the same version you're on). Having a clearly labeled switch for this feels much more natural I think. (And on that note, rpm-ostree initramfs could also use a switch like this. I can add that to make them more similar... in which case, maybe the switch should just be called --force?).

cgwalters · 2020-10-28T19:18:23Z

src/daemon/rpmostree-sysroot-upgrader.c

+
+  GInputStream *cpio_output = g_subprocess_get_stdout_pipe (cpio);
+  GOutputStream *gzip_input = g_subprocess_get_stdin_pipe (gzip);
+  g_output_stream_splice_async (gzip_input, cpio_output,


It'd be more efficient and simpler to just pass cpio's stdout as gzip's stdin, right?

cgwalters · 2020-10-28T19:19:58Z

src/daemon/rpmostree-sysroot-upgrader.c

+
+  const char *cpio_argv[] = {"cpio", "--create", "--format", "newc", "--quiet",
+                             "--reproducible", "--null", NULL};
+  g_autoptr(GSubprocess) cpio = g_subprocess_newv ((const char *const*)cpio_argv,


https://developer.gnome.org/gio/stable/GSubprocess.html#g-subprocess-communicate also takes stdin as a buffer, so we wouldn't need to splice anything in combination with the below.

cgwalters · 2020-10-28T19:25:48Z

src/daemon/rpmostreed-transaction-types.c

+}
+
+static gboolean
+initramfs_etc_transaction_execute (RpmostreedTransaction *transaction,


I thought we'd been trending a bit towards glomming everything into Deploy so it could be done in one go? I guess it doesn't much matter for this case, just curious if you had a reason to make this separate. (It is clearer to read separately, but more duplicated code)

cgwalters · 2020-10-29T21:49:43Z

OK I restarted CI on this and am OK merging as is, since it's ex we can fix things like the above in followups!

cgwalters · 2020-10-29T21:49:47Z

/lgtm

openshift-ci-robot · 2020-10-29T21:49:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, jlebon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cgwalters,jlebon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Get this in now so we get some early testing before releases next week. Also because it unblocks coreos/rpm-ostree#2170.

openshift-ci-robot added the do-not-merge/work-in-progress label Jul 24, 2020

openshift-ci-robot requested review from cgwalters and miabbott July 24, 2020 21:45

openshift-ci-robot added the approved label Jul 24, 2020

jlebon mentioned this pull request Jul 24, 2020

Prep patches for ex initramfs-etc #2171

Merged

jlebon force-pushed the pr/initramfs-etc branch from 8f9d718 to c166bb1 Compare July 25, 2020 14:38

jlebon mentioned this pull request Aug 7, 2020

Ship live PXE artifacts inside live ISO coreos/fedora-coreos-config#548

Merged

openshift-ci-robot added the needs-rebase label Aug 7, 2020

jlebon force-pushed the pr/initramfs-etc branch from c166bb1 to 6fe3364 Compare August 13, 2020 15:58

openshift-ci-robot removed the needs-rebase label Aug 13, 2020

jlebon force-pushed the pr/initramfs-etc branch from 6fe3364 to de26f73 Compare August 13, 2020 21:38

jlebon mentioned this pull request Aug 13, 2020

lib/deploy: Add support for overlay initrds ostreedev/ostree#2155

Merged

jlebon force-pushed the pr/initramfs-etc branch from de26f73 to 6a013dc Compare August 14, 2020 16:14

cgwalters reviewed Aug 18, 2020

View reviewed changes

cgwalters mentioned this pull request Sep 17, 2020

How to enable kdump feature on Fedora CoreOS coreos/fedora-coreos-tracker#622

Closed

jlebon force-pushed the pr/initramfs-etc branch from 6a013dc to 795746b Compare September 28, 2020 19:07

jlebon mentioned this pull request Oct 6, 2020

Add Packit integration ostreedev/ostree#2210

Merged

jlebon mentioned this pull request Oct 13, 2020

Release 2020.7 ostreedev/ostree#2214

Merged

jlebon added a commit to jlebon/fedora-coreos-config that referenced this pull request Oct 14, 2020

lockfiles: Fast-track ostree-2020.7-2.fc32

78655f4

Get this in now so we get some early testing before releases next week. Also because it unblocks coreos/rpm-ostree#2170.

jlebon force-pushed the pr/initramfs-etc branch from 795746b to 4a37d24 Compare October 18, 2020 22:19

jlebon force-pushed the pr/initramfs-etc branch from 4a37d24 to 7ba80d5 Compare October 21, 2020 19:35

jlebon marked this pull request as ready for review October 21, 2020 19:38

openshift-ci-robot removed the do-not-merge/work-in-progress label Oct 21, 2020

jlebon force-pushed the pr/initramfs-etc branch from 7ba80d5 to d5bd323 Compare October 21, 2020 20:39

jlebon force-pushed the pr/initramfs-etc branch from d5bd323 to 139cc28 Compare October 23, 2020 14:25

cgwalters reviewed Oct 28, 2020

View reviewed changes

openshift-ci-robot assigned cgwalters Oct 29, 2020

openshift-ci-robot added the lgtm label Oct 29, 2020

openshift-merge-robot merged commit 4da10dc into coreos:master Oct 29, 2020

This was referenced Oct 30, 2020

LUKS unlock screen always uses en-US keyboard layout fedora-silverblue/issue-tracker#3

Closed

Support kargs.d directories for default kargs (rebased) ostreedev/ostree#2217

Closed

jlebon mentioned this pull request Nov 26, 2020

Adding pquota to /etc/fstab for rootfs requires adding dracut rootflags argument #1059

Closed

kelvinfan001 pushed a commit to kelvinfan001/fedora-coreos-config that referenced this pull request Dec 14, 2020

lockfiles: fast-track ostree-2020.7-2.fc32

a8acaaf

Get this in now so we get some early testing before releases next week. Also because it unblocks coreos/rpm-ostree#2170.

jlebon mentioned this pull request Jan 14, 2021

overlay: nuke rhcos-growpart openshift/os#484

Merged

jlebon mentioned this pull request Jun 29, 2021

Automatic network config propagation in the initramfs when using Tang coreos/fedora-coreos-tracker#886

Open

jlebon mentioned this pull request Jul 14, 2021

How to handle machine-specific config in early boot coreos/fedora-coreos-tracker#464

Closed

jlebon deleted the pr/initramfs-etc branch April 23, 2023 23:32

mxj4 mentioned this pull request Aug 4, 2024

Misleading error message about initramfs regeneration #5033

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new `ex initramfs-etc` command #2170

Add new `ex initramfs-etc` command #2170

jlebon commented Jul 24, 2020 •

edited

Loading

openshift-ci-robot commented Jul 24, 2020

jlebon commented Jul 24, 2020

cgwalters commented Jul 28, 2020

jlebon commented Aug 13, 2020 •

edited

Loading

jlebon commented Aug 13, 2020

cgwalters commented Aug 18, 2020

cgwalters Aug 18, 2020

jlebon commented Aug 18, 2020

lucab commented Sep 17, 2020

jlebon commented Sep 28, 2020

jlebon commented Oct 19, 2020

jlebon commented Oct 21, 2020

travier commented Oct 22, 2020 •

edited

Loading

jlebon commented Oct 23, 2020

jlebon commented Oct 23, 2020

cgwalters commented Oct 27, 2020

jlebon commented Oct 28, 2020

cgwalters Oct 28, 2020 •

edited

Loading

cgwalters Oct 28, 2020

cgwalters Oct 28, 2020

cgwalters commented Oct 29, 2020

cgwalters commented Oct 29, 2020

openshift-ci-robot commented Oct 29, 2020

Add new ex initramfs-etc command #2170

Add new ex initramfs-etc command #2170

Conversation

jlebon commented Jul 24, 2020 • edited Loading

openshift-ci-robot commented Jul 24, 2020

jlebon commented Jul 24, 2020

cgwalters commented Jul 28, 2020

jlebon commented Aug 13, 2020 • edited Loading

jlebon commented Aug 13, 2020

cgwalters commented Aug 18, 2020

cgwalters Aug 18, 2020

Choose a reason for hiding this comment

jlebon commented Aug 18, 2020

lucab commented Sep 17, 2020

jlebon commented Sep 28, 2020

jlebon commented Oct 19, 2020

jlebon commented Oct 21, 2020

travier commented Oct 22, 2020 • edited Loading

jlebon commented Oct 23, 2020

jlebon commented Oct 23, 2020

cgwalters commented Oct 27, 2020

jlebon commented Oct 28, 2020

cgwalters Oct 28, 2020 • edited Loading

Choose a reason for hiding this comment

cgwalters Oct 28, 2020

Choose a reason for hiding this comment

cgwalters Oct 28, 2020

Choose a reason for hiding this comment

cgwalters commented Oct 29, 2020

cgwalters commented Oct 29, 2020

openshift-ci-robot commented Oct 29, 2020

Add new `ex initramfs-etc` command #2170

Add new `ex initramfs-etc` command #2170

jlebon commented Jul 24, 2020 •

edited

Loading

jlebon commented Aug 13, 2020 •

edited

Loading

travier commented Oct 22, 2020 •

edited

Loading

cgwalters Oct 28, 2020 •

edited

Loading