[WIP]: initrd: Use systemd #120015

ElvishJerricco · 2021-04-21T01:45:13Z

Motivation for this change

This parallelizes early boot.
Systemd generators like the cryptsetup and fstab ones can be used to handle mounting and prompting for passwords.
Plymouth can start much earlier, in early boot, making the boot graphics much cleaner.
Systemd services from other packages can be used.
systemd-ask-password can be used.

Things done

This is different approach than #74842. In the previous attempt, systemd was essentially only used to spawn the init script we had before. The goal was to gradually move code out of the init script and into systemd units. The new approach is to start with just the minimal systemd units and go from there. This ensures that things are working the way they're supposed to, and that the init script isn't acting as a crutch for a badly implemented migration to systemd.

The downside is that this version supports only a fraction of the existing initrd configuration options of NixOS. Many of these things will be easy to migrate, but I suspect getting everything implemented will be tedious. So I think it would be best to add special service units that run all the various pre/postFooCommands style options at the appropriate times. This should be able to bring back most or all of NixOS's initrd options. But these units should be disabled while working on migrating a feature to systemd units, in order to prevent the aforementioned crutch.

Another change in this approach has to do with extra-utils and the makeInitrd function. Systemd has a number of paths hard coded, such as the paths to the default generators, and to tools like the mount binary. It's usually possible to work around this as we do with extra-utils, but it can be challenging in some cases. It's far easier to put everything at their original paths and to rely on find-libs and manual effort to find dependencies rather than Nix closures, considering how few dependencies there tend to be that aren't found by find-libs and how obvious they usually are.

The initrd produced currently is quite a bit larger, at around 20M, but there's a lot of excess in it at the moment, so I'm sure it can be cut down quite a bit. This branch is no where near ready for any kind of serious use, but testing and feedback is appreciated.

PS: Having the emergency shell in initrd is extraordinarily useful, and being able to mess with things like devices or systemd units from initrd and retry the boot with systemctl default without rebooting or building a new initrd is a big time saver.

Things done

Ericson2314 · 2021-04-26T16:55:34Z

Good luck! :)

zimbatm · 2021-06-07T11:11:17Z

/cc @flokli

ElvishJerricco · 2021-06-08T21:32:41Z

I'm thinking I can rework this PR to be something like boot.initrd.useSystemd, similar to networking.useNetworkd, i.e. it could be merged and used as an opt-in feature rather than switching nixos over entirely.

Ericson2314 · 2021-06-08T22:38:30Z

Sounds good, and then someday we can change the default :).

nixos/modules/system/boot/luksroot.nix

arianvp · 2021-07-10T08:58:42Z

The rust program replacing the perl script seems to be the wrong direction to me. Nix now has a built-in called closureInfo we could use for dependency solving. E.g. my initrd generator looks like this: https://github.com/arianvp/server-optimised-nixos/blob/master/lib/make-initrd.nix

…

On Sat, 10 Jul 2021, 03:05 Will Fancher, ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In nixos/modules/system/boot/luksroot.nix <#120015 (comment)>: > environment.systemPackages = [ pkgs.cryptsetup ]; + + boot.initrd.emergencyPackages = [ pkgs.cryptsetup ]; Why would this mean that? This only governs the packages available on PATH in initrd in case you go to emergency/rescue mode in stage 1 — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#120015 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZNIZHGNSB4MMOXASRKULTW6MGRANCNFSM43JMAKKQ> .

arianvp · 2021-07-10T09:04:59Z

Sorry I commented without reading the rationale (was replying from my email client). So ignore my previous comment.

If i understand correctly from the PR description the rust program does more than just find the nix closure. I'll have to take a closer look to fully understand what's going on here. A commit message with the rationale would definitely help me understand better.

ElvishJerricco · 2021-07-10T20:52:44Z

Sorry I commented without reading the rationale (was replying from my email client). So ignore my previous comment.

If i understand correctly from the PR description the rust program does more than just find the nix closure. I'll have to take a closer look to fully understand what's going on here. A commit message with the rationale would definitely help me understand better.

It does less than finding the closure. It only resolves and preserves symlinks and finds shared libraries. It doesn't copy whole paths. This way I don't have to patch all kinds of things like systemd files or the systemd binary itself in ways that patchelf doesn't help with.

ElvishJerricco · 2021-07-10T20:56:37Z

To clarify, the point of the rust program is to do away with the "extra utils" idea in its entirety. Programs placed in their original paths in the initrd. So we don't want to copy whole closures, or even whole store paths. Just the files indicated by the nixos configuration, and minimal set of files those strictly depend on.

gytis-ivaskevicius · 2021-07-11T05:45:58Z

pkgs/build-support/kernel/make-initrd-ng.nix

+# Program used to compress the cpio archive; use "cat" for no compression.
+# This can also be a function which takes a package set and returns the path to the compressor,
+# such as `pkgs: "${pkgs.lzop}/bin/lzop"`.
+, compressor ? "gzip"


Out of curiosity - what is the reason of choosing gzip? No need for extra dependencies?

Just copying this:

nixpkgs/pkgs/build-support/kernel/make-initrd.nix

Line 28 in 14df81c

, compressor ? "gzip"

flokli · 2021-07-21T22:13:10Z

First, I'm very happy to see someone picking this up. systemd-in-initrd will solve many long-standing issues.

Some questions:

Can you elaborate on what make-initrd-ng is doing? Wondering if we need to write that in rust, or if a bash script might be more readable.

As for the apporach of not using the closure-info, but synthesizing and patchelf'ing something FHS-y: It seems to be very likely people might want to add their own services/port over some of the existing functionality into the initrd. I wonder if that approach will make this way more complicated, and how chasing dependencies (and the interface for it will look like).

I couldn't find the issue I discovered this in, but at least for sytemd-makefs, systemd-growfs etc., mkfs.* and mount should just be picked up from $PATH - so if we extend $PATH in that unit file to whatever binaries we have support enabled for, this should work without patching binaries.

ElvishJerricco · 2021-07-24T05:30:16Z

Wondering if we need to write that in rust, or if a bash script might be more readable.

I really tried to keep it down to bash, trust me :P But when I finally got a nearly functional bash implementation, 1) it took 3min to run (compared to this rust program's 2sec), and 2) I would hardly have called it readable.

Ultimately what the program does is copy objects into the target, respecting and replicating symlink structures, as well as recursing on ELF files or on directories. Symlinks are definitely the main reason it's complicated, which is kind of critical considering how systemd relies quite heavily on them. NixOS's plymouth implementation also relies on some symlinks that are ultimately what caused me to create this program.

but synthesizing and patchelf'ing something FHS-y:

And to clarify, the program does not patch any files at all; it copies them to their original locations and just puts some symlinks at places like /lib, /etc, and /bin so that some things are more trivial to make work. There's a number of binaries introduced in this approach that are very annoying to effectively patch; e.g. systemd hard codes the path to the mount command.

Ericson2314 · 2021-07-24T06:54:37Z

I also wonder if this rust script could be reused for a souped-up buildFHSEnv. That could be neat, and share the "burden" of this more wildly (though I am not concerned, bash is bad.)

andir · 2021-08-24T09:03:54Z

pkgs/build-support/kernel/make-initrd-ng/src/main.rs

@@ -0,0 +1,207 @@
+use std::collections::{HashSet, VecDeque};


A high-level description of what this tool is doing would be good to add here. It doesn't do a lot and you can grasp it by reading through it but the signal/noise ratio isn't that great. Helps future readers/debuggers to have a "one paragraph" idea of what goes in and out of here.

andir · 2021-08-24T09:06:05Z

pkgs/build-support/kernel/make-initrd-ng-tool.nix

@@ -0,0 +1,10 @@
+{ rustPlatform }:


You can drop the cargoSha256 and use the cargoLock attribute instead:

buildRustPackage { # ... cargoLock = { lockFile = ./Cargo.lock; }; # ... }

This will also allow pulling in dependencies and updating them without having to care about the vendored package hashsum

TIL, will do.

andir · 2021-08-24T09:06:24Z

pkgs/build-support/kernel/make-initrd-ng-tool.nix

+
+  src = ./make-initrd-ng;
+
+  cargoSha256 = "0vx7whn47372mg19i3vbf7ag5g0ylifx8rin2hcy81fyhmms16h6";


Would be good to add the usual meta attributes here as well.

andir

Thanks for working on this! I've had this on my backlog for a while now.

My primary concern with this big-bang migration is that we will break use-cases and only hear about it 6 months in (during the next release).

I think for a proper migration to systemd in stage1 we should

a) extract all the refactoring and land them already, basically being no-ops,
b) introduce a module system option to opt-in to systemd in stage1 to allow for early adopters. We can switch the default based on state version one release after initial support (or whatever the timeframe is).
d) Start with the minimal boot case at first & add a NixOS test for systemd in stage1 which we must extend with every new feature.
e) Start adding incremental support for features as we get to it / they are requested.

I am inline with your custom targets & units idea for backwards compatibility.

What do you say?

I am planning to work on this during the spring in December.

andir · 2021-08-24T09:07:39Z

nixos/default.nix

@@ -34,6 +34,7 @@ in

 {
  inherit (eval) pkgs config options;
+  inherit vmConfig;


Is this a leftover from debugging/still working? I don't think it is technically required, is it?

It's not technically required, so yea I can remove it from this PR. I do find it tremendously useful to have in nixos/default.nix though, FWIW.

andir · 2021-08-24T09:10:12Z

nixos/modules/system/boot/find-libs.nix

@@ -0,0 +1,46 @@
+{ writeShellScriptBin, patchelf }:


can we split out the refactoring of moving these two scripts into different files? I'd love to pick them into master already (as long as they are a no-op). This reduces the amount of changes this PR introduces and we can rule out some issues during the transition.

find-libs gets removed in a later commit. The rust program handles that work now.

andir · 2021-08-24T09:13:24Z

nixos/modules/virtualisation/qemu-vm.nix

@@ -714,6 +731,16 @@ in
            noCheck = true; # fsck fails on a r/o filesystem
          };
      });
+    boot.initrd.extraUnits."rw-store.service" = lib.mkIf cfg.writableStore (pkgs.writeText "rw-store.service" ''


Can't we use our regular systemd unit generation for this? It is a trivial unit yet doesn't allow any composition (without adding a unit named rw-store.service.d/overrides.conf or such?).

I thought about this, but I wasn't 100% sure if that would be too much. I'll look into it...

andir · 2021-08-24T09:15:04Z

nixos/modules/system/boot/stage-1.nix

@@ -454,7 +454,7 @@ in
      {
        symlink = "/etc/modules-load.d/nixos.conf";
        object = pkgs.writeText "nixos.conf"
-          (lib.concatStringsSep "\n" (config.boot.initrd.availableKernelModules ++ config.boot.initrd.kernelModules));
+          (lib.concatStringsSep "\n" config.boot.initrd.kernelModules);


Can you add some rational to "only load the minimum in modules-load.d"? Perhaps we did it over excessive? Would be good to add to the commit message in case this backfires.

No, including availableKernelModules was a mistake in my code in the first place. NixOS has never auto-loaded those; they're there for the kernel to request to be loaded if needed.

andir · 2021-08-24T09:17:23Z

nixos/modules/system/boot/systemd-units.nix


  # For some reason, running 'udevadm info --cleanup-db' in initrd causes
  # LUKS devices to get SYSTEMD_READY=0 in stage 2 (but not other devices)
  # which prevents LUKS encrypted file systems from being mounted in stage 2.
  rm $out/system/initrd-udevadm-cleanup-db.service

+  ln -s ../plymouth-start.service $out/system/initrd.target.wants/plymouth-start.service
+
+  rm $out/system/proc-sys-fs-binfmt_misc.automount


Do we have a good advice of developers when to add a line here? Previously units were opt-in and now we have some list of units that we don't want anymore.

For example I could see why we want the random seed in the initrd when unlocking the system using some network interaction that requires cryptography.

These services were simply the ones that created errors during boot. I didn't spend much time trying to figure out why they caused errors.

andir · 2021-08-24T09:18:09Z

nixos/modules/system/boot/stage-1.nix

  };

  fstab = pkgs.writeText "fstab" (lib.concatMapStringsSep "\n"
    ({ fsType, mountPoint, device, options, ... }:
      "${device} /sysroot${mountPoint} ${fsType} ${lib.concatStringsSep "," options}") fileSystems);

+  groups = [


What is the source of this list? When do we add groups here? Is that the list of groups that systemd somewhere specifies?

udev warned about non-existent groups in the journal, so I threw these in, in hopes they'd fix whatever I was debugging at the time. I don't think they're necessary at all, and can probably be removed.

ElvishJerricco · 2021-08-29T23:58:43Z

@andir I'm working on a re-tooling of this to address some of the things you mentioned, in particular separating this out into an alternative initrd, as opposed to a total replacement. I'll also check out if reusing the existing systemd module is a good idea (though I worry that it won't play nicely with the fact that the new initrd doesn't implicitly include the entire closure).

ElvishJerricco · 2021-08-30T00:42:18Z

Would it be too aggressive to factor out all the parts of nixos/modules/system/boot/systemd.nix that are relevant to both initrd and stage 2 into a separate file? Because certainly not all of it is relevant, and dotting the whole systemd.nix file with mkIfs depending on if we're evaluating initrd or stage 2 options seems confusing and error prone.

bobvanderlinden · 2022-03-13T01:40:19Z

Really cool PR! Thanks for the work so far. I tried to get systemd in initrd in the past, but got stuck a number of times. This PR was a good starting point for me 👍

I worked a bit on this PR today. Brought it up-to-date with nixos-unstable. I also worked on another improvement in makeInitrdNG so that it can avoid doing copying of files (it currently copies files to a temporary directory to make cpio happy). It speeds up the initrd creation a tad bit more. The changes are here: https://github.com/bobvanderlinden/nixpkgs/tree/rewrite-initrd-systemd

Also, I was made aware on Matrix that there is a room dedicated to Stage 1 Systemd. I thought it might be useful to mention it here too: https://matrix.to/#/#stage1systemd:helsinki-systems.de

samueldr · 2022-03-13T05:08:00Z

About making cpio archives, Linux provides a utility to do so without copying files around, and I have something made here in a WIP project to "borrow" the utility from Linux:

https://github.com/celun/celun/blob/41bd4c1f870ce73d59bb1348de6cabac7a145da3/pkgs/mkCpio/default.nix

And a module that produces the file format used by that utility:

https://github.com/celun/celun/blob/41bd4c1f870ce73d59bb1348de6cabac7a145da3/modules/wip/stage-1/contents.nix

I have no idea if it's any better than what you're using in your rework, @bobvanderlinden, but that's another option to look at I guess.

bobvanderlinden · 2022-03-13T16:32:38Z

@samueldr That does also look like quite an improvement. The usage of https://github.com/torvalds/linux/blob/master/usr/gen_init_cpio.c is nice, it has a better interface than cpio and would avoid the use of proot in my changes. I might look into integrating gen_init_cpio.c into my changes.

Determining which files need to go on the initrd is better in @ElvishJerricco's branch and the additional changes in mine. Instead of specifying each package (or file) that needs to go on the initrd, it can find the dependencies of files and only include those. So, instead of including the full systemd package, it'll only include the files that ${systemd}/lib/systemd/systemd refers to. It does this recursively. @ElvishJerricco did this using patch_elf. It results in a smaller initrd.

I also added that it would look for /nix/store/XXXXX... paths in other (non-ELF) files. That means that whenever a config file (like systemd service files) refer to an executable, those executables (and all of their .so dependencies) are included in the initrd automatically. Any implicit dependencies do need to be added manually. I think this would help with other NixOS modules that need to add something to initrd: only define a systemd service file and it should have most of the dependencies available.

Today I worked on refactoring the NixOS systemd module so that it will be easier to reuse unit configuration for initrd's systemd. See #164016

I think next up is looking for a way to make this PR compatible with extraUtilsCommands, postDeviceCommands and postMountCommands.

@dasJ gave some feedback on Matrix (btw, stage1-systemd room there is very appropriate for this PR: https://matrix.to/#/#stage1systemd:helsinki-systems.de). I revise my above comment: the next step is making sure systemd can be used as an option instead of a replacement of stage-1.nix.

ElvishJerricco · 2022-03-20T08:04:23Z

Closing in favor of #164943

flokli · 2023-01-31T12:07:58Z

nixos/modules/tasks/filesystems.nix

+      [Unit]
+      IgnoreOnIsolate=yes
+      [Service]
+      Environment=PATH=${lib.concatMapStringsSep ":" (p: "${lib.getBin p}/bin") config.system.fsPackages}


Did this adding to $PATH end up somewhere?

@flokli I believe it's covered by the DefaultEnvironment in system.conf

ElvishJerricco added 5 commits April 20, 2021 18:15

Add vmConfig to nixos/default.nix

6987073

initrd: Rewrite to use systemd

d3e6e2a

systemd: Use systemd-makefs and growfs

5c50976

qemu-vm: Support systemd initrd

dc756fe

initrd: Support luks with systemd and /etc/crypttab

cad9847

ElvishJerricco requested review from infinisil and nbp as code owners April 21, 2021 01:45

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Apr 21, 2021

ElvishJerricco marked this pull request as draft April 21, 2021 01:45

initrd: Only load the minimum in modules-load.d

1404a51

flokli requested a review from arianvp June 8, 2021 12:14

flokli mentioned this pull request Jun 8, 2021

Declarative partitioning with systemd-repart/growfs/makefs #87073

Closed

github-actions bot added the 6.topic: kernel The Linux kernel label Jul 9, 2021

ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jul 9, 2021

ElvishJerricco force-pushed the rewrite-initrd-systemd branch 2 times, most recently from afb5257 to baa14e3 Compare July 9, 2021 17:57

ElvishJerricco added 2 commits July 9, 2021 14:00

systemd-initrd: Use a Rust script for dependency solving

7fd29f6

systemd-initrd: Cleanup some systemd units

c604154

ElvishJerricco force-pushed the rewrite-initrd-systemd branch from baa14e3 to d3f0dcd Compare July 9, 2021 18:01

SuperSandro2000 reviewed Jul 9, 2021

View reviewed changes

nixos/modules/system/boot/luksroot.nix Show resolved Hide resolved

gytis-ivaskevicius reviewed Jul 11, 2021

View reviewed changes

systemd-initrd: Remove defunct make-initrd.nix and find-libs.nix

78d09d5

ElvishJerricco force-pushed the rewrite-initrd-systemd branch from d3f0dcd to 78d09d5 Compare July 11, 2021 19:20

zhaofengli mentioned this pull request Aug 24, 2021

luks: automatic tpm unlocking #134577

Closed

11 tasks

andir reviewed Aug 24, 2021

View reviewed changes

samueldr mentioned this pull request Sep 6, 2021

PSA: systemd in initramfs #72401

Closed

dasJ mentioned this pull request Sep 8, 2021

nixos/stage-2-init: Pass arguments #137122

Closed

12 tasks

pasqui23 mentioned this pull request Sep 17, 2021

[WIP]: initialRamdisk: Use systemd for initrd's /init #74842

Closed

19 tasks

TLATER mentioned this pull request Sep 26, 2021

sign bootloader to make it compatible with UEFI secure boot? #42127

Open

ju1m mentioned this pull request Oct 23, 2021

nixos/security.gnupg: provisioning GnuPG-protected secrets through the Nix store #93659

Closed

26 tasks

flokli mentioned this pull request Dec 8, 2021

Expand support for and document systemd-cryptenroll #139433

Closed

Artturin mentioned this pull request Dec 25, 2021

nixos/systemd: disable systemd-gpt-auto-generator by default #146497

Closed

12 tasks

bobvanderlinden mentioned this pull request Mar 13, 2022

nixos: systemd: split module up into multiple files #164016

Merged

13 tasks

bobvanderlinden mentioned this pull request Mar 15, 2022

nixos: systemd: split off helper functions into systemd-lib #164317

Merged

13 tasks

ElvishJerricco closed this Mar 20, 2022

flokli reviewed Jan 31, 2023

View reviewed changes


		src = ./make-initrd-ng;

		cargoSha256 = "0vx7whn47372mg19i3vbf7ag5g0ylifx8rin2hcy81fyhmms16h6";

[WIP]: initrd: Use systemd #120015

[WIP]: initrd: Use systemd #120015

Conversation

ElvishJerricco commented Apr 21, 2021

Motivation for this change

Things done

Things done

Ericson2314 commented Apr 26, 2021

zimbatm commented Jun 7, 2021

ElvishJerricco commented Jun 8, 2021

Ericson2314 commented Jun 8, 2021

arianvp commented Jul 10, 2021 via email

arianvp commented Jul 10, 2021

ElvishJerricco commented Jul 10, 2021

ElvishJerricco commented Jul 10, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flokli commented Jul 21, 2021

ElvishJerricco commented Jul 24, 2021

Ericson2314 commented Jul 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andir left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ElvishJerricco commented Aug 29, 2021

ElvishJerricco commented Aug 30, 2021 • edited Loading

bobvanderlinden commented Mar 13, 2022

samueldr commented Mar 13, 2022

bobvanderlinden commented Mar 13, 2022 • edited Loading

ElvishJerricco commented Mar 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ElvishJerricco commented Aug 30, 2021 •

edited

Loading

bobvanderlinden commented Mar 13, 2022 •

edited

Loading