nixos-rebuild on proxmox-lxc container fails with busctl error, and causes other configuration change from base image

[smacz42]

Summary

Running nixos-rebuild on a customized (or vanilla) proxmox-lxc image is unsuccessful, renders the container into an unmanageable state, and some configuration seems to be removed.

Steps to reproduce:

1. Build the container
   a. nix run --extra-experimental-features nix-command --extra-experimental-features flakes github:nix-community/nixos-generators --format proxmox-lxc -c /tmp/firstboot.nix
   b. cat << EOF > /tmp/firstboot.nix

{ config, pkgs, ... }:

{
  # Set up a systemd service
  systemd.services.startup = {
    description = "Sets up the NixOS container on startup";
    wantedBy = [ "multi-user.target" ];
    script = "echo 'Hello World'"
  }
}
EOF

2. Run the container in Proxmox
3. Create a minimal configuration.nix:

{config, pkgs, ... }:

{
  imports = [ <nixpkgs/nixos/modules/virtualisation/lxc-container.nix> ];

  environment.variables = {
    HISTFILESIZE = "";
    HISTSIZE = "";
    HISTTIMEFORMAT = "%F %T ";
    NIX_SSL_CERT_FILE = "/etc/ssl/certs/ca-certificates.crt";
  };

  systemd.mounts = [{
    where = "/sys/kernel/debug";
    enable = false;
  }];

  environment.systemPackages = with pkgs; [
    vim
    binutils
  ];
}

4. Run nixos-rebuild test

'/nix/store/zgzrbba39fsn341s5dyl89wi7cdavsf0-system-path/bin/busctl --json=short call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager ListUnitsByPatterns asas 0 0' exited with value 1 at /nix/store/nhys1a2wsn5x5xm5bv5msk6ynqrhya4q-nixos-system-nixos-23.11.5408.8ac30a39abc5/bin/switch-to-configuration line 145.

a. This actually borks the system the exact same as a switch would, even though it's only a test.
5. Re-run nixos-rebuild switch

building Nix...
building the system configuration...
trace: warning: system.stateVersion is not set, defaulting to 23.11. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
stopping the following units: network-local-commands.service, systemd-networkd-wait-online.service, systemd-networkd.service, systemd-networkd.socket, systemd-resolved.service
activating the configuration...
setting up /etc...
removing obsolete symlink ‘/etc/resolv.conf’...
removing obsolete symlink ‘/etc/man_db.conf’...
removing obsolete symlink ‘/etc/systemd/networkd.conf’...
removing obsolete symlink ‘/etc/systemd/resolved.conf’...
restarting systemd...
Failed to list users: Failed to activate service 'org.freedesktop.login1': timed out (service_start_timeout=25000ms)
Unable to close the file handle to loginctl at /nix/store/nhys1a2wsn5x5xm5bv5msk6ynqrhya4q-nixos-system-nixos-23.11.5408.8ac30a39abc5/bin/switch-to-configuration line 890.
warning: error(s) occurred while switching to the new configuration

Issues

Pre-reboot:

1. busctl issue above as output of nixos-rebuild
2. Cannot shutdown as root in container:

Failed to set wall message, ignoring: Access denied
Call to Reboot failed: Access denied

3. dbus errors in logs:

dbus-daemon[280]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.5" (uid=0 pid=16558 comm="systemd-run -E LOCALE_ARCHIVE -E NIXOS_INSTALL_BOO" label="unconfined") interface="org.freedesktop.systemd1.Manager" member="StartTransientUnit" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/run/current-system/systemd/lib/systemd/systemd" label="unconfined")
dbus-daemon[280]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.6" (uid=0 pid=16563 comm="/nix/store/zgzrbba39fsn341s5dyl89wi7cdavsf0-system" label="unconfined") interface="org.freedesktop.systemd1.Manager" member="ListUnitsByPatterns" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/run/current-system/systemd/lib/systemd/systemd" label="unconfined")
dbus-daemon[280]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.7" (uid=0 pid=16571 comm="shutdown -r now" label="unconfined") interface="org.freedesktop.login1.Manager" member="SetWallMessage" error name="(unset)" requested_reply="0" destination="org.freedesktop.login1" (uid=0 pid=291 comm="/nix/store/dzp7d4k1d94s1x49p9171mvcsfyxr7bj-system" label="unconfined")
dbus-daemon[280]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.7" (uid=0 pid=16571 comm="shutdown -r now" label="unconfined") interface="org.freedesktop.login1.Manager" member="RebootWithFlags" error name="(unset)" requested_reply="0" destination="org.freedesktop.login1" (uid=0 pid=291 comm="/nix/store/dzp7d4k1d94s1x49p9171mvcsfyxr7bj-system" label="unconfined")

Post-reboot:

1. Hostname is reset to nixos
   a. This does not happen if I use the vanilla image from hydra.
2. Custom systemd services disappear
   a. This is probably because it's not defined in the configuration.nix file that got rebuilt against, but I wasn't expecting this behavior.
3. Odd changes when running nixos-rebuild switch again:

setting up /etc...
removing obsolete symlink ‘/etc/resolv.conf’...
removing obsolete symlink ‘/etc/man_db.conf’...
removing obsolete symlink ‘/etc/systemd/resolved.conf’...
removing obsolete symlink ‘/etc/systemd/networkd.conf’...
setting up tmpfiles
Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted
reloading the following units: dbus.service
restarting the following units: nix-daemon.service
starting the following units: network-local-commands.service
the following new units were started: dhcpcd.service, network-setup.service, resolvconf.service

Suspicions

I suspect this is because:

1. The /etc/nixos/configuration.nix is overriding whatever the build was built with, which disables everything that the container was built with (including hostname.)
2. I also suspect that the installation of glibc and whatever else gets installed causes an error for the (unprivileged) container restarting systemd services.
   a. I'm not sure, but this might have something to do with: nixos-rebuild ready image #86

I would expect (without having my understanding of the internals of nixos) to be able to take the base image of the container, and to create a /etc/nixos/configuration.nix and run nixos-rebuild switch that does not break and/or modify the configuration of the base image, or have a way in which to include the configuration of the base image in such a way as to preserve the existing configuration.

I'm happy to do any further testing required in regards to these issues :)

[smacz42]

Just to follow-up... are we not supposed to be able to run nixos-rebuild on these images? I guess I don't understand. Basically, is my "I would expect..." line above inaccurate?

[mayl]

I don't use proxmox or lxc so I can't comment on any of that specifically.

Broadly, nixos configurations as part of being declarative and reproducible, need to be complete. The result you get running nixos-rebuild is a function only of the config you pass, not the state of the current system. I describe this to say the expectation of a merge with current state you seem to describe is not a good mental model for what happens.

I suspect if you copy your config into the container and rebuild from that it could work, but I can't speak too much to that either. The way I use nixos-generators is to specify the full config up front, and rebuild a new container if I need changes. Except I mostly build vms, not containers :⁠-⁠)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-proxmox-lxc-not-rebuilding-using-wiki-provided-configuration/47104/8

[hogcycle]

Glad to see I'm not the only one having issues. So is this a feature, not a bug? Should I be doing all of my configuration and packing it into the tarball? It seems unnecessary to have to do that for even the smallest of changes.