nixos/security.gnupg: provisioning GnuPG-protected secrets through the Nix store

[ju1m]

Motivation for this change

This security.gnupg proposal aims at granularly sending, decrypting, and installing secrets (and if need be reloading services depending on them) coming from GnuPG encrypted files put into the Nix store, by leveraging password-store, gpg-agent, systemd and nix copy.

Thanks in advance for your feedbacks.

Features

• Read from a local password-store's $PASSWORD_STORE_DIR or any .gpg loadable into the Nix store by any mean available.
• Check at build-time that needed secrets exist (in config.security.pass.secrets).
• No need to be able to decrypt the secrets at build-time nor locally.
• Send only secrets actually used (not the whole password-store), and do not resend them unless they've changed.
• Load only in RAM the passphrase to use the decrypting OpenPGP key(s) (with gpg-preset-passphrase).
• Remove volatile secrets when unconfigured (leveraging RuntimeDirectory), or persistent secrets only if their postStop is configured to do so.
• Restart or reload only needed services (one file in the Nix store and one secret-${secret}.service per entry in config.security.gnupg.secrets allow fine grained postStart, after, wants, or requires).
• Allow decorating secrets with non-secret bits (using the option pipe).
• Simpler setup than using extraBuiltins.exec calling pass and/or nixops send-keys.
• AFAIU, trusting or signing the activated NixOS generation removes the need to verify secrets.
• Use a dedicated gnupg user instead of root to run gpg-agent.
• Provide a helper (config.security.gnupg.agent.sendKeys) to send through ssh a decrypting key and its password to gpg-agent.

TODO

1. Get more real-life tests and feedbacks.
2. Put the present documentation in the manual.
3. Write a unit test in nixos/tests/.
4. Test using smartcard on the target machine.

Inconvenients

1. An OpenPGP key (preferably dedicated for the task and maybe per target machine) must be generated (with the encryption capability).
2. Encrypted secrets are readable by all users in the Nix store, if this is a concern this can maybe be restricted by leveraging apparmor (for which I happen to be proposing the PR apparmor: fix and improve the service #93457 ).

Example

First just rebuild your configuration without configuring secrets, just to enable gpg-agent.service, using:

security.gnupg.agent.enable = true;

Then make sure secrets are (re)encrypted to the right OpenPGP recipients (eg. [email protected]'s key):

pass init -p machines/example @admin1@ @admin2@ @[email protected]

Now configure and use some secret (eg. pass hosts/foo/transmission/settings.json) :

{ config, lib, pkgs, ... }:
let inherit (config.security.gnupg) secrets; in
{
# Although it is a path, it is used by `security.gnupg`
# so that only needed `.gpg` files within it will go into the Nix store
# (unless it is set to a path beginning by an entry of the `flake.nix`'s `inputs`,
# in which case it is already inside the Nix store).
security.gnupg.store = .password-store/hosts/example;

# root:root ownership with 400 mode by default
security.gnupg.secrets."transmission/settings.json" = {};

# Enable gpg-agent.service and the security.gnupg.agent.sendKeys script
# to send the encrypt subkey specified by its keygrip
# (obtained with: gpg --list-secret-keys --with-keygrip --with-keygrip)
security.gnupg.agent = {
  # Activate `gpg-agent` on the target host.
  # Unless you choose to [forward the agent](https://wiki.gnupg.org/AgentForwarding)
  # of the orchestrating host, to `/var/lib/gnupg/.gnupg/S.gpg-agent`.
  enable = true;
  keyring."9AA84E6F6D71F9163C46BF396B141A0806219077" = {
    # Defaults to config.security.gnupg.store + "/keygrip/9AA84E6F6D71F9163C46BF396B141A0806219077.gpg"
    passwordGpg = .password-store/hosts/example/root.pass.gpg;
  };
};

# It's not needed here because `credentialsFile` is set up by the `root` user
# but for some services their user would have to be added to the `keys` group.
#users.users.transmission.extraGroups = [ config.users.groups.keys.name ];

systemd.services.transmission = {
  after = [ secrets."transmission/settings.json".service ];
  # No reload for transmission: always restart if the secret has changed.
  # Use `wants` and `security.gnupg.secrets."path/to/secret".systemdConfig.postStart`
  # for a service that can be reloaded.
  requires = [ secrets."transmission/settings.json".service ];
};

services.transmission = {
  enable = true;
  credentialsFile = secrets."transmission/settings.json".path;
};
}

Without a flake.nix

From there you can use <nixpkgs/nixos> as usual, interleaving a call to sendKeys:

nix-build ./nixos.nix -A system -o nixos
nix copy --to ssh://[email protected] --substitute-on-destination ./nixos
nix-build ./nixos.nix -A security.gnupg.agent.sendKeys -o sendKeys && ./sendKeys/bin/gnupg-agent-sendKeys
ssh [email protected] "
 nix-env --profile /nix/var/nix/profiles/system --set $(readlink -m ./nixos) &&
 /nix/var/nix/profiles/system/bin/switch-to-configuration switch"

Logged in as [email protected], you can then list the systemd units responsible to install those secrets with:

systemctl list-units 'secret-*' --all

and check the ones decrypted on the default destination, like so:

ls -l /run/keys/gnupg/**/file

With a flake.nix

{
inputs.nixpkgs.url = "flake:nixpkgs";
inputs.flake-utils.url = "github:numtide/flake-utils";
inputs.pass = { type = "path"; path = "./pass"; flake = false; };
outputs = inputs:
  {
    # Usage: nix -L build .#nixosConfigurations.$hostName.config.system.build.toplevel
    nixosConfigurations = builtins.mapAttrs (hostName: hostConfig:
      let cfg = import hostConfig { inherit inputs; }; in
      import (inputs.nixpkgs + "/nixos/lib/eval-config.nix") (cfg // {
        extraArgs = {
          inherit hostName inputs;
          hosts = inputs.self.nixosConfigurations;
        } // (cfg.extraArgs or {});
        modules = cfg.modules ++ [({pkgs, ...}: {
          nix.extraOptions = "experimental-features = nix-command flakes";
          security.gnupg.store = inputs.pass + "/hosts/${hostName}";
        })];
      }))
      {
        losurdo = hosts/losurdo.nix;
        mermet  = hosts/mermet.nix;
      };
  }
  // inputs.flake-utils.lib.eachDefaultSystem (system:
    let pkgs = inputs.nixpkgs.legacyPackages.${system}; in
    {
    apps = builtins.mapAttrs (hostName: { config, ... }: let
      system = config.system.build.toplevel;
      target = "root@${config.networking.hostName}.${config.networking.domain}";
      profile = "/nix/var/nix/profiles/system";
      in rec {

      # Usage: nix run .#$hostName.switch
      "switch" = {
        type = "app";
        program = (pkgs.writeShellScript "switch" ''
          set -eux
          nix-store --add-root hosts/${hostName}.root --indirect --realise ${system}
          nix copy --to ssh://'${target}' --substitute-on-destination ${system}

          ${sendkeys.program}

          ssh '${target}' \
            nix-env --profile '${profile}' --set '${system}' '&&' \
            '${profile}'/bin/switch-to-configuration switch
        '').outPath;
      };

      # Usage: nix run .#$hostName.sendkeys
      "sendkeys" = {
        type = "app";
        program = config.security.gnupg.agent.sendKeys + "/bin/gnupg-agent-sendKeys";
      };
    }) inputs.self.nixosConfigurations;}
  );
}

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[Mic92]

Interesting I was working on this: https://github.com/Mic92/sops-nix

[Mic92]

So does this service starts a new gnupg daemon? Because it bind mounts /var/lib/gnupg/empty?

[d-goldin]

Sorry for the late response, was kinda swamped.

Removing myself from reviewers, since I'm personally not very keen on an approach tied to a specific "backend", but also don't want to block this PR in any way.

Aside from that, without having tried it, I think this is a rather small addition easy enough to get in and can be useful enough to some people/setups with only moderate transition costs should something else come around.

What I think could be important here to get more feedback is an updated documentation on the usage scenarios early on, including scenarios in which this would not work well (i.e. to aid reviewers/testers gain understanding quicker). For instance, from first glance it seems to me like this would not work with services utilizing DynamicUser, of which we can expect more in the future based on https://github.com/NixOS/rfcs/blob/master/rfcs/0052-dynamic-ids.md. Maybe it would be possible to add some checks to break early on for services which don't fulfill those potential expectations.

What would be also good to check is whether this could benefit from some likely upcoming related changes in systemd and how this might change the interface (see systemd/systemd#16568).

[Mic92]

Sorry for the late response, was kinda swamped.

Removing myself from reviewers, since I'm personally not very keen on an approach tied to a specific "backend", but also don't want to block this PR in any way.

Aside from that, without having tried it, I think this is a rather small addition easy enough to get in and can be useful enough to some people/setups with only moderate transition costs should something else come around.

What I think could be important here to get more feedback is an updated documentation on the usage scenarios early on, including scenarios in which this would not work well (i.e. to aid reviewers/testers gain understanding quicker). For instance, from first glance it seems to me like this would not work with services utilizing DynamicUser, of which we can expect more in the future based on https://github.com/NixOS/rfcs/blob/master/rfcs/0052-dynamic-ids.md. Maybe it would be possible to add some checks to break early on for services which don't fulfill those potential expectations.

What would be also good to check is whether this could benefit from some likely upcoming related changes in systemd and how this might change the interface (see systemd/systemd#16568).

DynamicUser can be also supported like this: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/monitoring/prometheus/alertmanager.nix#L170
By using envsubst and EnvironmentFile one can work-around permission by making systemd read the secrets.

[ju1m]

I've updated according to @rycee's review. Beware that I've also removed wantedBy = ["multi-user.target"]; for the decrypting services, to avoid decrypting unused secrets, this means that you now must add systemd dependencies indications (after/before and requires/requiredBy or wants/wantedBy+postStart = "systemctl reload foo"), example:

security.gnupg.secrets."tor/onion/${onion}/hs_ed25519_secret_key" = {
  systemdConfig.before = [ "tor.service" ];
  systemdConfig.requiredBy = [ "tor.service" ];
};

I've also moved postStart and postStop to systemdConfig.postStart and systemdConfig.postStop.

[ju1m]

Introducing the possiblity to nix -L run .#nixosConfigurations.${hostName}.config.security.gnupg.agent.sendKeys.
I've also changed --homedir from /var/lib/gnupg to /var/lib/gnupg/.gnupg to be able to ssh gnupg@ without setting a GNUPGHOME= (though I could). At this point, I'm not making ssh gnupg@ the default because using root is more straightforward for the "sending and installing secrets" use case. But the gnupg user and the gnupg group are now there to explore more use cases.

[ju1m]

Opt-out security.gnupg.agent.enable by default because it can now be enabled without any secret configured. Useful to start gpg-agent.service in a first rebuild without blocking indefinitely when sending secret that can't be installed because the password could not have been sent before that first rebuild activating gpg-agent.service, as explained in the updated intro.

[ju1m]

Improved the documentation a little bit and fixed UMask typo.

[ju1m]

Fixed permission of /run/user/$uid for non-root users.

[ju1m]

Sorry, I forgot I had rebased a bit so Github's force-pushed link is not pretty; the diff is b3dc3af437f..58410e3735e, changelog is:

• Support relative paths in security.gnupg.agent.keyring.${keygrip}.passwordGpg, like "gnupg/root.gpg". I am keeping the default to "keygrip/${keygrip}.gpg", but I know it's a bit catch22 when the key is created from an encrypted password which can't obviously already have the keygrip in it's path.
• Add "-o" "StrictHostKeyChecking=yes" to the default security.gnupg.agent.keyring.${keygrip}.ssh, to better ensure secrets are not sent where they should not.
• Use lib.mapNullable.
• Fix typo in tracing comment.

[ju1m]

• Fix config.security.gnupg.agent.sendKeys: the absence of the secret key on the target's keyring was not detected properly.
• Add $TARGET environment variable to override temporarily the SSH target (eg. when there is no hostname configured for the target, it's convenient to just use something like [email protected] nix -L run .#target.sendkeys).
• Add a no-op default to config.security.gnupg.agent.sendKeys, useful in case there is no secret configured for a particular host, but the script is being called systematically for all hosts.

[ju1m]

• Add option security.gnupg.agent.keyring.${keygrip}.passwordFile for unattended decryption.

[ju1m]

What would be also good to check is whether this could benefit from some likely upcoming related changes in systemd and how this might change the interface (see systemd/systemd#16568).

systemd v250 shall bring the systemd-creds utility (wrongly announced for v249 in that manpage). I'll have to actually use the new feature to be sure, but so far I'm looking forward to abandon this security.gnupg module in favor of LoadCredentialEncrypted= and SetCredentialEncrypted=.

Some pros:

• Built in systemd (when built with +OPENSSL which is done by default on NixOS), this brings more portability, and reviewing than this security.gnupg module.
• No GnuPG shenanigans...
• No ExecStartPre=/EnvironmentFile= workarounds for handling User=/DynamicUser=.
• Can decrypt using a TPM2 security chip and/or a root-owned persistent media in /var/lib/systemd/credential.secret. I guess this file should enable both unattended and attended decryption (sending this file), but this will need testing.
• No headache to setup after=/wants=/requires= constraints: credentials are always decrypted only when and as long as a service is using them.
• Credentials can be propagated to containers.
• Credentials can expire.
• At some point it should also work in the initrd.

Some cons:

• Credentials are currently not available in ExecStartPre=, hence wrapping ExecStart= in a script is needed when the credentials must be merged into a config file (eg. services.transmission).
• The new systemd-creds must be used instead of the venerable gpg and password-store.
• I don't see any systemd-creds option for using a credential.secret outside /var/lib/systemd/, thus when preparing on an orchestrating host the credentials for some other host, one may have to reach to that host: systemd-ask-password -n | ssh root@example systemd-creds encrypt --name=foo - -" >hosts/example/foo.cred.
• Currently, systemd enforces an accumulated credential size limit of 1 MB per unit, should not be a problem for passwords but security.gnupg is less limiting.