Unlock encrypted root filesystem via SSH

[terem42]

This commit add new feature for Debian-based distributions to
unlock encrypted root partition over SSH This feature is very handy on any
headless NAS or VPS cloud servers. To use this feature, you will need to
install dropbear-initramfs package during the system setup.

Signed-off-by: Andrey Prokopenko [email protected]

Motivation and Context

Many NAS and cloud-based servers requiring encrypted root filesystem do not have displays or keyboard connected, hence the usage of encrypted partitions is limited to servers with physical access to keyboard or the usage of rather ugly and insecure schemas where key is saved along with the initramfs image, making it easily prone for breaking into. Proposed pull request removes this vulnerability for Debian based systems and makes usage of encrypted root filesystems on headless systems, requiring encryption, much more convenient.

Description

New section inside initramfs zfs setup script checks for the presense of working dropbear SSH process ( to be installed via dropbear-initramfs package) and if it's present, opens two in and out named pipes, then waits for the password input and sends the pool decryption result back.
On the login via SSH, if dropbear module is present and running, the user is presented with password prompt. Once the password is accepted, dropbear initramfs process will be killed and user is requested to reconnect via normal user credentials.

How Has This Been Tested?

The standard set of zfs-linux debian packages was built via Github Actions free tier build based on my workflow, see more details here: https://github.com/andrey42/zfs-debian/blob/master/.github/workflows/debian-packages-build.yml
Two VM with encrypted partitions were created using aforementioned packages to test booting process with and without encrypted root filesystem. Test scripts for hosting provider are available here: https://github.com/andrey42/zfs-hetzner-vm
Two complete scripts for creating barebone installation using deboostrap, experimental version of ZFS packages and dropbear-initramfs with SSH keys config on Debian 10 and Ubuntu 18.04 are located here: https://github.com/andrey42/zfs-hetzner-vm/tree/vmtest

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[codecov]

Codecov Report

Merging #10027 into master will decrease coverage by <1%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master   #10027    +/-   ##
========================================
- Coverage      79%      79%   -<1%     
========================================
  Files         386      386            
  Lines      122431   122431            
========================================
- Hits        97023    96892   -131     
- Misses      25408    25539   +131

Flag	Coverage Δ
#kernel	79% <ø> (ø)	⬆️
#user	66% <ø> (-1%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ecbbdac...d104fcd. Read the comment docs.

[felixdoerre]

Just as an idea: if the target is debian, why not use the /lib/cryptsetup/askpass utility from the cryptsetup-bin package? This utility already has the functionality of simultaneously asking for the password on the console, plymouth, systemd and a fifo, which can be used from a dropbear-ssh session with cryptroot-unlock. Thereby we always try to ask on the console and never accidently lock-out an user (who may have installed dropbear-initramfs but not set up an sshkey for dropbear yet)

[terem42]

Just as an idea: if the target is debian, why not use the /lib/cryptsetup/askpass utility from the cryptsetup-bin package? This utility already has the functionality of simultaneously asking for the password on the console, plymouth, systemd and a fifo, which can be used from a dropbear-ssh session with cryptroot-unlock. Thereby we always try to ask on the console and never accidently lock-out an user (who may have installed dropbear-initramfs but not set up an sshkey for dropbear yet)

Unfortunately this is not a normal login, its an SSH session, hence tty misses graphical features. Also, zfs script already uses systemd-ask-password utility for non-SSH unlock features, hence I'ce decided to reuse it to minimize changes to initramfs hooks.

[terem42]

Tiny update on Dropbear detection mechanism. As per advices given, some Debian derivatives do not have /var/run folder present inside initramfs, so I've changed dropbear process detection to classic ps | grep approach.

[sam-lunt]

I feel like this over-complicates the process. You are installing the client script (zfsunlock) using an initramfs hook, but that assumes too much about the client (that it's building using initramfs-tools and has zfs installed).

Other setups (ArchLinux's mkinitcpio hook for unlocking zfs remotely, for example) simply wait for the user to ssh in and manually unlock the rootfs. That seems easier, since it allows for any client with the appropriate key to login and unlock the filesystem. Your setup requires the client to install the zfsunlock script, since you use specially created FIFOs in order to send the data.

I think a simpler solution would be to support a method of configuring the initramfs such that it confirms there is an ssh daemon running and then simply waits for a client to login and unlock any drives that it doesn't have keys for.

[terem42]

I feel like this over-complicates the process. You are installing the client script (zfsunlock) using an initramfs hook, but that assumes too much about the client (that it's building using initramfs-tools and has zfs installed).

Other setups (ArchLinux's mkinitcpio hook for unlocking zfs remotely, for example) simply wait for the user to ssh in and manually unlock the rootfs. That seems easier, since it allows for any client with the appropriate key to login and unlock the filesystem. Your setup requires the client to install the zfsunlock script, since you use specially created FIFOs in order to send the data.

I think a simpler solution would be to support a method of configuring the initramfs such that it confirms there is an ssh daemon running and then simply waits for a client to login and unlock any drives that it doesn't have keys for.

This approach assumes user is prompted with password login with option to cancel it, it needed.
Provided correct version of dropbear-initramfs package (version 2018 and later, previous have bugs) is installed and user has the same set of host keys shared between ssh and initramfs, no special procedures or separate setup (like the one spread across numerous incompetent "guides" offering separate port for initramfs SSH prompt etc.) or other special knowledge or learning from user side is required whatsoever. Manual unlock requires user to know what command to enter, my approach is more "end user friendly". For examples of proper setup see test setup script I've developed for Ubuntu 18 and Debian 10.
For non-encrypted root pool, the password prompt will be silently ignored, even if dropbear-initramfs package is installed. If user wants to execute service operations, he can always cancel the password prompt and run the procedures manually.

[sam-lunt]

I feel like this over-complicates the process. You are installing the client script (zfsunlock) using an initramfs hook, but that assumes too much about the client (that it's building using initramfs-tools and has zfs installed).
Other setups (ArchLinux's mkinitcpio hook for unlocking zfs remotely, for example) simply wait for the user to ssh in and manually unlock the rootfs. That seems easier, since it allows for any client with the appropriate key to login and unlock the filesystem. Your setup requires the client to install the zfsunlock script, since you use specially created FIFOs in order to send the data.
I think a simpler solution would be to support a method of configuring the initramfs such that it confirms there is an ssh daemon running and then simply waits for a client to login and unlock any drives that it doesn't have keys for.

This approach assumes user is prompted with password login with option to cancel it, it needed.
Provided correct version of dropbear-initramfs package is installed and user has the same set of host keys shared between ssh and initramfs, no special procedures or separate setup (like the one spread across numerous incompetent "guides" offering separate port for initramfs SSH prompt etc.) or other special knowledge or learning from user side is required whatsoever. Manual unlock requires user to know what command to enter, my approach is more "end user friendly". For examples of proper setup see test setup script I've developed for Ubuntu 18 and Debian 10.
For non-encrypted root pool, the password prompt will be silently ignored, even if dropbear-initramfs package is installed. If user wants to execute service operations, he can always cancel the password prompt and run the procedures manually.

You're right, I misunderstood. The zfsunlock script is copied to the bin directory and that that command is added to the end of the root user's ~/.profile file. That still seems a little fragile (e.g. it's making assumptions about the contents of PATH, it's assuming that no one else changes ~/.profile, it's assuming that if someone ssh-es in during boot it's in order to unlock the drives).

It's also not robust to the situation where multiple filesystems need to be unlocked (i.e. when ZFS_INITRD_ADDITIONAL_DATASETS contains at least one drive that needs to be unlocked), since it uses the same /run/zfsunlockcomplete file no matter what dataset it's unlocking.

I think it's safe to assume that if users can understand how to install zfs on root, configure the initramfs tools, and ssh into a system running from its initramfs, then they can also learn to run the zfs load-key command. I think this adds a lot of complexity for a minor gain in convenience. Personally, I would rather it support the same interface that the Arch Linux mkinitcpio hook does (which just waits for the user to ssh in and manually load the key). That's just my opinion as a user though.

[terem42]

It's also not robust to the situation where multiple filesystems need to be unlocked (i.e. when ZFS_INITRD_ADDITIONAL_DATASETS contains at least one drive that needs to be unlocked), since it uses the same /run/zfsunlockcomplete file no matter what dataset it's unlocking.

Thanks for the error noted, I will move /run/zfsunlockcomplete after the cycle for unlocking all the pools has been completed. As for the unlocking multiple pools, so far the script assumes only the root pool requires decryption. I'll extend the script so it will ask in cycle for several keys if there is more than one encrypted pool requiring key present.

[terem42]

New commit version now can recognize and prompt for several encrypted pools password.
Each password prompt will show unique pool name to let user distinguish one pool from another.
If there are no encrypted pools, no prompt will appear. After all pools are unlocked, the boot sequence is resumed, dropbear process and subprocesses will be killed by init-bottom/dropbear script, the user is requested to reconnect to main rootfs ssh daemon.
To break out of password prompt inside initramfs user can simply press Ctrl-C, then later, enter zfsunlock command to repeat the password entry. To delay password brufeforce attack, after 3 consecutive password errors against any encrypted pool the system will automatically reboot.

[terem42]

You're right, I misunderstood. The zfsunlock script is copied to the bin directory and that that command is added to the end of the root user's ~/.profile file. That still seems a little fragile (e.g. it's making assumptions about the contents of PATH, it's assuming that no one else changes ~/.profile, it's assuming that if someone ssh-es in during boot it's in order to unlock the drives).

in new version of the commit zfsunlock now will work even if the user cancel the prompt, go to shell and run zfsunlock again, so even if the profile has been accidentally overwritten by some package, the user can run zfsunlock manually.

[rlaager]

Just as an idea: if the target is debian, why not use the /lib/cryptsetup/askpass utility from the cryptsetup-bin package? This utility already has the functionality of simultaneously asking for the password on the console, plymouth, systemd and a fifo, which can be used from a dropbear-ssh session with cryptroot-unlock. Thereby we always try to ask on the console and never accidently lock-out an user (who may have installed dropbear-initramfs but not set up an sshkey for dropbear yet)

Unfortunately this is not a normal login, its an SSH session, hence tty misses graphical features.

What does this mean?

Also, zfs script already uses systemd-ask-password utility for non-SSH unlock features, hence I'ce decided to reuse it to minimize changes to initramfs hooks.

Using systemd-ask-password for the actual prompting isn't necessarily a problem. But it's unclear to me if you've addressed the point about simultaneously prompting on the console and via dropbear. If this requires someone to SSH into the machine, that's bad. If the network is broken, for example, they will have a mess.

[terem42]

Using systemd-ask-password for the actual prompting isn't necessarily a problem. But it's unclear to me if you've addressed the point about simultaneously prompting on the console and via dropbear. If this requires someone to SSH into the machine, that's bad. If the network is broken, for example, they will have a mess.

You're right, it has to be addressed too. I'll come back with extended version of the script, covering the scenario with simultaneous prompts appearing on boot console and SSH session.

[terem42]

Updated version now simultaneously prompting on the console and via dropbear. Installing dropbear-initramfs now is optional.

[felixdoerre]

I think, my point didn't come across really well, so I'll try again. The askpass utility (from Debian's cryptsetup-bin) will by default ask on the UI (via plymouth, systemd and directly on the console) and a fifo simultaneously. If e.g the plymouth-prompt fails this has no impact on the other prompting mechanisms. The fifo is intended for supplying the password when logged in via ssh/dropbear. This mechanism seems to be exactly what is needed here to unlock a zfs dataset.

Unfortunately this is not a normal login, its an SSH session, hence tty misses graphical features.

Yes that's exactly the point of this askpass utility. With the fifo password method (https://salsa.debian.org/cryptsetup-team/cryptsetup/-/blob/b2cfa897bfcf0f5b91c5e76757b30814aca40fa7/debian/askpass.c#L460), askpass will wait for a password supplied by the cryptroot-unlock utility (https://salsa.debian.org/cryptsetup-team/cryptsetup/-/blob/b2cfa897bfcf0f5b91c5e76757b30814aca40fa7/debian/initramfs/cryptroot-unlock#L84) which is intended to be used via dropbear.

The whole askpass program seems to be much more exact and clean than writing program name substrings into files and hoping that no other process line from ps will match. Maybe cryptroot-unlock will need to be replaced with something zfs-specific but the askpass-utility can probably be (re)used without modifications. Generally handling pids manually in shell scripts (e.g. by sorting) can be pretty unstable.

[tcaputi]

Looks good to me, but I haven't had a chance to try it yet. looking forward to this addition!

[terem42]

If anyone wants to test SSH unlock using Debian 10 live image and VirtualBox you can use test install script I've made for this purpose.
Create VM (2Gb and 20Gb storage will be enough), load live system, then run this script:

wget -qO- https://raw.githubusercontent.com/andrey42/zfs-hetzner-vm/vmtest/virtualbox-debian10-zfs-setup.sh  | sudo bash -

[Sithuk]

That's probably fine then. This code is only going to show up in newer releases anyway. If it gets backported somewhere (or the user builds from source), it's easy enough to fix.

So if I wanted to install patch this onto my Ubuntu 20.04 machine (building from source), what would I need to do other than just make pkg?

You can use my Debian repo, it does contain ZFS Debian packages with ZSTD patches and unlock ZFS patches installed. I've used original Debian team repository as a base for packaging topology, so my packages can be installed on any Debian derivatives.

Also, if you clone the repo (follow readme instructions) and add your own signing keys, then you can add/remove any patches you wish on your own, there is a Github Action workflow, that pulls openzfs/zfs master branch, applies patches, then builds, tests and creates complete set of ZFS Debian packages saved as Gihhub pages content, so there is no need for any PPA or additional testing infrastructure whatsoever.

Hi @Andrey42 are you certain your repo contains this patch #10027? I installed ubuntu 20.04 and dropbear-initramfs, but entering "zfsunlock" did nothing when I logged into dropbear. I couldn't find "zfsunlock" on the system either.

[edit: @Andrey42 I've re-updated from your repo and this patch is now included. Thank you.]

[tcaputi]

@Andrey42 I just tried installing this on my Ubuntu 20.04 personal server. I got this error:

root@roy-ubuntu:~# ssh [email protected]


BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu6) built-in shell (ash)
Enter 'help' for a list of built-in commands.

# zfsunlock

Unlocking encrypted ZFS filesystems...
Enter the password or press Ctrl-C to exit.

Encrypted ZFS password for rpool: ************
Password for rpool accepted.
ps: invalid option -- 'a'
/usr/bin/zfsunlock: line 28: head: not found
BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu6) multi-call binary.

Usage: ps

Show list of processes

        w       Wide output
        l       Long output
#

The error comes from this line in zfsunlock:

zfs_console_askpwd_pid=$(ps a -o pid= -o args | grep -v grep | grep "$zfs_console_askpwd_cmd" | cut -d ' ' -f3 | sort -n | head -n1)

It appears that busybox's ps does not have take an a parameter and head doesn't exist in busybox at all. This could probably be resolved by installing an environment other than busybox into the initramfs, but it is the default and the directions in the README don't indicate that I need to change anything.

Any thoughts about a good way to fix this?

EDIT: I ran the remaining commands manually after getting the PID myself and the rest seems to work well, which is good news. :)

[rlaager]

So this is interesting. On Ubuntu, if you have the busybox-static package installed, you get a busybox that supports ps a and head. If you only have busybox-initramfs, you don't. This is reproducible on a running system like this:

/usr/lib/initramfs-tools/bin/busybox ps a -o pid= -o args
vs
/bin/busybox ps a -o pid= -o args

Debian, at least on Buster, does not have a busybox-initramfs. It has a busybox package that supports both ps a and head.

@tcaputi / @Andrey42 Does this work: ps | grep -v grep | grep "$zfs_console_askpwd_cmd" | cut -d' ' -f1 | sort -n | tr "\n" " " | cut -d" " -f1

[tcaputi]

Testing right now....

[tcaputi]

@rlaager It looks like cut isn't working the way you'd like it...

# ps | grep -v grep | grep "load-key" | cut -d' ' -f1

# ps | grep -v grep | grep "load-key"
  825 root      8952 S    /sbin/zfs load-key rpool

[tcaputi]

awk exists (at least in my busybox). So this seems to work:

# ps | grep -v grep | grep "load-key" | awk '{ print $1 }'
825
# ps | grep -v grep | grep "load-key" | awk '{ print $1 }' | sort -n | tr "\n" " " | cut -f1
825
#

[rlaager]

If you have awk, then this should work:
ps | grep -v grep | awk "/$zfs_console_askpwd_cmd/ { print \$1 }" | sort -n | awk '{print}'
or this:
ps | awk '!'"/grep/ && /$zfs_console_askpwd_cmd/ { print \$1 }" | sort -n | awk '{print}'

[tcaputi]

I'm not sure what the point of the last awk is in either example, but I'm guessing thats just because I'm not very good at awk :P

[rlaager]

It was supposed to replace the head. But it needs to be awk '{print;exit}'. That said, busybox ps seems to be outputting the pids sorted anyway. I'm not sure if we can rely on that. If we can, ps | awk '!'"/grep/ && /$zfs_console_askpwd_cmd/ { print \$1; exit }" is fine.

[tcaputi]

I'm not sure if that will work with normal ps...

root@roy-ubuntu:~# ps aux | awk '!'"/grep/ && /sleep/ { print \$1; exit }"
root

We might need to try to detect what kind of ps we're running.

[rlaager]

It's not meant to work with normal ps. This is specifically for the initramfs environment, right? Does that ever end up with GNU ps?

[rlaager]

$ sleep 30 &
[2] 14096
$ sleep 30 &
[3] 14157
$ zfs_console_askpwd_cmd=sleep ; /usr/lib/initramfs-tools/bin/busybox ps | /usr/lib/initramfs-tools/bin/busybox awk '!'"/grep/ && /$zfs_console_askpwd_cmd/ { print \$1; exit }"
14096
$ zfs_console_askpwd_cmd=sleep ; /bin/busybox ps | /bin/busybox awk '!'"/grep/ && /$zfs_console_askpwd_cmd/ { print \$1; exit }"
14096

[tcaputi]

@rlaager Right. I'm just saying that if anyone (for whatever reason) has a real version of ps in their initramfs it won't work. Probably a very small percentage of users though.

[terem42]

@rlaager Right. I'm just saying that if anyone (for whatever reason) has a real version of ps in their initramfs it won't work. Probably a very small percentage of users though.

Thanks for noting it. Better to cover all possible angles.
I was withessing many cases where dropbear and busybox was installed manually via dpkg, breaking dependencies, when people use low-quality instructions floating around on installing dropbear manually.

[innerand]

Hi,

as the documentation is still todo, could somebody give me a hint on how to use this with Debian Buster?
I mean is this a package, is it already available in repositories or should the files be copied manually (which?)?
Unfortunately most links from the discussions above are dead.

Thanks a lot!
innerand

[terem42]

The code has been merged in master repo, so you can cherrypick then add patch from this PR to the source code, then compile the ZFS package, for example latest version 0.8.5. The actual code will be part of 2.0 package or earlier as per users request.
To use it, all you need is to install dropbear-initramfs package and add your authorized keys to the dropbear folder.

 cp /home/myuser/.ssh/authorized_keys /etc/dropbear-initramfs/authorized_keys

Then on every encrypted system boot, the system will open SSH connection and wait for keys to entered via the SSH by human user or via script, by calling zfsunlock command on SSH prompt. At the same time, if you enter ZFS password into console, SSH server is quitetly shut down, and boot resumes in usual manner, i.e. entering ZFS encryption password via SSH and console can safely coexists.

This solution is extremely usable in cases where you cannot trust credentials storage of your datacenter vendor (AWS, Azure, Google etc.) and need to rotate ZFS root filesystem encryption keys without any hosting vendor involvement, automatically excluding machines from the distributed pool in case the system is compromised, or there is a threat it might be.

[Fmstrat]

@terem42 I'm not sure how you'd like to receive user requests, but I would very much like to see this in a current version vs waiting for 2.0. Not as much as raidz expansion, but it's up there on my list to replace my current mdadm workflow.

[terem42]

Well, I'm not using official distributions anyway. Normally I take master and apply my own set of patches and PRs.
If you use Ubuntu 18.4 or Debian 10 for Prod systems, you can fork my repo https://github.com/terem42/zfs-debian
or use it as a unofficial Debian repo, it does include this commit as well as others. The workflow incluses full-blown set tests on each commit, so preliminary zfs packaje stability is assured.

[n0rc]

ZFS 2.0 is also already available in Debian 10 backports.