Filesystem created via zfs send --raw between encrypted roots fails to mount

[stewartadam]

System information

Type	Version/Name
Distribution Name	Fedora
Distribution Version	26
Linux Kernel	4.12.9-300.fc26.x86_64
Architecture	x86_64
ZFS Version	0.7.1-1 (with encryption patches)
SPL Version	0.7.1-1

I've installed SPL from the upstream f26 repo and compiled 0.7.1 from source, patching in:

• Add libtpool (thread pools) (46364cb)
• Send / Recv Fixes following b52563 (9b84076)
• Native Encryption for ZFS on Linux (b525630)

Transferring a filesystem between two encryption roots with zfs send --raw succeeds, but cannot be mounted.

rm -f /root/src.img /root/replica.img

truncate -s 100M /root/src.img
truncate -s 100M /root/replica.img

zpool create src /root/src.img
zpool create replica /root/replica.img
zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt src/encrypted
zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt replica/encrypted

zfs create src/encrypted/myfiles
dd if=/dev/urandom of=/src/encrypted/myfiles/random.bin bs=1M count=30
zfs snapshot -r src@migration

zfs send -cpv --raw src/encrypted/myfiles@migration | zfs recv -s replica/encrypted/myfiles

Results in:

full send of src/encrypted/myfiles@migration estimated size is 30.7M
total estimated size is 30.7M
TIME        SENT   SNAPSHOT
filesystem 'replica/encrypted/myfiles' can not be mounted: Permission denied
cannot mount 'replica/encrypted/myfiles': Invalid argument

Attempting to import all keys fails to request keys for the transferred filesystem:

zpool import -d /root replica
zfs load-key -a
Enter passphrase for 'replica/encrypted': 
1 / 1 key(s) successfully loaded

filesystem 'replica/encrypted/myfiles' can not be mounted: Permission denied
cannot mount 'replica/encrypted/myfiles': Invalid argument

In this scenario I would expect the raw transfer to result in a new encryption root and then zfs load-key would to prompt for both replica/encrypted and replica/encrypted/myfiles, but that's clearly not the case:

zfs get -r encryptionroot replica/encrypted
NAME                                 PROPERTY        VALUE              SOURCE
replica/encrypted                    encryptionroot  replica/encrypted  -
replica/encrypted/myfiles            encryptionroot  replica/encrypted  -
replica/encrypted/myfiles@migration  encryptionroot  replica/encrypted  -

We should clarify (a) if nested encryption roots are supported and (b) validate raw transfer between encrypted filesystems.

[stewartadam]

Furthermore, sending a filesystem that is a grandchild of an encryption root does not result in any errors but also cannot be mounted:

rm -f /root/src.img /root/replica.img

truncate -s 100M /root/src.img
truncate -s 100M /root/replica.img

zpool create src /root/src.img
zpool create replica /root/replica.img
zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt src/encrypted
zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt replica/encrypted

zfs create src/encrypted/parent
zfs create replica/encrypted/parent

zfs create src/encrypted/parent/myfiles
dd if=/dev/urandom of=/src/encrypted/parent/myfiles/random.bin bs=1M count=30
zfs snapshot -r src@migration

zfs send -cpv --raw src/encrypted/parent/myfiles@migration | zfs recv -s replica/encrypted/parent/myfiles

Results in:

30+0 records in
30+0 records out
31457280 bytes (31 MB, 30 MiB) copied, 1.5498 s, 20.3 MB/s
full send of src/encrypted/parent/myfiles@migration estimated size is 30.7M
total estimated size is 30.7M
TIME        SENT   SNAPSHOT

The filesystem is not mounted:

# find /replica
/replica
/replica/encrypted
/replica/encrypted/parent
# zfs mount -a
cannot mount '/replica/encrypted/parent/myfiles': encryption key not loaded

So let's load the key with zfs load-key -a -- that simply returns with no output. So let's be explicit:

# zfs load-key replica/encrypted/parent/myfiles
Key load error: Keys must be loaded for encryption root of 'replica/encrypted/parent/myfiles' (replica/encrypted/parent).
# zfs load-key replica/encrypted/parent/myfiles
zfs load-key replica/encrypted/parent
Key load error: Keys must be loaded for encryption root of 'replica/encrypted/parent' (replica/encrypted).
# zfs load-key replica/encrypted
Key load error: Key already loaded for 'replica/encrypted'.

However, this time it does create a new encryptionroot as I would have expected:

NAME                                        PROPERTY        VALUE                     SOURCE
replica                                     encryptionroot  -                         -
replica/encrypted                           encryptionroot  replica/encrypted         -
replica/encrypted/parent                    encryptionroot  replica/encrypted         -
replica/encrypted/parent/myfiles            encryptionroot  replica/encrypted/parent  -
replica/encrypted/parent/myfiles@migration  encryptionroot  replica/encrypted/parent  -

[stewartadam]

Also; please advise if I will need to destroy these filesystems and re-transfer, or if we can make them mountable after some patches to fix these issues.

[behlendorf]

@stewartadam there are some additional fixes/cleanup in #6595 pending review but I'm not sure any of them address your issue. @tcaputi can you please look at this issue and advise.

[tcaputi]

Ive figured out the issue here. Userspace is calling the hidden FORCE INHERIT command when it shouldn't be. Specifically, top-level datasets with properties should never be issued this command. I will incorporate the fix into #6595 tomorrow morning.

@stewartadam Thank you very much for finding this. The easiest way to resolve this issue will be to resend the data. If this data is particularly important to you and replacing it would be extremely difficult, I can probably provide you some code for a small binary that will call the FORCE NEW KEY command (the opposite of FORCE INHERIT) which should allow you to mount the dataset again. Send me an email if that is something you would like to do. Otherwise I should have this integrated tomorrow.

[tcaputi]

I have implemented the fix for this and should have it added to #6595 today.

For the documentation, the man page currently says this about nested encryption roots:

Any descendant datasets will inherit their encryption key from the encryption root, meaning that loading, unloading, or changing the key for the encryption root will implicitly do the same for all inheriting datasets. If this inheritance is not desired, simply supply a new encryption and keyformat when creating the child dataset or use zfs change-key to break the relationship.

Is this good enough, or is there something else that needs to be clarified?

As far as adding validation goes, this is actually a bit harder than it seems. We do have validation for most basic kinds of raw send / recv in the zfs test suite. However, there are a ton of variations here and it isnt currently feasible to test them all. I can add testing for this particular issue now, but long-term I think we need some way to exhaustively check all of the options when it comes to send / recv.

[stewartadam]

Thanks @tcaputi for the fast fix! I can re-transfer the data from backup worst case, but it's ~7TB so would take a long while - will shoot you an email.

Re: documentation, I think we should further clarify this sentence:

If this inheritance is not desired, simply supply a new encryption and keyformat when creating the child dataset or use zfs change-key to break the relationship.

That makes it sound like I need to change keyformat in order to force a new encryption root -- what if I would like both the parent and child root to both use a passphrase keyformat (e.g. this situation could feasibly occur if I zfs send --raw between two passphrase-encrypted roots per above)?

[tcaputi]

What if we just add a sentence afterwards:

The new keyformat may match that of the parent.

[stewartadam]

May I suggest:

If this inheritance is not desired, simply supply a new set of encryption settigns (e.g. keyformat) when creating the child dataset or use zfs change-key to break the relationship. The new keyformat may match that of the parent.

It would be clearer too if keyformat, encryption and keylocation showed the encryption root it inherited from in zfs list.

[tcaputi]

If this inheritance is not desired, simply supply a new set of encryption settigns (e.g. keyformat) when creating the child dataset or use zfs change-key to break the relationship

This wording isn't exactly correct. For instance, you can specify a new value for encryption which will change the cipher suite, but if you don't change the keyformat you won't get a new encryption root. This can be useful in cases where you want to use the same key for a new dataset, but a stronger (and more computationally expensive) encryption algorithm.

It would be clearer too if keyformat, encryption and keylocation showed the encryption root it inherited from in zfs list.

Matt and I talked for a long time about this. The problem is that these values don't work like regular inherited values. encryption inherits from the parent only at creation time. After that it is set in stone, similar to the case sensitivity setting. You can't call zfs inherit or zfs set on either of these properties.

keylocation and keyformat also don't really work with standard inheritance. For instance, a clone must always share an encryption key with its origin, regardless of its parent's settings. I attempted at one point to alter the inheritance code so that properties could inherit from an origin before a parent but this quickly became a huge mess of edge cases, not to mention the fact that zfs doesn't have limits on how many clones-of-clones you can have, which would mean that looking up properties inherited via an origin could become (theoretically) infinitely slow.

Because of all these complications, Matt and I eventually settled on the encryptionroot property which can be thought of as the "inheriting parent" of a given dataset, but which semantically works differently than standard inheritance. We were then able to make it so the keylocation and keyformat properties were simple non-inheriting properties that only applied to encryption roots, which makes sense because these values only really have a meaning for the encryption roots. I hope that clears this up a bit.

[stewartadam]

Gotcha, that makes sense. For the original man page update, I think avoiding words that imply the need for a different/change setting would be clearer while remaining factually accurate:

If this inheritance is not desired, simply supply an encryption and keyformat when creating the child dataset or use zfs change-key to break the relationship to create a new encryption root. Note that the child's keyformat may match that of the parent while still creating a new encryption root, and that changing the encryption property alone does not create a new encryption root; this would simply use a different cipher using the same encryption key as its encryption root.

It would be good to mention something about the behavior of encryption properties with regards to inheritance as well, perhaps something like this (but my wording maybe could be better?) to the existing sentence under the Encryption section:

[...]. Encryption root inheritence can be tracked via the read-only encryptionroot property. Note that some encryption-related properties (keyformat, keylocation and keystatus) do not inherit like normal ZFS properties and instead directly return the current value for the filesystem, which is sometimes dictated by the encryption root.

[tcaputi]

How is this wording for the entire paragraph:

Creating an encrypted dataset requires specifying the encryption and keyformat properties at creation time, along with an optional keylocation and pbkdf2iters. After entering an encryption key, the created dataset will become an encryption root. Any descendant datasets will inherit their encryption key from the encryption root by default, meaning that loading, unloading, or changing the key for the encryption root will implicitly do the same for all inheriting datasets. If this inheritance is not desired, simply supply a keyformat when creating the child dataset or use zfs change-key to break an existing relationship, creating a new encryption root on the child. Note that the child's keyformat may match that of the parent while still creating a new encryption root, and that changing the encryption property alone does not create a new encryption root; this would simply use a different cipher suite with the same key as its encryption root. The one exception is that clones will always use their origin's encryption key. As a result of this exception, some encryption-related properties (namely keystatus, keyformat, keylocation, and pbkdf2iters) do not inherit like other ZFS properties and instead use the value determined by their encryption root. Encryption root inheritance can be tracked via the read-only encryptionroot property.

[stewartadam]

+1 very clear!

[ikozhukhov]

hi, what is status of this update?

[tcaputi]

The man page update was merged along with #6595. Closing this issue. Thanks for finding it @ikozhukhov

[cumafo]

Hello. I'm using /lib/modules/4.18.0-17-generic/extra/zfs/zfs/zfs.ko version: 0.8.3-1 and experience still the same issue. In my case the transfer is also quite big (10TB).
Is there anything to do? @tcaputi Does the binary you talked about exist?

[tcaputi]

@cumafo All of this was merged 2 years ago and should be in any recent version of zfs. If you are experiencing an issue similar to this it would probably be best if we treated it as a new bug. Can you create a new issue with all of your symptoms / steps to reproduce?

[cumafo]

Hello and thanks for quick attention. Since things are a bit urgent in the middle of an big migration, I had to do send/receive of the individual file systems and that is ongoing. As soon as I have the migration in place (end of next week), I will try to reproduce with a smaller dataset and document every step. Regards Curt. From: Tom Caputi <[email protected]> Reply to: openzfs/zfs <[email protected]> Date: Saturday, 2 May 2020 at 12:21 To: openzfs/zfs <[email protected]> Cc: Curt Lestrup <[email protected]>, Mention <[email protected]> Subject: Re: [openzfs/zfs] Filesystem created via zfs send --raw between encrypted roots fails to mount (#6624) @cumafo<https://github.com/cumafo> All of this was merged 2 years ago and should be in any recent version of zfs. If you are experiencing an issue similar to this it would probably be best if we treated it as a new bug. Can you create a new issue with all of your symptoms / steps to reproduce? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#6624 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACXETTWDEKOSE4OS74HDMLDRPPXY5ANCNFSM4D2IZGBA>.