Raw Send Feature (Encrypted + compressed)

[tcaputi]

With the coming encryption changes (see #494), I think we need to start looking at other use cases users could have for encryption. The biggest one I see is the ability to take encrypted backups with zfs send to an untrusted machine. In addition, it would be nice for a system administrator to be able to take backups of encrypted datasets without needing to load encryption keys. With such a feature, ZFS could be a true platform for end-to-end encryption.

The fundamental issue as far as implementation goes is that all of the ZFS send code is designed to go through the ARC and work with arc_buf_t's. The ARC currently stores data decompressed and decrypted, so that it can be used on request. That said, there are WIP changes coming that will enable the ARC to store compressed data (see the WIP at #4768) and further changes that will allow users to send data from the compressed ARC without decompressing (still not upstreamed to OpenZFS yet). However, this work doesn't really make sense for encryption because, fundamentally, compressed data can always be decompressed, but encrypted data can only be decrypted if the encryption key is loaded. Furthermore, from an efficiency standpoint, it makes sense to want compressed data in the ARC so that we can fit more data in there, while there is no real reason to want encrypted data in the ARC.

Therefore, I would like to put forth the idea of a raw send that reads data from disk instead of the ARC. The basic implementation here would be to abstract the arc_read() calls on the send side so that they can also use zio_read() with ZIO_FLAG_RAW. The receive side would simply need to write this data exactly as is to the disk.

On top of that, we would need a way to send the DSL Keychain object, which currently resides in the MOS. I think this could be included as a new DRR_* enum type that would simply be handled slightly differently so that it is written to the MOS instead of the dataset. It is also possible, since the encryption patch isn't upstreamed yet, that we could move the DSL Keychain into its dataset, but I suspect that might be more trouble than it is worth (although I will look into it).

I am going to start working on a proof of concept of this soon and will submit it as a WIP when I feel it is close enough to needing review. Please let me know if anyone has any suggestions or issues.

[GregorKopka]

Having decrypted data in the ARC while the key encryption key is unavailable should not happed since it opens the possibility to leak cleartext.

So I see reason behind the idea to cache the block encrypted, only decrypt it on demand and drop the decrypted block as soon as the consumer releases it, similar to handling compressed data in #4768. In case it is decided to keep frequently accessed data unencrypted then the implementation must make sure that the moment the encryption key vanishes all decrypted data related to it is wiped and removed from ARC.

[tcaputi]

Having decrypted data in the ARC while the key encryption key is unavailable should not happed since it opens the possibility to leak cleartext.

Could you explain this? At some point the data needs to be decrypted in memory anyway in order to be returned to the user. The current encryption implementation of ZFS encryption (not considering encrypted sends) does not drop data from memory once the key is removed.

[GregorKopka]

Maybe worded a bit wrong, trying again:

In case the dataset gets locked (by unmounting, for example) any cleartext associated with it needs to be wiped from the ARC and cleartext needs to be unable to ever make it into L2ARC.

( If that shouldn't happen then cleartext from the encrypted dataset could be held in RAM (or even on disk) in the system for a very long time, which would be bad since encryption should protect the whole dataset against access from everyone without the key. So the moment the key is unloaded by zfs key -u the cleartext needs to go away for the encryption feature to be secure. Maybe I should have added this to the comments on #4329 instead )

Thus my comment to treat encrypted blocks like with the upcoming compressed ARC feature: cache them, even cache unencrypted ones when frequently used but forcefully remove everything cleartext when the key gets unloaded - preferably by overwriting the buffers to make sure it is gone. So there needs to be some way to distinguish in-ARC between cached encrypted on-disk blocks and decrypted data delivered to consumers, which for encrypted sends could be used to pull blocks already in ARC from it to avoid the extra roundtrip through the disks.

My feeling is that should you bypass ARC for encrypted zfs send then performance will suffer, especially on the (at least on many of my servers primary) use case of repeatedly incremental send of in-use datasets for near-realtime backup purposes, since normal operation would feel the additional impact of zfs send consuming disk bandwith to read blocks that already are in ARC - which could be avoided by structuring the encryption feature similar to compressed ARC and compressed send/receive. Didn't dig that far into your code, so an advanced sorry should I have just wasted your time with this.

[tcaputi]

In case the dataset gets locked (by unmounting, for example) any cleartext associated with it needs to be wiped from the ARC and cleartext needs to be unable to ever make it into L2ARC.

The current implementation does its best to clear keys out from memory as soon as they are unloaded from the system. It also ensures that anything going to the L2ARC is reencrypted with the L2ARC key (which only exists in memory and is recreated after each reboot). However, it does not immediately zero out ARC buffers associated with it.

The reasoning for this is that we cannot protect against that kind of attack anyway. The plaintext will still exist, at least in the page cache or application memory, no matter how much we zero out from the ZFS kernel module. I personally don't see a way around this because at some point, the data must exist in plaintext if it is to be used. I think the best we can do is simply to make sure that we are zeroing out data when we free it so that it cannot possibly be returned to the next user.

[tcaputi]

I have not had much success finding people talking about this subject online, but I did (quickly) look at dm-crypt from the Linux kernel and they do not appear to make special effort to zero out in-memory buffers once they are done with them.

[sneak]

However, it does not immediately zero out ARC buffers associated with it.

The reasoning for this is that we cannot protect against that kind of attack anyway. The plaintext will still exist, at least in the page cache or application memory, no matter how much we zero out from the ZFS kernel module.

Having sensitive data zero-or-one places in RAM is better than having sensitive data in one-or-two places in RAM (consider cold boot attacks on RAM contents). FS encryption is, in many respects, a life-or-death matter, and a belt and suspenders approach (zeroing keys and zeroing cleartext in ARC) is warranted.

Certainly, as an admin, if I unmounted a device or unloaded keys, it would significantly violate the principle of least surprise if zfs were to retain cleartext in system memory that which it could well have zeroed. Simply because applications or the page cache might keep the data around doesn't mean zfs should as well. Subsequent users with root shouldn't be able to read cleartext (via /dev/mem or /dev/kmem) once a device has been unmounted/keys unloaded - and if the page cache is keeping it around, that's a second bug, not a reason to not practice good cleartext hygiene.

It is also an additional layer of protection against bugs related to the L2ARC encryption code (which I have not yet read, so forgive me if this isn't a valid point).

[tcaputi]

@sneak
I am not sure that I agree with you.

Certainly, as an admin, if I unmounted a device or unloaded keys, it would significantly violate the principle of least surprise if zfs were to retain cleartext in system memory that it could well have zeroed.

I don't think this is true, although perhaps we should be careful about the documentation surrounding this feature. Cryptographers divide data protection into 3 categories: data in transit, data at rest, and data in use. Each of these cases makes different assumptions regarding what is safe. Encryption in transit, for instance, assumes that the sending user is not already compromised, thus the TLS standard does not have anything to say about how data to be transmitted should be read from the filesystem. Encryption in use doesn't provide any protection against attackers who can read the files generated from the encrypted memory. The same logic applies to data at rest. My patch does not do any certificate trust verification (as TLS does) and it does not make any attempts to protect decrypted data in RAM. The goal of #494 is to provide data protection at rest.

Having sensitive data zero-or-one places in RAM is better than having sensitive data in one-or-two places in RAM (consider cold boot attacks on RAM contents). ...if the page cache is keeping it around, that's a second bug, not a reason to not practice good cleartext hygiene.

I see your point here. However I think it is a bad idea to make a security promise that I can't really deliver on. If I added this feature would the docs read something along the lines of "The zfs code will protect your data in memory in the event of a RAM attack, but your OS will inevitably leak the information anyway"? I also think calling it a bug (either in the OS or in zfs) is a bit of an exaggeration. It is simply a design limitation that could be dealt with in a later patch when the required hardware / OS infrastructure exists (such a patch would definitely require the support of both).

Does all of this make #494 worthless? I don't think so. Cold boot attacks are hard and unreliable to do and most OS's don't expose a /dev/mem or /dev/kmem anymore for exactly these reasons. Security is often (as you said) a "life-or-death" concept in terms of "did the attacker get your data or not?", but in reality, it is also a matter of trade-offs. Even with all the security in the world, at some point it becomes feasible for an attacker to simply brute force the encryption key. The existence of an attack like that does not make encryption worthless.

It is also an additional layer of protection against bugs related to the L2ARC encryption code (which I have not yet read, so forgive me if this isn't a valid point).

This is true, but it comes with trade-offs as well. Off the top of my head, the ARC would need to be significantly reimplemented so that encrypted buffers could keep track of the number of mounted, encrypted datasets referencing them. The current ARC simply is not aware of what dataset a buffer might belong to, since it might belong to many through the existence of snapshots / clones / dedup. In addition, there would be an extra cost associated with having to zero out (potentially) hundreds of gigabytes of memory and having to re-decrypt data every time it is used. That is a lot of overhead for trying to prevent a bug.

[tcaputi]

@kpande Yes I do. I'm actually done with the prototype and am working on getting all the testing done so that I can merge it into #5769. The work is in the raw_sends branch on my fork if you'd like to poke around, but keep in mind its not done quite yet. I'm hoping sometime this week.

[tcaputi]

A working implementation of raw sends has been merged into #5769. Some additional security features will be coming in the near future, but I will close this issue now since comments should probably left on the main PR page.