Fix raw sends on encrypted datasets when copying back snapshots #11221

gamanakis · 2020-11-19T19:21:41Z

Motivation and Context

When sending raw encrypted datasets the user space accounting is present
when it's not expected to be. This leads to the subsequent mount failure
due a checksum error when verifying the local mac.

Closes #10523.

Description

Fix this by clearing the OBJSET_FLAG_USERACCOUNTING_COMPLETE and reset
the local mac. This allows the user accounting to be correctly updated
on first mount using the normal upgrade process.

How Has This Been Tested?

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the OpenZFS code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
I have run the ZFS Test Suite with this change applied.
All commit messages are properly formatted and contain Signed-off-by.

gamanakis · 2020-11-19T19:27:19Z

@tcaputi would you mind taking a look at this?

behlendorf

Thanks! Let's also add a new test to the userquota group for this. The reproducer in #10523 (comment) is a great place to start, but it should really also verify the resulting object accounting is correct. The existing userspace_encrypted.ksh test does a similar test but doesn't check sending it back to the source.

gamanakis · 2020-11-20T20:50:14Z

45085ad: added a test. The object accounting seems to be correct between source and target. In 2/10 tests I see a discrepancy of 1024 bytes for a 20MB file. Is this to be expected? Right now the test rounds up the object accounting to MB (instead of bytes).

behlendorf · 2020-11-20T21:03:40Z

Yes, some variance isn't entirely unexpected.

tcaputi

This generally looks good. Thanks a lot for picking this up. I kind of dropped the ball on fixing this. My apologies.

module/os/freebsd/zfs/zio_crypt.c

module/os/linux/zfs/zio_crypt.c

module/zfs/dsl_crypt.c

gamanakis · 2020-11-21T13:26:25Z

955f99b: Fixes from @tcaputi review. Thank you!

behlendorf · 2020-11-24T18:31:42Z

@gamanakis looks like there's one more issue to investigate. With the change the zpool_import_errata3 test is consistently failing.

Tests with results other than PASS that are unexpected:
    FAIL cli_root/zpool_import/zpool_import_errata3 (expected PASS)

When sending raw encrypted datasets the user space accounting is present when it's not expected to be. This leads to the subsequent mount failure due a checksum error when verifying the local mac. Fix this by clearing the OBJSET_FLAG_USERACCOUNTING_COMPLETE and reset the local mac. This allows the user accounting to be correctly updated on first mount using the normal upgrade process. Closes openzfs#10523. Signed-off-by: George Amanakis <[email protected]>

gamanakis · 2020-11-25T11:51:36Z

I think what is happening is that in Errata3 the useraccounting_complete flag is not set but ZFS expects a non-zeroed local_mac. Was encryption implemented before user accounting? I solved this by checking the encryption version, applied in 4c5efed.

behlendorf

Thanks, checking the zk_version in this context makes sense to me since the errata was specifically checking for an intermediate version of encryption which only existed for a short while in master between major releases. @tcaputi would you mind looking over this again.

tcaputi

Thanks for making the adjustments and for fixing errata 3 handling. This looks good to me.

When sending raw encrypted datasets the user space accounting is present when it's not expected to be. This leads to the subsequent mount failure due a checksum error when verifying the local mac. Fix this by clearing the OBJSET_FLAG_USERACCOUNTING_COMPLETE and reset the local mac. This allows the user accounting to be correctly updated on first mount using the normal upgrade process. Reviewed-By: Brian Behlendorf <[email protected]> Reviewed-By: Tom Caputi <[email protected]> Signed-off-by: George Amanakis <[email protected]> Closes openzfs#10523 Closes openzfs#11221

Raw receiving a snapshot back to the originating dataset is currently impossible because of user accounting being present in the originating dataset. One solution would be resetting user accounting when raw receiving on the receiving dataset. However, to recalculate it we would have to dirty all dnodes, which may not be preferable on big datasets. Instead, we rely on the os_phys flag OBJSET_FLAG_USERACCOUNTING_COMPLETE to indicate that user accounting is incomplete when raw receiving. Thus, on the next mount of the receiving dataset the local mac protecting user accounting is zeroed out. The flag is then cleared when user accounting of the raw received snapshot is calculated. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: George Amanakis <[email protected]> Closes #12981 Closes #10523 Closes #11221 Closes #11294 Closes #12594 Issue #11300

Raw receiving a snapshot back to the originating dataset is currently impossible because of user accounting being present in the originating dataset. One solution would be resetting user accounting when raw receiving on the receiving dataset. However, to recalculate it we would have to dirty all dnodes, which may not be preferable on big datasets. Instead, we rely on the os_phys flag OBJSET_FLAG_USERACCOUNTING_COMPLETE to indicate that user accounting is incomplete when raw receiving. Thus, on the next mount of the receiving dataset the local mac protecting user accounting is zeroed out. The flag is then cleared when user accounting of the raw received snapshot is calculated. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: George Amanakis <[email protected]> Closes openzfs#12981 Closes openzfs#10523 Closes openzfs#11221 Closes openzfs#11294 Closes openzfs#12594 Issue openzfs#11300

behlendorf added the Status: Code Review Needed Ready for review and testing label Nov 19, 2020

behlendorf reviewed Nov 19, 2020

View reviewed changes

gamanakis force-pushed the raw_sends_squashed branch from bb5884a to 45085ad Compare November 20, 2020 20:43

behlendorf approved these changes Nov 20, 2020

View reviewed changes

tcaputi suggested changes Nov 20, 2020

View reviewed changes

module/os/freebsd/zfs/zio_crypt.c Show resolved Hide resolved

module/os/linux/zfs/zio_crypt.c Show resolved Hide resolved

module/zfs/dsl_crypt.c Show resolved Hide resolved

gamanakis force-pushed the raw_sends_squashed branch from 45085ad to 955f99b Compare November 21, 2020 13:24

gamanakis force-pushed the raw_sends_squashed branch from 955f99b to 4c5efed Compare November 25, 2020 11:31

behlendorf approved these changes Nov 28, 2020

View reviewed changes

behlendorf added Status: Accepted Ready to integrate (reviewed, tested) and removed Status: Code Review Needed Ready for review and testing labels Dec 1, 2020

tcaputi approved these changes Dec 4, 2020

View reviewed changes

behlendorf merged commit d1d4769 into openzfs:master Dec 4, 2020

gamanakis mentioned this pull request Jun 24, 2021

[WIP] Reset user accounting when raw receiving a snapshot #11300

Closed

13 tasks

gamanakis mentioned this pull request Sep 30, 2021

Sometimes raw send on encrypted datasets does not work when copying snapshots back #12594

Closed

gamanakis mentioned this pull request Jan 16, 2022

Introduce a flag to skip comparing the local mac when raw sending #12981

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix raw sends on encrypted datasets when copying back snapshots #11221

Fix raw sends on encrypted datasets when copying back snapshots #11221

gamanakis commented Nov 19, 2020 •

edited

Loading

gamanakis commented Nov 19, 2020

behlendorf left a comment

gamanakis commented Nov 20, 2020

behlendorf commented Nov 20, 2020

tcaputi left a comment

gamanakis commented Nov 21, 2020

behlendorf commented Nov 24, 2020

gamanakis commented Nov 25, 2020 •

edited

Loading

behlendorf left a comment

tcaputi left a comment

Fix raw sends on encrypted datasets when copying back snapshots #11221

Fix raw sends on encrypted datasets when copying back snapshots #11221

Conversation

gamanakis commented Nov 19, 2020 • edited Loading

Motivation and Context

Description

How Has This Been Tested?

Types of changes

Checklist:

gamanakis commented Nov 19, 2020

behlendorf left a comment

Choose a reason for hiding this comment

gamanakis commented Nov 20, 2020

behlendorf commented Nov 20, 2020

tcaputi left a comment

Choose a reason for hiding this comment

gamanakis commented Nov 21, 2020

behlendorf commented Nov 24, 2020

gamanakis commented Nov 25, 2020 • edited Loading

behlendorf left a comment

Choose a reason for hiding this comment

tcaputi left a comment

Choose a reason for hiding this comment

gamanakis commented Nov 19, 2020 •

edited

Loading

gamanakis commented Nov 25, 2020 •

edited

Loading