ZFS send/recv with ashift 9->12 leads to data corruption

[aither64]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	20.04
Kernel Version	5.4.0-90-generic
Architecture	x86_64
OpenZFS Version	zfs-2.1.99-528_g8ac58c3f5

Describe the problem you're observing

ZFS send/recv from pool with ashift=9 to pool with ashift=12 silently damages files on disk after a subsequent send/recv. No error is reported anywhere, zpool scrub reports no errors, but files are corrupted.

Describe how to reproduce the problem

zdb -C tank | grep ashift
                ashift: 9
zdb -C dozer | grep ashift
                ashift: 12
zfs set compression=on tank/test
zfs set compression=on dozer
... put some data to /tank/test, I'm using a container image ...
zfs snapshot tank/test@snapshot
zfs send -c -L tank/test@snapshot | zfs recv dozer/test-first
zfs send -c -L dozer/test-first@snapshot | zfs recv dozer/test-second

Now data in dozer/test-second is corrupted. I'm just looking at text files. All files are there, metadata like timestamps and file size seem to match, but the contents are garbled. Please note that the data is apparently ok after the first send/recv, but after the second it is not.

If I repeat the above steps, but remove the -c switch, the data survives.

We have tracked it to commit c634320, If I revert it, I can't reproduce the problem anymore.

[pcd1193182]

I'm looking into this, and the error is odd... when you examine the zdb outputs for a file in test-first and test-second, they appear identical. The l0 blkptrs have the same lsize, same psize, same asize, and same checksum. The datasets themselves also have the same properties, aside from the necessarily different ones like guid and objsetid. So why is the file so messed up?

[pcd1193182]

Ah, if you pass the proper flags to zdb to get the full blkpr information, it looks like in test-second, the blkptr is listed as not compressed. It looks like we're dropping the compression information when we send it on, presumably because the lsize and the psize are the same?

[pcd1193182]

The issue here is that the way we do compression detection in the send path is to compare the psize to the lsize. The fix is to have a direct check of the blockpointer and the zio flags to see if the buffer is compressed or not.

[marker5a]

Long shot here, but I've been spending time over in #12594 and in the process of debugging, stumbled into this issue which now seems like a candidate for a dataset that is not mountable. It's an encrypted dataset that was migrated from one pool to another using raw sends and there is a non-zero chance I went from a lower shift to a larger shift during the migration, thus causing corruption... at least from the looks of this issue.

Question: If I suffered from this, is it safe to assume that my data is totally garbage now? Or is there a hope of getting some/all of it back?

The odd thing when reading this issue is that the symptom is garbled data. My symptom is that I cannot even mount the dataset (I get input/output errors) to begin with, so it's not clear to me if the blocks themselves are garbled and zfs is throwing errors because it detected that, or if the i/o error stems from some other issue.

[rincebrain]

If I read the error condition right, you could try to cut a patch that would try uncompressing unconditionally, even if the blkptr doesn't think it's marked compressed...

[psy0rz]

Because we cant always 100% trust zfs, I'm trying to create an intelligent zfs-compare tool that will compare the latest common snapshots in two pools. It will shasum the actual zvols and files, instead of relying on zfs. It will also transfer a remote dataset thats encrypted and only has a local key, so that the encryption key isnt needed remotely. Does anyone want this tool as well?

[gdevenyi]

Any idea here of what versions this affects? I have a 140TB migration to do from a 9->12 pool and not super happy having to upgrade to master to get the send working properly...

[rincebrain]

2.0.7 and 2.1.2 both include ports of 3760273, so you could go either route.

[gdevenyi]

2.0.7 and 2.1.2 both include ports of 3760273, so you could go either route.

Thanks, that's helpful. Any idea of 0.8.6 is affected?

I see the issue is -c so I guess I could also avoid that.

[rincebrain]

Well, the commit the original reporter is pointing to was from September 2021, and 0.8.6 was December 2020, so one assumes not.

Amusingly, 2.0.x does not appear to have gotten the original commit, just the fix...2.1.1 had it, though, so if you were running 2.1.1 specifically, it could happen.

[lapo-luchini]

Can this be detected on the zfs recv side, after the fact?
Example given: would a zpool scrub detect this?

[psy0rz]

i think not, which is the biggest ans scariest issue.

the data is corrupt before the checksums etc are calculatuted

[rincebrain]

I could imagine, for ease of recovery purposes, making a tree that puts a recovery mechanism behind a tunable. As I understand it, the outcome of this is that you've got compressed data on disk marked uncompressed, so ZFS hands it back to you verbatim. What you could do is have a default-off codepath that tries each decompressor on anything it tries to read that's marked uncompressed, and if it succeeds, hand that back instead. (Alternately, if you cheat and tell the codepath which compressor was used, it could just forcibly try only that...) You could also try extending zdb -r to brute-force that way, or I think -R might already have such a bruteforce option. But if I'm understanding the failure mode correctly, it should be possible to extract any data that's in an unfortunate state from this.

…

On Thu, Feb 3, 2022 at 6:58 AM DatuX ***@***.***> wrote: i think not, which is the biggest ans scariest issue. the data is corrupt before the checksums etc are calculatuted — Reply to this email directly, view it on GitHub <#12762 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUI7LT2FMCYCSC2DL77NDUZJUW7ANCNFSM5H64B6YA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[ipaqmaster]

Experienced this myself last night rebuilding a desktop as ashift=12. Discovered that my laptop of default ashift=9 was able to receive my desktop's encrypted root dataset, load-key it and mount it just fine. But my desktop, rebuilt to use ashift=12 on its nvme, got the "Input/Output error" experience exactly as described here.

I tested it this morning in a VM with a 100gb qcow2 as well which presents over VirtIO as a 512b sector virtual disk. It was able to receive the snapshot and read it using a zpool of default settings (Which picked ashift=9 as per what the virtual disk reports. But when I recreated that vm's zpool using ashift=12, the same IO errors and a single "permanent" error in the zpool status output when I tried to mount it and scrubbing made it go away as expected.

I'm glad the data is actually safe on the remote nas I have the backup stored on. For now, I rsync'd that to my desktop and started fresh in new encrypted datasets overnight so I can work this morning. Much slower than sending raw blocks but oh well.

Looking forward to the patch for this.

[psy0rz]

Please check out https://github.com/psy0rz/zfs_autobackup/wiki/zfs-check

It might be a usefull tool to check your actual data.

[asomers]

edit I determined that the ashift on the original pool was also 9, so my problem description is basically identical to the OP's.

[asomers]

I have a good workaround to repair datasets that have been corrupted by this bug. I added a zstream decompress subcommand that will decompress any records that got wrongly marked as uncompressed due to this bug. To use it send the corrupted dataset, filter the stream with zstream, and receive it again. It will fix corruption in any snapshot. For example:

zfs send -LRec <SRC_SNAP> | zstream decompress <INODE,OFFSET>... | zfs recv -s <DST_DS>

My current version only attempts LZ4, because that's the compression algorithm used on all of my affected datasets. But it could be easily modified to handle others, too. Would this tool be useful to anybody else? I can submit a PR if so.

[rincebrain]

I have no datasets broken by this, but would find that command useful for various other sundry experiments, so +1 to a PR from me.

[Kodiman]

Hi, Is this bug fixed? How to work around the issue? Thanks

[edited this message for clarity]

[rincebrain]

You could use #13256 to work around it if you're affected by the issue on existing datasets; the underlying bug which produced it has been patched for a while (2.1.2 and newer).

…

On Sun, Apr 24, 2022 at 1:58 PM Kodiman ***@***.***> wrote: Hi, Who is able to fix this issue? [ZFS send/recv with ashift 9->12 leads to data corruption] At least have the commands report human readable errors rather than silently making data corruption. Looking forward to patches for this. Much appreciated. Thanks — Reply to this email directly, view it on GitHub <#12762 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUI7MRNDQ6NCVGFVXV7EDVGWDS3ANCNFSM5H64B6YA> . You are receiving this because you commented.Message ID: ***@***.***>

[Kodiman]

Excellent. Thanks.

[aither64]

Unless there's some reason to keep this issue open, I'm closing it, since it is long fixed.