-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reproducible CKSUM errors after 2 drives taken OFFLINE on raidz2 #5806
Comments
It definitely appears that way. It looks as if I agree we should address this before the next tag. |
I can confirm that this reproduces on Illumos. |
@grwilson may also be interested in this. |
Is anyone already working on this? I think this is also reproducible on 2+ disks raidz1, which seems quite troubling. |
@loli10K Do you have a way to reproduce it on raidz1? I had to take 2 drives offline and do IO to reproduce on raidz2, but raidz1 can't do any IO if two drives have been taken offline. |
@thegreatgazoo i've never used raidz-n before (all my pools are mirrors) so i may be doing something wrong. That said, reproducer here: https://gist.github.com/loli10K/cc5b56612aa74871397066c2f6ac75d8. |
I can reproduce this (raidz2) on FreeBSD 11.0, too. |
@loli10K Thanks for the great script. I was able to reproduce it and I understand what's causing the problem. @grwilson and I are discussing what the best fix will be. The basic problem is:
|
Reviewed by: George Wilson [email protected] If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806
Reviewed by: George Wilson [email protected] If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". Closes openzfs#5806
Authored by: Matthew Ahrens <[email protected]> Reviewed by: George Wilson <[email protected]> Reviewed-by: loli10K <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Ported-by: Matthew Ahrens <[email protected]> If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". OpenZFS-issue: https://www.illumos.org/issues/8166 OpenZFS-commit: openzfs/openzfs#372 Closes #5806 Closes #6103
Reviewed by: George Wilson [email protected] If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806
Reviewed by: George Wilson [email protected] Reviewed by: Brad Lewis <[email protected]> If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 Closes #372
illumos/illumos-gate@2d2f193 illumos/illumos-gate@2d2f193 https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 Reviewed by: George Wilson [email protected] Reviewed by: Brad Lewis <[email protected]> Approved by: Richard Lowe <[email protected]> Author: Matthew Ahrens <[email protected]> git-svn-id: svn+ssh://svn.freebsd.org/base/head@318943 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
illumos/illumos-gate@2d2f193 illumos/illumos-gate@2d2f193 https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 Reviewed by: George Wilson [email protected] Reviewed by: Brad Lewis <[email protected]> Approved by: Richard Lowe <[email protected]> Author: Matthew Ahrens <[email protected]>
Authored by: Matthew Ahrens <[email protected]> Reviewed by: George Wilson <[email protected]> Reviewed-by: loli10K <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Ported-by: Matthew Ahrens <[email protected]> If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". OpenZFS-issue: https://www.illumos.org/issues/8166 OpenZFS-commit: openzfs/openzfs#372 Closes openzfs#5806 Closes openzfs#6103
illumos/illumos-gate@2d2f193 illumos/illumos-gate@2d2f193 https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 Reviewed by: George Wilson [email protected] Reviewed by: Brad Lewis <[email protected]> Approved by: Richard Lowe <[email protected]> Author: Matthew Ahrens <[email protected]> git-svn-id: https://svn.freebsd.org/base/vendor-sys/illumos/dist@318942 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
illumos/illumos-gate@2d2f193 illumos/illumos-gate@2d2f193 https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 Reviewed by: George Wilson [email protected] Reviewed by: Brad Lewis <[email protected]> Approved by: Richard Lowe <[email protected]> Author: Matthew Ahrens <[email protected]> git-svn-id: https://svn.freebsd.org/base/head@318943 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
illumos/illumos-gate@2d2f193 illumos/illumos-gate@2d2f193 https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 Reviewed by: George Wilson [email protected] Reviewed by: Brad Lewis <[email protected]> Approved by: Richard Lowe <[email protected]> Author: Matthew Ahrens <[email protected]> git-svn-id: svn+ssh://svn.freebsd.org/base/head@318943 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Reviewed by: George Wilson [email protected] Reviewed by: Brad Lewis <[email protected]> If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806
MFV r318942: 8166 zpool scrub thinks it repaired offline device https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 PR: 219537 Sponsored by: The FreeBSD Foundation
MFV r318942: 8166 zpool scrub thinks it repaired offline device https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 PR: 219537 Approved by: re (kib) Sponsored by: The FreeBSD Foundation
MFV r318942: 8166 zpool scrub thinks it repaired offline device https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 PR: 219537 Sponsored by: The FreeBSD Foundation git-svn-id: https://svn.freebsd.org/base/stable/10@319625 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
MFV r318942: 8166 zpool scrub thinks it repaired offline device https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 PR: 219537 Approved by: re (kib) Sponsored by: The FreeBSD Foundation git-svn-id: https://svn.freebsd.org/base/stable/11@319624 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Authored by: Matthew Ahrens <[email protected]> Reviewed by: George Wilson <[email protected]> Reviewed-by: loli10K <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Ported-by: Matthew Ahrens <[email protected]> If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". OpenZFS-issue: https://www.illumos.org/issues/8166 OpenZFS-commit: openzfs/openzfs#372 Closes #5806 Closes #6103
illumos/illumos-gate@2d2f193 illumos/illumos-gate@2d2f193 https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 Reviewed by: George Wilson [email protected] Reviewed by: Brad Lewis <[email protected]> Approved by: Richard Lowe <[email protected]> Author: Matthew Ahrens <[email protected]>
illumos/illumos-gate@2d2f193 illumos/illumos-gate@2d2f193 https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806 Reviewed by: George Wilson [email protected] Reviewed by: Brad Lewis <[email protected]> Approved by: Richard Lowe <[email protected]> Author: Matthew Ahrens <[email protected]>
Reviewed by: George Wilson [email protected] Reviewed by: Brad Lewis <[email protected]> If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also openzfs/zfs#5806
I was able to 100% reproduce it on current/vanilla master (spl at 9704820, zfs at 100790a). Simple to reproduce:
The two drives were taken offline right after pool creation, so the amount of lost data/parity on them should be fairly close. Looking at the bold text above, the sde resilver fixed 39.0M but the sdn resilver only fixed 12K. So it looked like the 2nd resilver missed quite some blocks. The 2nd scrub fixed 38.7M, and if we add that to the 12K fixed by the 2nd resilver, it'd get fairly close to the 39.0M fixed by the 1st resilver. So looked like the 2nd scrub was actually fixing the blocks missed by the 2nd resilver.
Since the difference between resilver and scrub is that resilver would look at DTL_PARTIAL to decide whether to check a block, I guess something messed up the DTLs before the 2nd resilver - therefore the 1st scrub looked very fishy. Then I did the same thing again, except I didn't do the scrub between the 2 resilvers:
This time there's 0 error, and the 2 resilvers fixed about the same amount of data/parity. Everything above is 100% repeatable. Which seemed to verify my guess that the scrub between resilvers messed the DTL somehow.
The resilver/scrub and raidz code really hasn't changed much - I also used zfs_vdev_raidz_impl="original" to disable the new fancy parity routines - so I'd suspect it'd affect older ZFS versions as well, maybe even ZFS on other OS. This is probably something we'd want to fix before the next release. I realized the 14-drive raidz2 I used in the tests was not a common configuration, but it's not crazy either.
The text was updated successfully, but these errors were encountered: