[WIP] draid rebase #9558

ofaaland · 2019-11-06T23:14:06Z

Motivation and Context

This is a WIP rebase of draid from (most recent at top):
@mmaybee zfs/tree/crayzfs-0.8.x
which was a rebase of:
@don-brady PR #8016
which was a rebase of:
@thegreatgazoo PR #7078

To allow continued community development using a branch based on most recent master. The rebase is the only improvement.

Very little testing has been done and there are likely new issues resulting from the rebases/merges in addition to whatever issues existed in the earlier PRs.

Description

Rebasing Notes:

Updated to use new types and functions relevant to range trees
Background and to-do list as of May 2019 DRAID workshop: https://docs.google.com/document/d/1SDuFWjiAZbsqJSYOt0iO2Dh4RvU-CzcixtQqLW6TuIw/edit#heading=h.zd12b6w407aq

How Has This Been Tested?

Minor manual testing of draid configs

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the ZFS on Linux code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
All new and existing tests passed.
All commit messages are properly formatted and contain Signed-off-by.

ofaaland · 2019-11-06T23:24:04Z

@richardelling this is a reminder to push the additional tests you mentioned, when you get a chance, and I'll cherry-pick them here so we get improvements all in one place. Thanks!

mmaybee · 2019-11-06T23:27:15Z

Very cool Olaf. Please be aware that zfs/tree/crayzfs-0.8.x (which is a rebase from Don's branch) is still a little rough. There are, recently discovered, merge issues in metaslab.c and spa.c.

ofaaland · 2019-11-06T23:45:15Z

I fixed style issues and bad arguments I'd provided to rs_get_start() an an earlier version.

mmaybee · 2019-11-06T23:53:06Z

The merge issues in spa.c are in spa_vdev_attach() (we are not starting a rebuild when we should be). Ther merge issues in metaslab.c are more pervasive: metaslab_block_picker() and metaslab_df_alloc() both need work (because of changes to the algorithms). I can provide details/fixes.

ofaaland · 2019-11-07T00:01:09Z

The merge issues in spa.c are in spa_vdev_attach() (we are not starting a rebuild when we should be). Ther merge issues in metaslab.c are more pervasive: metaslab_block_picker() and metaslab_df_alloc() both need work (because of changes to the algorithms). I can provide details/fixes.

OK great, thanks!

PrivatePuffin · 2019-11-12T17:16:33Z

@ofaaland @mmaybee
Is there somewhat of a list of what actually needs work on this PR?

Any specific bugs?
Any specific issues?
Just testing?
More code review?

Being a WIP is nice and all, but I think it makes it hard for other people to start sharing in the work and thought process without a little more background information on the current state of this WIP (after all those years)...

ofaaland · 2019-11-12T17:22:00Z

@Ornias1993 I believe @behlendorf has such a list and is looking for it, for those reasons. Stay tuned...

sjorge · 2019-11-13T14:17:49Z

Someone should probably also update the feature matrix and link to this PR instead of the abandoned one.

ofaaland · 2019-11-13T19:00:58Z

Someone should probably also update the feature matrix and link to this PR instead of the abandoned one.

@sjorge I'm happy to see if I can update it. Where is it?

ahrens · 2019-11-13T19:02:19Z

@ofaaland I already updated it, but it's here: https://docs.google.com/spreadsheets/d/1CFapSYxA5QRFYy5k6ge3FutU7zbAWbaeGN2nKVXgxCI/edit#gid=0

PrivatePuffin · 2019-11-17T10:40:12Z

@ofaaland Guess i've found the answer to my own question already:
I think you meant @behlendorf was looking for this document?

https://docs.google.com/document/d/1SDuFWjiAZbsqJSYOt0iO2Dh4RvU-CzcixtQqLW6TuIw/edit#heading=h.zd12b6w407aq

Maybe add it to the PR under "Motivation and Context" or "Description", because it answers a LOT of questions.

edit
Found it while reading up on the monthly meeting notes, for other folks thinking they missed something, must read.

jkiebzak · 2019-12-05T20:20:26Z

hopefully this is the correct place for this.

I am getting PANICs while trying to create a draid zpool:

Message from syslogd@bb-129 at Dec  5 15:14:21 ...
 kernel:[  120.156895] VERIFY3(start == vdev_draid_get_astart(vd, start)) failed (17179869184 == 17179877376)

Message from syslogd@bb-129 at Dec  5 15:14:21 ...
 kernel:[  120.156915] VERIFY3(start == vdev_draid_get_astart(vd, start)) failed (34359738368 == 34359742464)

Message from syslogd@bb-129 at Dec  5 15:14:21 ...
 kernel:[  120.156917] PANIC at vdev_draid.c:478:vdev_draid_check_block()

Message from syslogd@bb-129 at Dec  5 15:14:21 ...
 kernel:[  120.157156] PANIC at vdev_draid.c:478:vdev_draid_check_block()

I've tried various draid configurations, all of which PANIC. Here is the most basic configuration

draidcfg -p 1 -d 2 -s 1 -n 4 4.nvl
zpool create -f data draid1 cfg=4.nvl /dev/sd{a..d}

Here's my system:

cat /etc/debian_version
9.11

uname -a
Linux bb-129 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux

Here were my installation steps:

apt update && apt upgrade

apt install git
git clone https://github.com/zfsonlinux/zfs.git
cd zfs
git pull origin pull/9558/head

apt -y install build-essential autoconf automake libtool gawk alien fakeroot ksh
apt -y install zlib1g-dev uuid-dev libattr1-dev libblkid-dev libselinux-dev libudev-dev
apt -y install libacl1-dev libaio-dev libdevmapper-dev libssl-dev libelf-dev
apt -y install linux-headers-$(uname -r)
apt -y install python3 python3-dev python3-setuptools python3-cffi

sh autogen.sh
./configure --enable-debug
make -s -j$(nproc)
make deb
dpkg -i *.deb

nano /etc/modprobe.d/zfs.conf
  options zfs_vdev_scrub_max_active=10 zfs_vdev_async_write_min_active=4 draid_debug_lvl=5
echo zfs >> /etc/modules
/sbin/modprobe zfs
shutdown -r now

mmaybee · 2019-12-05T21:33:09Z

These panics are probably due to the merge issues in the metaslab alloc code path that I mentioned above. I have been meaning to get the fixes pushed, but have not yet found the time. Sorry about the delay.

Vlad1mir-D · 2019-12-20T06:25:44Z

Is it possible to remove or add VDEV into the ZFS pool which contains draid VDEV?

richardelling · 2019-12-21T23:52:24Z

Is it possible to remove or add VDEV into the ZFS pool which contains draid VDEV?

For top-level vdevs:

add: yes
remove: no, due to the same restrictions facing raidz

This patch implements the dRAID vdev driver and a new rebuild mechanism. This is still work in progress the user interface may change as well as the on-disk format. Code changes: + module/zfs/vdev_draid.c: vdev driver for dRAID and dRAID spare + module/zfs/vdev_scan.c: sequential rebuild for dRAID and mirror + cmd/draidcfg/*.[ch]: user space tools, mainly to create permutations + module/zfs/vdev_raidz.c: the parity functions need to include dRAID skip sectors for computation and reconstruction. + module/zfs/vdev_mirror.c: minor changes to support mirror_map_t allocated by dRAID vdev (for hybrid mirror support) + module/zfs/metaslab.c: to add support for dRAID hybrid mirror, also disallow block allocation during rebuild The following new ztest command options were added to test dRAID. -K draid|raidz|random -- kind of RAID to test -D <value> -- dRAID data drives per redundancy group -G <value> -- dRAID redundancy group count -S <value> -- dRAID distributed spare drives -R <value> -- RAID parity (raidz or dRAID) -L -- (Existing -G (dump log option) was renamed -L) Added dRAID specific section to zloop to exercise dRAID across a wide range of configuration parameters. For example: ztest -VVVV -K draid -D 7 -G 6 -S 1 -m 0 -r 1 -R 1 -v 0 -a 12 -s 384m The goal is to change existing code in a way that when draid is not in use the effective change is none. Todo: * Merge the draidcfg utility into zpool command so that there is no additional step before creating a draid pool * Update the new sequential rebuild code to use the generic mechanism for enable/disabling allocations to metaslabs. * Debug dRAID hybrid mirror code (currently disabled). * Clean up rebuild in-memory and persisted states so that it doesn't share any state with the resilver states. This will allow concurrent rebuild and resilver, e.g. a raidz vdev can be resilvering while a draid vdev is rebuilding. * Extend the ZTS to include dRAID test coverage. Co-authored-by: Isaac Huang <[email protected]> Co-authored-by: Mark Maybee <[email protected]> Co-authored-by: Don Brady <[email protected]> Co-authored-by: Srikanth N S <[email protected]> Co-authored-by: Stuart Maybee <[email protected]> Co-authored-by: Brian Behlendorf <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> External-issue: ZFS-12 ZFS-35 ZFS-36 ZFS-17 ZFS-56 ZFS 95 ZFS-96 External-issue: ZFS-100 ZFS-103 ZFS-106 ZFS-110 ZFS-111 ZFS-117 Issue openzfs#9558

ofaaland · 2020-01-31T19:12:49Z

Just pushed 2f51d38 Brian ginned up

rebased on master
includes fixes by @mmaybee & co at Cray
includes fixes by @behlendorf
squashed

This is the current best available draid patch that I am aware of

PrivatePuffin · 2020-01-31T19:16:24Z

This is some significant progress, thank you! :)

codecov · 2020-02-01T02:35:54Z

Codecov Report

Merging #9558 into master will decrease coverage by <1%.
The diff coverage is 81%.

@@            Coverage Diff            @@
##           master    #9558     +/-   ##
=========================================
- Coverage      79%      79%    -<1%     
=========================================
  Files         384      389      +5     
  Lines      121788   123800   +2012     
=========================================
+ Hits        96590    98067   +1477     
- Misses      25198    25733    +535

Flag	Coverage Δ
#kernel	`79% <18%> (-1%)`	⬇️
#user	`66% <81%> (-1%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 591505d...2f51d38. Read the comment docs.

This patch implements the dRAID vdev driver and a new rebuild mechanism. This is still work in progress the user interface may change as well as the on-disk format. Code changes: + module/zfs/vdev_draid.c: vdev driver for dRAID and dRAID spare + module/zfs/vdev_scan.c: sequential rebuild for dRAID and mirror + cmd/draidcfg/*.[ch]: user space tools, mainly to create permutations + module/zfs/vdev_raidz.c: the parity functions need to include dRAID skip sectors for computation and reconstruction. + module/zfs/vdev_mirror.c: minor changes to support mirror_map_t allocated by dRAID vdev (for hybrid mirror support) + module/zfs/metaslab.c: to add support for dRAID hybrid mirror, also disallow block allocation during rebuild The following new ztest command options were added to test dRAID. -K draid|raidz|random -- kind of RAID to test -D <value> -- dRAID data drives per redundancy group -G <value> -- dRAID redundancy group count -S <value> -- dRAID distributed spare drives -R <value> -- RAID parity (raidz or dRAID) -L -- (Existing -G (dump log option) was renamed -L) Added dRAID specific section to zloop to exercise dRAID across a wide range of configuration parameters. For example: ztest -VVVV -K draid -D 7 -G 6 -S 1 -m 0 -r 1 -R 1 -v 0 -a 12 -s 384m The goal is to change existing code in a way that when draid is not in use the effective change is none. Todo: * Update the new sequential rebuild code to use the generic mechanism for enable/disabling allocations to metaslabs. * Debug dRAID hybrid mirror code (currently disabled). * Clean up rebuild in-memory and persisted states so that it doesn't share any state with the resilver states. This will allow concurrent rebuild and resilver, e.g. a raidz vdev can be resilvering while a draid vdev is rebuilding. * Extend the ZTS to include dRAID test coverage. Co-authored-by: Isaac Huang <[email protected]> Co-authored-by: Mark Maybee <[email protected]> Co-authored-by: Don Brady <[email protected]> Co-authored-by: Srikanth N S <[email protected]> Co-authored-by: Stuart Maybee <[email protected]> Co-authored-by: Brian Behlendorf <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> External-issue: ZFS-12 ZFS-35 ZFS-36 ZFS-17 ZFS-56 ZFS 95 ZFS-96 External-issue: ZFS-100 ZFS-103 ZFS-106 ZFS-110 ZFS-111 ZFS-117 Issue openzfs#9558

PrivatePuffin · 2020-02-28T21:15:22Z

@behlendorf
I don't think this is going to be in OpenZFS2.0, it might be prudent to just create a placeholder project "OpenZFS2.0" and drop this there.

behlendorf · 2020-02-28T22:45:41Z

We should have an updated version of this ready next week. But yes, there's still a fair bit of work needed and we don't plan to block the OpenZFS release on this feature. I've gone ahead and removed it from the project.

This patch adds a new top-level vdev type called dRAID, which stands for Distributed parity RAID. This pool configuration allows all dRAID vdevs to participate when rebuilding to a distributed hot spare device. This can substantially reduce the total time required to restore full parity to pool with a failed device. A dRAID pool can be created using the new top-level `draid` type. Like `raidz`, the desired redundancy is specified after the type: `draid[1,2,3]`. No additional information is required to create the pool and reasonable default values will be chosen based on the number of child vdevs in the dRAID vdev. zpool create <pool> draid[1,2,3] <vdevs...> Unlike raidz, additional optional dRAID configuration values can be provided as part of the draid type as colon separated values. This allows administrators to fully specify a layout for either performance or capacity reasons. The supported options include: zpool create <pool> \ draid[<parity>][:<data>d][:<children>c][:<spares>s] \ <vdevs...> - draid[parity] - Parity level (default 1) - draid[:<data>d] - Data devices per group (default 8) - draid[:<children>c] - Expected number of child vdevs - draid[:<spares>s] - Distributed hot spares (default 0) Abbreviated example `zpool status` output for a 68 disk dRAID pool with two distributed spares using special allocation classes. ``` pool: tank state: ONLINE config: NAME STATE READ WRITE CKSUM slag7 ONLINE 0 0 0 draid2:8d:68c:2s-0 ONLINE 0 0 0 L0 ONLINE 0 0 0 L1 ONLINE 0 0 0 ... U25 ONLINE 0 0 0 U26 ONLINE 0 0 0 spare-53 ONLINE 0 0 0 U27 ONLINE 0 0 0 draid2-0-0 ONLINE 0 0 0 U28 ONLINE 0 0 0 U29 ONLINE 0 0 0 ... U42 ONLINE 0 0 0 U43 ONLINE 0 0 0 special mirror-1 ONLINE 0 0 0 L5 ONLINE 0 0 0 U5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 L6 ONLINE 0 0 0 U6 ONLINE 0 0 0 spares draid2-0-0 INUSE currently in use draid2-0-1 AVAIL ``` When adding test coverage for the new dRAID vdev type the following options were added to the ztest command. These options are leverages by zloop.sh to test a wide range of dRAID configurations. -K draid|raidz|random - kind of RAID to test -D <value> - dRAID data drives per group -S <value> - dRAID distributed hot spares -R <value> - RAID parity (raidz or dRAID) The zpool_create, zpool_import, redundancy, replacement and fault test groups have all been updated provide test coverage for the dRAID feature. Co-authored-by: Isaac Huang <[email protected]> Co-authored-by: Mark Maybee <[email protected]> Co-authored-by: Don Brady <[email protected]> Co-authored-by: Stuart Maybee <[email protected]> Co-authored-by: Matthew Ahrens <[email protected]> Co-authored-by: Brian Behlendorf <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#9558

This patch adds a new top-level vdev type called dRAID, which stands for Distributed parity RAID. This pool configuration allows all dRAID vdevs to participate when rebuilding to a distributed hot spare device. This can substantially reduce the total time required to restore full parity to pool with a failed device. A dRAID pool can be created using the new top-level `draid` type. Like `raidz`, the desired redundancy is specified after the type: `draid[1,2,3]`. No additional information is required to create the pool and reasonable default values will be chosen based on the number of child vdevs in the dRAID vdev. zpool create <pool> draid[1,2,3] <vdevs...> Unlike raidz, additional optional dRAID configuration values can be provided as part of the draid type as colon separated values. This allows administrators to fully specify a layout for either performance or capacity reasons. The supported options include: zpool create <pool> \ draid[<parity>][:<data>d][:<children>c][:<spares>s] \ <vdevs...> - draid[parity] - Parity level (default 1) - draid[:<data>d] - Data devices per group (default 8) - draid[:<children>c] - Expected number of child vdevs - draid[:<spares>s] - Distributed hot spares (default 0) Abbreviated example `zpool status` output for a 68 disk dRAID pool with two distributed spares using special allocation classes. ``` pool: tank state: ONLINE config: NAME STATE READ WRITE CKSUM slag7 ONLINE 0 0 0 draid2:8d:68c:2s-0 ONLINE 0 0 0 L0 ONLINE 0 0 0 L1 ONLINE 0 0 0 ... U25 ONLINE 0 0 0 U26 ONLINE 0 0 0 spare-53 ONLINE 0 0 0 U27 ONLINE 0 0 0 draid2-0-0 ONLINE 0 0 0 U28 ONLINE 0 0 0 U29 ONLINE 0 0 0 ... U42 ONLINE 0 0 0 U43 ONLINE 0 0 0 special mirror-1 ONLINE 0 0 0 L5 ONLINE 0 0 0 U5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 L6 ONLINE 0 0 0 U6 ONLINE 0 0 0 spares draid2-0-0 INUSE currently in use draid2-0-1 AVAIL ``` When adding test coverage for the new dRAID vdev type the following options were added to the ztest command. These options are leverages by zloop.sh to test a wide range of dRAID configurations. -K draid|raidz|random - kind of RAID to test -D <value> - dRAID data drives per group -S <value> - dRAID distributed hot spares -R <value> - RAID parity (raidz or dRAID) The zpool_create, zpool_import, redundancy, replacement and fault test groups have all been updated provide test coverage for the dRAID feature. Co-authored-by: Isaac Huang <[email protected]> Co-authored-by: Mark Maybee <[email protected]> Co-authored-by: Don Brady <[email protected]> Co-authored-by: Stuart Maybee <[email protected]> Co-authored-by: Matthew Ahrens <[email protected]> Co-authored-by: Brian Behlendorf <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#9558 Conflicts: module/zfs/vdev.c