RAIDZ Expansion feature discussion #15232
Replies: 22 comments 80 replies
-
|
So to summarize that: Afaik, a copy command will rewrite the data with the new width after expansion. I think I saw a script sloshing around that basically does that for an entire array, cleaning up after itself. Does it file by file for space reasons. Takes a minute, but, you know. Only have to do it once. From my perspective that’s plenty good enough. Raid expansion is a niche feature, if much beloved by that niche. Adjusting width is then a niche within the niche. Handling that with a community script sounds like an entirely reasonable thing to do. Permissions and paths stay as is, filenames stay, last accessed and creation date changes and width changes. To explain the niche comments: I am part of that niche. Home user but without the funds to start at full width. I did a poor man’s raid expansion eventually, from 5 to 8. So the niche are people who start at a smaller width than they know they’ll need, and will only run one vdev. By definition home users, and then only a fraction of those. The niche within the niche are users where rewriting archival data makes an appreciable difference, or that care even though it doesn’t. That need not be handled in the file system. Handling it with good UX outside seems great. Setups that use ZFS like TrueNAS and Proxmox could even integrate such a script into their UI. |
Beta Was this translation helpful? Give feedback.
-
|
@TurkeyMan |
Beta Was this translation helpful? Give feedback.
-
|
I'm part of that niche crowd, and I'm cool with it. If I decide to expand, I'm fine with keeping the old setup until it gets a rewrite. Storage is pretty cheap these days, which is why I'm looking to expand. That being said, I think my use case is decent. Right now, I've got two vdevs, but one of them has a bunch of older, 8TB drives. It just doesn't make sense to swap an 8TB with another 8TB when a 18TB is way cheaper per TB. As such, it just makes much more sense to expand my current vdev with 18TB drives and end up removing my old vdevof 8TB drives. So, I much follow the path that you did. With that said, when I remove my 2nd vdev, my understanding is zfs will shift my data over to the 1 remaining vdev. If this code/process exists, couldn't that process be called, albeit maybe modified, to balance it out again when expanding? Maybe not since its just moving the data from vdev to vdev vs adding a drive to expand which I guess would be a very different flow. This is where I may be a bit lost. |
Beta Was this translation helpful? Give feedback.
-
|
You can swap individually each 8TB with 18TB drives and once all of them are 18TB you can expand them to occupy entire space of the new drives. |
Beta Was this translation helpful? Give feedback.
-
|
This script, with caveats (read the readme), helps automate rewriting your data by copying/pasting/deleting/moving at the filesystem level. https://github.com/markusressel/zfs-inplace-rebalancing |
Beta Was this translation helpful? Give feedback.
-
|
FYI, Link is bad - goes to https://github.com/openzfs/zfs/discussions/url |
Beta Was this translation helpful? Give feedback.
-
|
Fixed. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
The script doesn't use direct IO which means the data might end up sitting in caches and you're only hashing stuff from the page cache, not reading back what makes it to disk. I think this leaves a small window for data corruption, especially on systems without ECC memory. |
Beta Was this translation helpful? Give feedback.
-
|
This is cool, and I think solves the problem well if you don't have snapshots. If you wanted to preserve space sharing between snapshots and clones, you could use |
Beta Was this translation helpful? Give feedback.
-
|
@Dulanic you can’t removed a raidz vdev from a pool. Mirror vdevs only (single disk vdev is special case of mirror) can be removed. You may be able to create a new pool with your 18TB drives, copy data over, then expand that. This really depends on how much you’re using already though. This is why a lot of home users stick with one vdev, lest they are “stuck” with two to manage. I have one 8x8 raidz2 which will likely last me forever. But in case it ever doesn’t, I intend to replace drives as they fail with larger-capacity ones. Once the last drive has been replaced, I’ll have more space. |
Beta Was this translation helpful? Give feedback.
-
|
Oh no! Lol I am still learning! That's a problem lol I have 80TB already! If I have 2 vdevs can I expand 1 if them? I wasn't sure if I needed to match the width. |
Beta Was this translation helpful? Give feedback.
-
|
My current take on this, the stripe-width expansion seems less of an urgent issue than the fact it only adds one drive at a time. |
Beta Was this translation helpful? Give feedback.
-
|
Unsurprisingly people have been making wider raidz2 vdevs for years, even without official expansion support. The “use sparse files for parity and offline them” trick has gotten good mileage :). |
Beta Was this translation helpful? Give feedback.
-
|
FYI: Time to expand is limited by CPU usage. Improving it by 10x would be a relatively straightforward follow-on project. Also note that the 50MB/s that I measured was with 512-byte sector size. With 4K sector size it would be around 8x that (~400MB/s). |
Beta Was this translation helpful? Give feedback.
-
Is that something likely to be done in the near-ish future? 😄 |
Beta Was this translation helpful? Give feedback.
-
I think that would be make raidz expansion much more viable. |
Beta Was this translation helpful? Give feedback.
-
Not by me, but PR's are welcome 😄 |
Beta Was this translation helpful? Give feedback.
-
|
Is RAIDZ "contraction" support included in this PR? 🤔 |
Beta Was this translation helpful? Give feedback.
-
|
Reasonable. |
Beta Was this translation helpful? Give feedback.
-
|
I’d really rather not see any such thing. ZFS is wondrously complex as is. Per-dataset parity adds another level of complexity for no gain that I can see. In this expansion PR, all data have the same parity in the expanded vdev. It’s the stripe width that changes, not the parity. If I recall right there has been discussion of a follow-on PR that’d expand (but not contract) parity, such as from raidz1 to raidz2 or raidz3. That’d be separate work, and whether it will ever be tackled is a good question. The existing mechanisms for changing pool layouts - make a new pool, and/or add vdevs - work for the use cases I can see. Raidz expansion is a lovely hobbyist use case and I am stoked it’s getting there; raidz parity expansion would be another hobbyist use case but even more niche. |
Beta Was this translation helpful? Give feedback.
-
|
The way expansion is implemented now the contraction is simply impossible. Even in unlikely case that end of the vdev is absolutely empty to be dropped, contraction would not produce sufficient level of redundancy to sustain disk failures. |
Beta Was this translation helpful? Give feedback.
-
|
@yorickdowne @amotin thanks for reply!
|
Beta Was this translation helpful? Give feedback.
-
|
@GrayXu While some of that sounds like a potentially interesting discussion topic, it's probably best to create a new discussion for it. It's not very on-topic for this one. 😄 |
Beta Was this translation helpful? Give feedback.
-
|
To @farkeytron: Discussion is here, not in the PR. IxSystems is paying for this work, so it’ll definitely come into TrueNAS. SCALE first likely, since that is where canary features land, then Core in due time. |
Beta Was this translation helpful? Give feedback.
-
|
Lol, very appropriate meme :) This feature will involve more middleware and UI changes to support properly, which means it will be targeted at SCALE only. CORE is mostly feature frozen, so we aren't making big sweeping changes to features / backend / front-end. SCALE is where active development resides. |
Beta Was this translation helpful? Give feedback.
-
|
"next major update to our ZFS base" What timeframe is this? Next major build after Cobia? Sorry if I've missed this info. |
Beta Was this translation helpful? Give feedback.
-
|
24.10 (Electric Eel - October release) is the next major TrueNAS SCALE release that will include ZFS 2.3. |
Beta Was this translation helpful? Give feedback.
-
|
Reading this makes me want to upgrade my core server to scale but migrating things like nextcloud configurations and data makes me concerned. Is it a certainty that this feature wouldn't make its way to core? (even if it was just for cli access) |
Beta Was this translation helpful? Give feedback.
-
|
That feature won't make it into 13.3, so odds are low it will land in CORE in the next 12-24 months. If you need a hand with migrating nextcloud we've had plenty of other folks do that. Might be worth asking questions here: |
Beta Was this translation helpful? Give feedback.
-
Hah, I take you don't follow the issue tracking on github too carefully, then. The amount of issues I see with the encryption support (and not just performance problems, but like, swallowed my data or corrupted it on a replication ones) has made me pretty hesitant to recommend ZFS's native encryption support for anyone. |
Beta Was this translation helpful? Give feedback.
-
Yeah, not at all. I think this discussion and it's matching PR are the only things I'm subscribed to in the repo.
Damn. Sounds like we'll need to investigate that, and potentially find an alternative to ZFS then. Or at least an alternative to native ZFS encryption. 😦 Can't be having unreliable storage backend for production systems. 🤦 |
Beta Was this translation helpful? Give feedback.
-
|
I use encryption in production: it's not worse than many other features like zvol (#11407). |
Beta Was this translation helpful? Give feedback.
-
|
@don-brady @ahrens Thank you SO much for getting the ZFS expansion PR across the line. It'll likely be a key capability of ZFS from here on (in the next release onwards of course). 😄 |
Beta Was this translation helpful? Give feedback.
-
|
is the expansion option for all setups including stripe? |
Beta Was this translation helpful? Give feedback.
-
|
it's for raidz vdevs specifically, which have 1-3 devices worth of parity. |
Beta Was this translation helpful? Give feedback.
-
|
This Feature is not for switching from stripe to raidz1-3! |
Beta Was this translation helpful? Give feedback.
-
|
Quick question: if I update to zfs master and enable corresponding feature flag for raidz expansion, will it be active or enabled after expansion completes? Thus, can I downgrade to stable branch after that or there will be some incompatibilities in on disk layout and I'm limited to master branch after that until the next stable release. |
Beta Was this translation helpful? Give feedback.
-
|
On-disk format after expansion is different from one supported before this feature. So I bet the feature stays active forever. |
Beta Was this translation helpful? Give feedback.
-
|
That's correct. After expansion the feature will be "active", and you can't downgrade the software to an earlier version that doesn't support RAIDZ Expansion. |
Beta Was this translation helpful? Give feedback.
-
|
A couple of days ago @mmatuska updated OpenZFS in FreeBSD main, including RAIDZ Expansion. You can try this out with the next set of 15-CURRENT snapshot images. |
Beta Was this translation helpful? Give feedback.
-
|
I tried to read through the DOCS here but still not sure if I can extend my 2-disk VDEV ( only option to create was a mirror ) Can this be expanded with a 3rd disk as RAIDZ-1 ? |
Beta Was this translation helpful? Give feedback.
-
|
No. A mirror vdev is not raidz-1. But this is what you can do: Break the mirror, remove one drive. Take that drive, a second fresh one, and a sparse file, and make your raidz1 in a new pool, not the same pool! Offline the sparse file. You now have two pools with one vdev each, and no redundancy. Hope you have a good backup. zsend/zrecv from the old pool to the new one verify everything is there, twice. Because the next step means you can’t go back. destroy the old pool, replace the offlined sparse file in the raidz1 with the now free drive. Let the raidz1 resilver. This would also work with a raidz2 and two new drives. Which is a bit more future proof. Raidz1 with larger drives has a tendency to break during resilver. |
Beta Was this translation helpful? Give feedback.
-
|
With my N00b level of understanding you did a great job of explaining. With that said I think I lean towards Backup-copy My 2 disk mirrored Vdev, then create a brand new with my now 3 disks and then copy back the data into the new vdev? ( more failsafe and faster ? or am I missing something , got 14 TB disks) |
Beta Was this translation helpful? Give feedback.
-
|
With 14TB disks raidz-1 is risky when rebuilding. I recommend you do a 4-wide raidz2, which you can then expand later if desired. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds like a good solution ... just need to squeeze my wallet upsidedown and hope some monetas will fall out from it =) ( but yes If I don´t I will probably regret it later on ) |
Beta Was this translation helpful? Give feedback.
-
|
“Buy nice or buy twice” :) |
Beta Was this translation helpful? Give feedback.
-
|
This is arriving at a really good moment for me, I was about to expand a 4-disk raidz2 into a 6-disk by the old method (create an entirely new pool then So I must ask:
TIA! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @yorickdowne and @darkbasic for your responses. And I just read a little more about the limitations, and it seems the data-to-parity ratio isn't recalculated either, so a ton of space would end up being lost if the expansion is done using this new functionality.
EDIT: the situation is much less worse than I was led to believe, see my next comment below, |
Beta Was this translation helpful? Give feedback.
-
That is just until you rewrite the existing data. If your data is going to change over time (or if you don't care about snapshots and simply rewrite it) you will regain the lost space. |
Beta Was this translation helpful? Give feedback.
-
That is incorrect. You can expand existing RAIDZ groups. |
Beta Was this translation helpful? Give feedback.
-
That is awesome, I've never been so happy to be proven wrong :) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @ahrens and @darkbasic for the additional corrections/clarifications. So the situation is much better than I was led to believe. The major (only?) conses seem to be the (a) inability of the upgraded array to be handled by previous versions of the software not supporting this feature, and (b) desirability of rewriting all data in order to "rebalance" already-existing data (which I understand means both to redistribute its data on the new array including the added vdevs, and recalculate its parity blocks for the new scheme). (a) is not a major problem (just test the new version well enough to be damn sure no version downgrading will be necessary) and (b) is just an inconvenience for my use case (I presume that, once I have the necessary disk space, I can just shuttle the data inside it from the previous datasets to new ones to force the rewriting with zfs send|recv, and then destroying the pre-rebalance datasets and renaming the new ones to the old names). I will now mark this on my original comment above. |
Beta Was this translation helpful? Give feedback.
-
|
How does expand address fragmentation, if my cap is under 50% does the end result remove most/all fragmentation? What if my capacity is over 50%, does the end result in more fragmentation then before? |
Beta Was this translation helpful? Give feedback.
-
|
The newly-available space will be one big free chunk, but expansion doesn't change fragmentation of the existing data. |
Beta Was this translation helpful? Give feedback.
-
|
Gotcha, I figured expansion when re-writing between disks would fix fragmentation if the free space was greater than 50%. |
Beta Was this translation helpful? Give feedback.
-
|
I'm considering upgrading the disks in my system with something faster, and add an extra disk in the process. So let's say I first add the extra disk and let ZFS expand the pool, then swap out the pre-existing disks one by one (or perhaps connect the faster one alongside) and let it resilver each time. Would the resulting pool have the updated data-parity ratio? I would assume so since eventually all disks would have newly written data, but because it happened in response to a resilver it may not take the earlier expansion (fully) into account? As in: disk X is pretty much copied bit-for-bit to replacement disk X, resulting in the same original ratio (only that disk has to be rebuilt so there's no (re)balancing of data-parity). I don't think there's enough room to connect all the new disks alongside the original ones and just send it over. :DD |
Beta Was this translation helpful? Give feedback.
-
|
You realize this functionality is still a year or two away, yes? |
Beta Was this translation helpful? Give feedback.
-
|
Ever heard of planning? If the scenario I described wouldn't work then it's probably better to find a way to make room for a new pool and send over the data after all, which I could do right this moment. Otherwise, having to wait for 1-2 years with the current setup is perfectly acceptable. |
Beta Was this translation helpful? Give feedback.
-
|
I'm planning to use ZFS for my homeserver with proxmox. Would it be possible to setup a 2 Drive RAIDZ1 right now and add a third drive later using this method when it releases? |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, that's the kind of thing this is for. 😄 |
Beta Was this translation helpful? Give feedback.
-
|
Awesome thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Oh, that's a good point. I think the smallest # of drives I've tried with a RAIDZ1 is actually 3. I'm not actually sure about 2. Err... oops? 🤦 |
Beta Was this translation helpful? Give feedback.
-
|
Sure, RAIDZ1 with 2 drives is perfectly valid. I'm using that right now. |
Beta Was this translation helpful? Give feedback.
-
|
2 drive raidz is just a mirror with crippled performance. You better of
creating real mirror now and one you have all 3 drive - split the mirror
and create degraded raidz from 2 of 3 drives and zfs send all the data from
the 3rd. After that you can attach 3rd to make it redundant again. You will
sacrifice redundancy during migration though.
…On Thu, Feb 29, 2024, 02:55 Quadrubo ***@***.***> wrote:
Awesome thank you!
Wasn't even sure if RAIDZ1 with just 2 drives is a valid configuration.
—
Reply to this email directly, view it on GitHub
<#15232 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABXQ6HJZEA2NGDSHANUI3LDYV5HP7AVCNFSM6AAAAAA4HOSFKSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DMMRQGM3DM>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi when is this feature going to be available? From what I know it was said that will be added with "OpenZFS 2.3" version but do we have some ETA? Thanks |
Beta Was this translation helpful? Give feedback.
-
|
It is supposed to be added to TrueNAS Scale 24.10 Electric Eel. So probably before October. |
Beta Was this translation helpful? Give feedback.
-
|
so TrueNAS Scale 24.10 got released 6 days ago and its in the rchangelog 🎉 |
Beta Was this translation helpful? Give feedback.
-
|
@Xyz00777 Cool. Yeah, that looks interesting: https://www.truenas.com/docs/scale/gettingstarted/scalereleasenotes/ 
Seems like there's even documentation in place for it too: https://www.truenas.com/docs/scale/scaletutorials/storage/managepoolsscale/#extending-a-raidz-vdev- Note that TrueNAS Scale 24.10 release is still a beta though, not the full release. So the software is clearly still "in development". Hopefully that gets a bunch of people testing it out, and if there's bugs then they get fixed before it becomes widely available for the rest of us (ZFS 2.3?). |
Beta Was this translation helpful? Give feedback.
-
|
ixSystems just posted this release 24.10 version notes a week ago. It has a clear answer to your question (and many other peoples as well). Put on a big grin on your face and be ready for this Thanksgiving or Christmas gift, depending on how adventurous you are. Mark the date on your calendar, and make your hard drive shopping list accordingly. 😄 highlight by me for those with bad eyes like me.
This wait had been like years. Kudos to @don-brady and all who contributed. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, special thank you to everybody on the OpenZFS side who helped get this one over the finish line. We've got this in our TrueNAS 24.10 BETA1 release already for testing, anticipating updating to OpenZFS 2.3 as soon as it is tagged. |
Beta Was this translation helpful? Give feedback.
-
|
IMO "rewriting the data" should be part of the expansion process and part of this feature before release. I suppose making it optional at the time of expansion would be fine, but not too happy its up to users to manage/delete snapshops and copy/delete data to be able to take advantage of the expanded free space properly |
Beta Was this translation helpful? Give feedback.
-
|
So do I understand it correct? |
Beta Was this translation helpful? Give feedback.
-
|
That's not what I said. If zfs didn't support snapshots it would have been possible (even advisable) to automatically rewrite data after an expansion in order to regain space. Since zfs does support snapshots such task cannot be automated because it requires deleting the existing snapshots first. |
Beta Was this translation helpful? Give feedback.
-
|
oh okay, but wouldnt it be an idea to configure it to ask an person if he have no snapshots if he want to rewrite the data and if he have snapshots to ask the person if he want to delete these to do a rewrite? So the person can choose what he want to do... like im planing to add 4 drives to my 4 drives z2 setup and than i would do an rewrite just after i added the last one... or is there any way else to trigger an rewrite without moving the files from the drive to an other and back? because is ne point in not asking to rewrite the data |
Beta Was this translation helpful? Give feedback.
-
|
@Xyz00777 There is just no code implementing the rewrite. People used some user-level scripts to rebalance vdevs, which is quite a similar problem, but it is unsafe to do under load. I am thinking about implementing that on ZFS level, but so far only thinking. |
Beta Was this translation helpful? Give feedback.
-
|
@amotin Yes please! |
Beta Was this translation helpful? Give feedback.
-
|
I was wondering if someone can comment on the state of this feature? The PR is complete, but it's not in OpenZFS 2.2 and it looks like it's slated for 2.3. iX is however shipping this feature in their next release, which is tentatively end of October 2024. Is iX shipping a feature not yet ready for general use? Or is OpenZFS simply holding this back for version reasons waiting on on other changes slated for 2.3? |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/openzfs/zfs/releases/tag/zfs-2.3.0-rc1 |
Beta Was this translation helpful? Give feedback.
-
|
It's not in 2.2, and will be in 2.3. You should probably ask specific vendors what they're shipping, if it's different to what's here. My understanding is that iX are shipping a 2.3 prerelease, but I don't know if they've made other changes. |
Beta Was this translation helpful? Give feedback.
-
|
Not exactly related to raidz expansion, but more so changing the form of a ZFS pool. What are the blockers on allowing users to switch their pool from a mirrored setup, to a radiz setup (either offline, or online). It seems like it should be possible to go from mirror -> raidz, but obviously not in reverse. |
Beta Was this translation helpful? Give feedback.
-
|
While 2-disk mirror and 2-disk RAIDZ might look pretty similar (if not identical) on the physical level, they have completely different block pointers and space accounting, that makes such conversion pretty difficult. |
Beta Was this translation helpful? Give feedback.
-
|
@amotin I thought they were asking about starting with a 2 disk mirror, adding a 3rd blank disk, then converting to a z1 pool using the 3rd disk as parity. This initially seemed rather 'easy'.
But if all the bookkeeping is that different between the VDEV types, that now seems WILDLY more difficult.
Assuming you could repeatedly shrink/grow the VDEVs, this is actually kind of a simple concept. Then again, if it was really that easy, we could have used the same process to add a drive to a raidz array, so I'm assuming that the process simply wouldn't work or that there is some major problem with it that I can't see. |
Beta Was this translation helpful? Give feedback.
-
|
As I have told, 2-disk mirror and 2-disk RAIDZ effectively have the same on-disk data since parity for one block is the block itself. But the problem is that when block pointer is created mirror vdevs are equivalent to single disk, just measuring offsets from the beginning of the disk to its end. RAIDZ same time measure offsets through all the disks in the vdev. The same is with space accounting: mirror capacity is equal to one disk, while RAIDZ -- a sum of all disks. That is why straight conversion would be problematic. But if you just need to convert from 2-disk mirror to 3-disk RAIDZ, you could take one of the disks from mirror, then create new 2-disk RAIDZ pool, copy all the data from the old pool and then add use RAIDZ expansion to get from 2-disks RAIDZ to 3-disk. Two downsides of this scheme is a loss of redundancy during the process and low space efficiency as result of conversion. To fix the last it would be nice if it would be possible to create 3-disk RAIDZ with one disk missing, but it is not supported now, even though was discussed several times and should not be hard. Or it may be possible (I haven't tried) to do the trick via a temporary file instead of one of the disks and delete it just after successful pool creation. |
Beta Was this translation helpful? Give feedback.
-
|
I've upgraded raidz pools many times by first scrubbing the old pool (don't forget the scrub!!!), then taking all redundant disks offline (so the pool is degraded, but can still be used), them used those disks while creating a new raidz including temporary (sparse) files as placeholders for new redundant disks, then taking the sparse files offline (so the new pool is degraded but can still be used), then copying the data from the old pool to the new pool, then deleting the old pool, and using the disks from there to replace the offline disks in the new pool, then resilver. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah in my case I have a 4 disk mirror I wanted to convert to a 4 disk RAIDZ (1 parity, 3 data). Alright, well it seems consensus for this is that I'm probably going to need to just get more disks and just move the data over. |
Beta Was this translation helpful? Give feedback.
-
|
Would it be possible to use a similar approach to implement RAIDZ Shrinking. So if less space is used we can remove a disk permanently without losing parity? |
Beta Was this translation helpful? Give feedback.
-
|
No. Imagine the current version keeps all block numbers when reshuffling all the data. Conceptually, in shrinking the range of numbers decrease and there is no guarantee that all numbers currently in use will be available in the new layout even if you do the process in reverse order. |
Beta Was this translation helpful? Give feedback.
and others
-
This is carrying on from PR #15022, so we don't crowd the PR out with general discussion traffic. 😄
Beta Was this translation helpful? Give feedback.
All reactions