ZoL is missing the "multi_vdev_crash_dump" feature

[FransUrbo]

Is apparently available in the Illumos code...

[behlendorf]

This is an interesting case. In fact, the single vdev crash dump code was explicitly removed from ZoL. Crash dumps under Linux are managed differently than under Illumos and do not rely on any crash dump specific support being provided by the filesystem. Therefore, even without this feature a crash dump kernel built with ZFS support should support single disk, mirror, and raidz configurations. However, to my knowledge no one has tested this.

[magnayn]

Unfortunately I've bashed up against this trying to import an illumos (SmartOS) pool into Linux.

It would seem that since a crash dump was triggered in smartos, the 'feature bit' was toggled to ON. And once it's enabled, there's no going back.

Sadly, even though I don't care at all about the dumps (indeed, the fs has now been deleted), the feature bit remains stubbornly on, so it seems I cannot import that pool into Linux (and my only option seems to be to dump and load to a new zpool, just for a feature bit I never wanted in the first place).

It'd be nice to support it, even if it's nonsensical for linux, if only to provide the ability to import pools from other systems...

[magnayn]

I think this is it: illumos/illumos-gate@810e43b

[behlendorf]

@magnayn That's a good point. For compatibility reason alone we should probably bring in this feature in some form.

[FransUrbo]

Sadly, even though I don't care at all about the dumps (indeed, the fs has now been deleted), the feature bit remains stubbornly on, so it seems I cannot import that pool into Linux (and my only option seems to be to dump and load to a new zpool, just for a feature bit I never wanted in the first place).

With #1050 (or is it #1350 - they both provide basically the same thing), you could always do a send+receive. Couldn't something similar be added to import (as in 'property to ignore')?

We probably need something like that sooner than later anyway....

[magnayn]

Hmm - I could possibly send + receive - I'd like to avoid it as the pool is so big.

I had a brief look at illumos/illumos-gate@810e43b - the patch looks surprisingly small, it's seemingly mostly adding a NOPARITY checksum option.

Perhaps unsurprisingly it won't apply cleanly to master. I'd try re-working it myself but I don't know the ZFS codebase history. I can, however, offer a dataset to test whether it works :). I suspect parts of it aren't even required as they seem to be about actually performing dumps.

[ryao]

@behlendorf My attempt to use kdump failed because Linux 3.14 would refuse to load the initramfs on a crashdump kernel. That being said, I think it might be wise to bring back the old code and just let it stay dormant. We need to support read/write to such pools for cross platform compatibility, even if it buys us nothing.

There were historical patches by SGI to support crash dumps like IRIX, but they were rejected. I imagine that they will appear again at some point. I would not be surprised if I try doing this for Gentoo at some point out of frustration with the existing tools. I imagine that putting the dump code back into ZoL would end up meeting my frustration halfway.

[FransUrbo]

I'm currently trying to backup my whole pool and I choose a FreeNAS on the dump host just for simplicity. Looking through some of it's documentation, I eventually found https://us-east.manta.joyent.com/smartosman/public/man5/zpool-features.5.html (from SmartOS, not FreeNAS but the feature I was looking for was com.delphix:hole_birth). There seems to be quite a lot of features that we're missing that might be 'good to have' ...

Most of them seem to come from Delphix and I don't know where the code might be, but should I create an issue for each of the ones we're missing, or should we update this issue to say something like

ZoL is missing features [existing in other ZFS implementations]

?

[dweeezil]

@FransUrbo Check out #2149.

[FransUrbo]

@dweeezil AWSOME!! Thanx, I'll start testing as soon as I can!

[darkfires]

Is there no way to ignore this feature for a read only recovery? Have a few zpools that were created on illumos that I need to recover somehow...

[behlendorf]

@darkfires you should be able to import the pool read-only using the the latest master code. If this doesn't work can you please report the exact error message your seeing.

[behlendorf]

Unfortunately, this functionality was implemented in a way which doesn't allow it to be read-only compatible. Specifically the addition of the NOPARITY checksum type.

GUID    com.joyent:multi_vdev_crash_dump
READ-ONLY COMPATIBLE    no
DEPENDENCIES    none

[Thomas-1985]

Hello

Is there any progress in implementing this feature flag? I need it to import a ZFS pool that has been created with FreeNAS (9.3.1) on my linux box. The only feature flag that is missing is "multi_vdev_crash_dump" and i thing this would really enhance the cross-platform compatibility of ZFS

Best,
Thomas

[behlendorf]

@Thomas-1985 this hasn't been a priority for us since it's not required for Linux. Although I agree it would be nice to have solely for improved compatibility.

[satmandu]

FYI for what it is worth, the most recent stable build of openzfsonosx appears to have this feature flag as an option:

sudo zpool status
  pool: externalzpool
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub in progress since Sun May  1 15:31:49 2016

sudo zpool upgrade
This system supports ZFS pool feature flags.

All pools are formatted using feature flags.


Some supported features are not enabled on the following pools. Once a
feature is enabled the pool may become incompatible with software
that does not support the feature. See zpool-features(5) for details.

POOL  FEATURE
---------------
externalzpool
      multi_vdev_crash_dump
      sha512
      skein
      edonr

(I haven't upgraded to support those feature flags yet for fear that I won't be able to access this zpool from ZoL, but I imagine new zpools from OS X with a stable openzfsonosx build will have those feature flags enabled by default.)

[darkfires]

FreeBSD supports the flags as well. I actually completely moved all of my ZFS systems to FreeBSD from ZoL because of too many problems and performance issues, also having ZFS natively integrated into the OS makes life so much easier when installing updates. Small things like ZFS's ARC stats included into top just add to the polish. Honestly wish I had done it way sooner than I did. No systemd on FreeBSD is also nice :)

[maaku]

I am now in a position of having my data stuck because of this issue.

For unrelated, irrelevant reasons I'm forced to move a server which has 72TB of storage from FreeBSD (FreeNAS) to GNU/Linux (Ubuntu). The only holdup is that multi_vdev_crash_dump is enabled on the pool.

I do not have a 2nd 72TB storage pool just lying around to do a send & receive from. So the only option seems to be adding support for this flag (or better yet, a tool to remove this flag as the feature was never once used by the pool). Any pointers for how best to do this? I'm not familiar with the ZFS codebase or ZoL so some indication of where to look or how complicated a task this is would be appreciated.

[DeHackEd]

While the implementation is fairly straight-forward, most of the changes are in files that are OS-specific. The vdev_disk and ZVOL drivers are all linux-specific and are where most of the modifications are. It makes porting less straight-forward than the other features since they must be re-implemented.

[behlendorf]

(or better yet, a tool to remove this flag as the feature was never once used by the pool)

@maaku I'd suggest taking a look at PR #3878, it extends the zhack command to allow you to force disable a feature flag. Now keep in mind that this is a developer tool and in general this not a safe thing to do! However, in the case of the multi_vdev_crash_dump clearing this feature flag should be safe and it will allow you to import the pool.

[DeHackEd]

Late to the party, 42db43e should close this issue.

[gmelikov]

@DeHackEd IIRC this commit is a dummy to support pools with this feature, but I think that we can close this issue.

[DeHackEd]

Well, yes. But the pool feature mainly introduces a new dataset property checksum=noparity which is "supported" (3c67d83) weakly from an older commit. This was just getting the feature flag explicitly listed which was the import blocker. Between the two of them Linux can manage with a pool imported from other systems just fine. It just can't read/write to such datasets, which is arguably of little use on Linux anyway since they are crash dumps from different operating systems.