SPL panic after power outage

[D4rk4]

If i try reboot or reimport pool i always take same SPL error:

[ 2484.550076] SPLError: 8669:0:(metaslab.c:1245:metaslab_init()) SPL PANIC
[ 2484.556797] SPL: Showing stack for process 8669
[ 2484.556802] Pid: 8669, comm: zpool Tainted: P IO 3.2.0-4-amd64 #1 Debian 3.2.60-1+deb7u3
[ 2484.556810] Call Trace:
[ 2484.556833] [] ? spl_debug_dumpstack+0x3d/0x45 [spl]
[ 2484.556838] [] ? spl_debug_bug+0x7f/0xc8 [spl]
[ 2484.556844] [] ? spl_PANIC+0x9d/0xc6 [spl]
[ 2484.556849] [] ? kfree+0x5b/0x6c
[ 2484.556854] [] ? kmem_free_debug+0xc5/0x10d [spl]
[ 2484.556885] [] ? space_map_open_impl+0x1c/0x4c [zfs]
[ 2484.556905] [] ? metaslab_init+0x11b/0x23f [zfs]
[ 2484.556924] [] ? vdev_metaslab_init+0x13f/0x1b9 [zfs]
[ 2484.556943] [] ? vdev_load+0x50/0x9a [zfs]
[ 2484.556960] [] ? vdev_load+0x1d/0x9a [zfs]
[ 2484.556979] [] ? spa_load+0xf2e/0x1898 [zfs]
[ 2484.556997] [] ? spa_activate+0x38e/0x3b5 [zfs]
[ 2484.557002] [] ? should_resched+0x5/0x23
[ 2484.557007] [] ? mutex_lock+0xd/0x2d
[ 2484.557025] [] ? spa_load_best+0x181/0x219 [zfs]
[ 2484.557044] [] ? spa_import+0x152/0x5a8 [zfs]
[ 2484.557061] [] ? get_nvlist+0x9b/0xb6 [zfs]
[ 2484.557068] [] ? nvlist_lookup_common+0x4c/0x98 [znvpair]
[ 2484.557084] [] ? zfs_ioc_pool_import+0xb2/0xf2 [zfs]
[ 2484.557100] [] ? zfsdev_ioctl+0x32d/0x3da [zfs]
[ 2484.557105] [] ? do_vfs_ioctl+0x459/0x49a
[ 2484.557109] [] ? __strncpy_from_user+0x18/0x48
[ 2484.557114] [] ? kmem_cache_free+0x2d/0x69
[ 2484.557116] [] ? sys_ioctl+0x4b/0x72
[ 2484.557120] [] ? system_call_fastpath+0x16/0x1b

[behlendorf]

@D4rk4 What version of ZoL are you using?

[D4rk4]

@behlendorf 0.6.3-13cd3939wheezy

[D4rk4]

Any suggestions how I can restore pool? I try import this pool to OpenIndiana, but OpenIndiana wan't import pool with msg 'unsupported features'.

[dswartz]

Any suggestions how I can restore pool? I try import this pool to
OpenIndiana, but OpenIndiana wan't import pool with msg 'unsupported
features'.

sorry i didn't see the earlier posts - have you tried latest&greatest omnios?

[D4rk4]

@dswartz on omnios, all devices online, but pull faulted with 'corrupted data'

[dswartz]

@dswartz on omnios, all devices online, but pull faulted with 'corrupted
data'

Hmmm, no idea then, sorry :(

[D4rk4]

Hmm... me try import this pool via 'zpool import -fF id-of-pool' and.... OmniOS crashed. Look like portable bug in ZFS code.

[ryao]

@D4rk4 Try doing a readonly import with -o readonly=on. You can also try to do transaction rollback manually with a transaction group id from zdb -lu /path/to/vdev/leaf and the -T $TXG_NUM option. Be certain to combine this with a readonly import until you are certain that you are satisfied with it because doing a normal import in this way will rewrite history from that txg commit forward. All changes between that txg commit and the current txg commit will be lost, so it might be useful to binary search the transaction group commit. You can use zdb to perform some of this work in userland test with zdb -t $TXG_NUM -cc -e $POOLNAME. That will traverse all block pointers to ensure that nothing leaked, which is not quite as comprehensive as an actual import. However, it can be done quickly in userland and it should sanity check the area that has a problem right now.

As for why Open Indiana did not support the latest features, you likely are running an old release. Right now, you need to be running Open Indiana hipster if you want to have support for the latest features. The last tagged release is very old, although the features supported in the 0.6.3 release were in Illumos approximately 2 years ago if I recall correctly.

Lastly, would you describe the hardware? In particular, does it use ECC and is this pool on solid state disks? Does this system use ECC? I am trying to reason about how this happened. One rather out there idea is that the system was able to continue operating for a few milliseconds after the loss of power, but it functioned in this state in an undefined way. This might sound far-fetched, but UPS systems used at home take 7 to 14 ms (I forget the exact figure) before switch to battery during a power failure, yet computers seem to continue functioning. My current conjecture is that your hardware continued to function for anything from 10 ms to 100 ms, but the CPU / system memory behaved incorrectly before the system went offline.

[D4rk4]

It's my home server with music, photos, videos from my action camera, CCTV and Debian mirrors. I builded this server on old Tempest i5400PW. This setup only have ECC on RAM. I use 10 desktop grade HDD for RAID-Z, 2 SSD for L2ARC and 2 7k SFF SATA HDD for logs. But I don't buy UPS =( As result I lose pool after power "blinking".

[D4rk4]

@ryao When I try import the pool with 'readonly=on', OmniOS crashed with zfs trace.
'zdb -t $TXG_NUM -cc -e $POOLNAME' successfully started with txg pulled from raidz member disk.
Do I understand correctly that after 'commit forward' with zdb, I'm likely to be able to properly import the pool?

[kernelOfTruth]

@D4rk4 if you're interested - you could try importing the pool with SystemRescueCD and custom built zfs kernel modules: kernelOfTruth/ZFS-for-SystemRescueCD@c0dfcc6
which support more features than 0.6.3

just copy SysRescueCD 4.3.0 to an usb stick and then the modules to the root of the module

after boot run depmod , then modprobe zfs

I've already tried the modules out during restore of my Gentoo installation and they work (built with march=nocona)

hope this helps, if needed

Good luck !

[behlendorf]

@D4rk4, @prakashsurya just put together a patch for Illumos to handle this error case and @FransUrbo ported to to ZoL in #2767. This should help you import your pool if you still need too.

[D4rk4]

Oh.. I destroy pool 2 days ago. Any data loss.

2014-10-06 22:11 GMT+04:00 Brian Behlendorf [email protected]:

@D4rk4 https://github.com/D4rk4, @prakashsurya
https://github.com/prakashsurya just put together a patch for Illumos
to handle this error case and @FransUrbo https://github.com/FransUrbo
ported to to ZoL in #2767 #2767.
This should help you import your pool if you still need too.

—
Reply to this email directly or view it on GitHub
#2745 (comment).

С уважением,
Дмитрий.