Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VERIFY(c < (1ULL << 17) >> 9) failed, PANIC at zio.c:263:zio_data_buf_alloc() on import #2932

Closed
akorn opened this issue Nov 25, 2014 · 42 comments
Milestone

Comments

@akorn
Copy link
Contributor

akorn commented Nov 25, 2014

Hi,

I have a pool that had the "ASSERTION(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT) failed, SPLError: 5408:0:(zio.c:263:zio_data_buf_alloc()) SPL PANIC" problem like in #2678:

kernel: resource.fs.zfs.statechange version=0x0 pool_guid=0xf949f2e99b5a4f92
kernel:  pool_context=0x1 vdev_guid=0x905ba2ae29ed389a vdev_state=0x7 time=[ 
kernel:  0x54744898 0xff08fce ] eid=0x1c
kernel: ASSERTION(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT) failed
kernel: SPLError: 5408:0:(zio.c:263:zio_data_buf_alloc()) SPL PANIC
kernel: SPL: Showing stack for process 5408
kernel: CPU: 4 PID: 5408 Comm: zpool Tainted: P           O 3.10.16-vs2.3.6.6-superguppy #1
kernel: Hardware name: Supermicro H8SGL/H8SGL, BIOS 3.0        08/31/2012
kernel:  0000000000000000 ffff8807f355b358 ffffffff814eb2db ffff8807f355b370
kernel:  ffffffffa010b40b 0000000000000000 ffff8807f355b390 ffffffffa010c65a
kernel:  000000000000003b ffffffffa02b8bc8 ffff8807f355b510 ffffffffa0115e78
kernel: Call Trace:
kernel:  [<ffffffff814eb2db>] dump_stack+0x19/0x1b
kernel:  [<ffffffffa010b40b>] spl_debug_dumpstack+0x46/0x49 [spl]
kernel:  [<ffffffffa010c65a>] spl_debug_bug+0x88/0xcd [spl]
kernel:  [<ffffffffa0115e78>] spl_PANIC+0x9d/0xc6 [spl]
kernel:  [<ffffffff8102e181>] ? __switch_to+0x13d/0x3b8
kernel:  [<ffffffff8108073e>] ? mmdrop+0x11/0x20
kernel:  [<ffffffff81081a10>] ? finish_task_switch+0x7b/0xab
kernel:  [<ffffffffa0262520>] zio_data_buf_alloc+0x3e/0x53 [zfs]
kernel:  [<ffffffffa02076e6>] spa_load_verify_cb+0x93/0x181 [zfs]
kernel:  [<ffffffffa01d5850>] traverse_visitbp+0x332/0x712 [zfs]
kernel:  [<ffffffffa01d5208>] ? traverse_prefetch_metadata+0x3e/0x9b [zfs]
kernel:  [<ffffffffa01d6311>] traverse_dnode+0x9b/0xaa [zfs]
kernel:  [<ffffffffa01d5ae6>] traverse_visitbp+0x5c8/0x712 [zfs]
kernel:  [<ffffffffa01d59bd>] traverse_visitbp+0x49f/0x712 [zfs]
kernel:  [<ffffffffa01d59bd>] traverse_visitbp+0x49f/0x712 [zfs]
kernel:  [<ffffffffa01d59bd>] traverse_visitbp+0x49f/0x712 [zfs]
kernel:  [<ffffffffa01d59bd>] traverse_visitbp+0x49f/0x712 [zfs]
kernel:  [<ffffffffa01d59bd>] traverse_visitbp+0x49f/0x712 [zfs]
kernel:  [<ffffffffa01d59bd>] traverse_visitbp+0x49f/0x712 [zfs]
kernel:  [<ffffffffa01d62d2>] traverse_dnode+0x5c/0xaa [zfs]
kernel:  [<ffffffffa01d5bb3>] traverse_visitbp+0x695/0x712 [zfs]
kernel:  [<ffffffffa01d5f34>] traverse_impl+0x304/0x3ec [zfs]
kernel:  [<ffffffffa01d604b>] traverse_dataset+0x2f/0x31 [zfs]
kernel:  [<ffffffffa0207653>] ? spa_async_suspend+0x8d/0x8d [zfs]
kernel:  [<ffffffffa01d6152>] traverse_pool+0x105/0x150 [zfs]
kernel:  [<ffffffffa0207653>] ? spa_async_suspend+0x8d/0x8d [zfs]
kernel:  [<ffffffffa020deb4>] spa_load+0x169f/0x1a5e [zfs]
kernel:  [<ffffffffa020d546>] spa_load+0xd31/0x1a5e [zfs]
kernel:  [<ffffffffa020a5ef>] ? spa_activate+0x45e/0x481 [zfs]
kernel:  [<ffffffffa020e2dd>] spa_load_best+0x6a/0x242 [zfs]
kernel:  [<ffffffffa0134dc8>] ? zpool_get_rewind_policy+0x4d/0x141 [zcommon]
kernel:  [<ffffffffa020e5d3>] spa_open_common+0x11e/0x317 [zfs]
kernel:  [<ffffffffa020e820>] spa_get_stats+0x39/0x275 [zfs]
kernel:  [<ffffffffa02406e9>] zfs_ioc_pool_stats+0x22/0x57 [zfs]
kernel:  [<ffffffffa02450d5>] zfsdev_ioctl+0x354/0x40c [zfs]
kernel:  [<ffffffff8113fd6f>] vfs_ioctl+0x18/0x34
kernel:  [<ffffffff81140577>] do_vfs_ioctl+0x362/0x41c
kernel:  [<ffffffff81081a10>] ? finish_task_switch+0x7b/0xab
kernel:  [<ffffffff814ee508>] ? __schedule+0x4b3/0x593
kernel:  [<ffffffff81140683>] SyS_ioctl+0x52/0x7d
kernel:  [<ffffffff81052b66>] ? do_page_fault+0x9/0xb
kernel:  [<ffffffff814efd56>] system_call_fastpath+0x1a/0x1f

I rebuilt zfsonlinux (spl as well as zfs) using current git master (so it has the fix from 4a7480a) and tried with that. Now I get a different panic message on import:

[  183.954795] resource.fs.zfs.statechange version=0x0 pool_guid=0xf949f2e99b5a4f92
[  183.961518]  pool_context=0x2 vdev_guid=0x905ba2ae29ed389a vdev_state=0x7 time=[ 
[  183.968563]  0x5474567f 0x1be4365a ] eid=0xf

[  203.365009] VERIFY(c < (1ULL << 17) >> 9) failed
[  203.368489] PANIC at zio.c:263:zio_data_buf_alloc()
[  203.372227] Showing stack for process 5225
[  203.375136] CPU: 3 PID: 5225 Comm: zpool Tainted: P           O 3.10.16-vs2.3.6.6-superguppy #1
[  203.382721] Hardware name: Supermicro H8SGL/H8SGL, BIOS 3.0        08/31/2012
[  203.388714]  0000000000000000 ffff880c07507468 ffffffff814eb2db ffff880c07507478
[  203.395274]  ffffffffa010f603 ffff880c07507610 ffffffffa010f6cb ffff880c07507620
[  203.401865]  ffffffff81089a5d ffff880c00000028 ffff880c07507620 ffff880c075075b8
[  203.408406] Call Trace:
[  203.409665]  [<ffffffff814eb2db>] dump_stack+0x19/0x1b
[  203.413686]  [<ffffffffa010f603>] spl_dumpstack+0x3d/0x3f [spl]
[  203.418483]  [<ffffffffa010f6cb>] spl_panic+0xc6/0xf9 [spl]
[  203.422920]  [<ffffffff81089a5d>] ? load_balance+0xc2/0x61a
[  203.427311]  [<ffffffff81035879>] ? native_sched_clock+0x39/0x3b
[  203.432167]  [<ffffffff8102e181>] ? __switch_to+0x13d/0x3b8
[  203.436558]  [<ffffffff8108073e>] ? mmdrop+0x11/0x20
[  203.440361]  [<ffffffff81081a10>] ? finish_task_switch+0x7b/0xab
[  203.445380]  [<ffffffffa03a571b>] zio_data_buf_alloc+0x3e/0x53 [zfs]
[  203.450699]  [<ffffffffa034ab59>] spa_load_verify_cb+0x93/0x189 [zfs]
[  203.456085]  [<ffffffffa0319917>] traverse_visitbp+0x336/0x722 [zfs]
[  203.461397]  [<ffffffffa02fc23c>] ? arc_read+0xbf4/0xc08 [zfs]
[  203.466187]  [<ffffffffa03a7f33>] ? zio_nowait+0x231/0x27c [zfs]
[  203.471118]  [<ffffffffa031a3f5>] traverse_dnode+0x9b/0xaa [zfs]
[  203.476079]  [<ffffffffa0319bb3>] traverse_visitbp+0x5d2/0x722 [zfs]
[  203.481390]  [<ffffffffa0319a87>] traverse_visitbp+0x4a6/0x722 [zfs]
[  203.486716]  [<ffffffffa0319a87>] traverse_visitbp+0x4a6/0x722 [zfs]
[  203.491981]  [<ffffffffa0319a87>] traverse_visitbp+0x4a6/0x722 [zfs]
[  203.497268]  [<ffffffffa0319a87>] traverse_visitbp+0x4a6/0x722 [zfs]
[  203.502595]  [<ffffffffa0319a87>] traverse_visitbp+0x4a6/0x722 [zfs]
[  203.507920]  [<ffffffffa0319a87>] traverse_visitbp+0x4a6/0x722 [zfs]
[  203.513234]  [<ffffffffa031a3b6>] traverse_dnode+0x5c/0xaa [zfs]
[  203.518212]  [<ffffffffa0319c84>] traverse_visitbp+0x6a3/0x722 [zfs]
[  203.523542]  [<ffffffffa031a00a>] traverse_impl+0x307/0x3f9 [zfs]
[  203.528598]  [<ffffffffa031a12b>] traverse_dataset+0x2f/0x31 [zfs]
[  203.533767]  [<ffffffffa034aac6>] ? spa_async_suspend+0x94/0x94 [zfs]
[  203.539150]  [<ffffffffa031a232>] traverse_pool+0x105/0x150 [zfs]
[  203.544315]  [<ffffffffa034aac6>] ? spa_async_suspend+0x94/0x94 [zfs]
[  203.549794]  [<ffffffffa0351381>] spa_load+0x16b5/0x1a6d [zfs]
[  203.554622]  [<ffffffffa03517a3>] spa_load_best+0x6a/0x242 [zfs]
[  203.559480]  [<ffffffffa0146e12>] ? zpool_get_rewind_policy+0x97/0x141 [zcommon]
[  203.565818]  [<ffffffffa0352dca>] spa_import+0x18b/0x5fb [zfs]
[  203.570607]  [<ffffffffa03835fd>] zfs_ioc_pool_import+0xac/0xf1 [zfs]
[  203.576025]  [<ffffffffa0388231>] zfsdev_ioctl+0x354/0x40c [zfs]
[  203.580850]  [<ffffffff81121036>] ? __cache_free.isra.48+0x1c7/0x1d5
[  203.586026]  [<ffffffff8113fd6f>] vfs_ioctl+0x18/0x34
[  203.589935]  [<ffffffff81140577>] do_vfs_ioctl+0x362/0x41c
[  203.594264]  [<ffffffff8113b4b2>] ? final_putname+0x2f/0x32
[  203.598647]  [<ffffffff8113b617>] ? putname+0x22/0x24
[  203.602514]  [<ffffffff81140683>] SyS_ioctl+0x52/0x7d
[  203.606442]  [<ffffffff81052b66>] ? do_page_fault+0x9/0xb
[  203.610652]  [<ffffffff814efd56>] system_call_fastpath+0x1a/0x1f

Would it make sense to try importing a previous txg using zpool import -T?

Also, fwiw, this box has ECC memory.

@akorn
Copy link
Contributor Author

akorn commented Nov 26, 2014

OK, this is probably all related to the xattr=sa vs. acltype=posixacl problem, as in openzfs/spl#390. Since the issue now occurs on importing the pool, is there any way to recover, perhaps even resorting to a hex editor? I don't have backups because these are the backups. :) I don't mind losing a filesystem or two, but losing the entire pool would be inconvenient.

And no, -T doesn't help -- I tried thousands of txgs (counting back from the one printed by zdb -l), but none of them appear to be importable.

And sorry about re-filing what now looks like a duplicate of openzfs/spl#390 -- I didn't realize this immediately.

@dweeezil
Copy link
Contributor

After finding the root cause of the dnode corruption (typically triggered by xattr=sa and acltype=posixacl), I gave up on trying to craft a workaround. It would be very difficult to catch all forms of corruption and, unfortunately, the handling of spill blocks spans several layers within the code, making a workaround that much more difficult.

@akorn
Copy link
Contributor Author

akorn commented Nov 26, 2014

Does that mean I'm screwed and will have to destroy the pool? Can't I somehow find the corrupt dnodes and corrupt their checksum or something?

Or maybe destroy specific filesystems without importing the pool?

I'm pretty desperate here. :)

@dweeezil
Copy link
Contributor

@akorn Were either or both of the stack traces above the result of zpool import -T (or any of the other forms of rewind)? Is you goal to import the pool for the purposes of recovering the data or is it to restore the pool to a writable state?

Have you tried setting spa_load_verify_metadata=0 and/or spa_load_verify_data=0? Those options control which data are traversed during rewind imports (they're pretty new module options).

@akorn
Copy link
Contributor Author

akorn commented Nov 26, 2014

No, the stack traces are without -T. With -T it doesn't crash, it just says the pool is not importable because some of the devices are missing.

My goal would be to remove the corrupted data (entire corrupted filesystems if necessary, that's not a big deal) but restore the pool to a usable state.

I have tried neither spa_load_verify_metadata=0 nor spa_load_verify_data=0 -- should setting these allow me to import the pool and go on to destroy filesystems? I think I have a fair idea which ones I have to destroy, so that'd be great.

@akorn
Copy link
Contributor Author

akorn commented Nov 26, 2014

Woot, yes, setting those two options allowed me to import the pool!

Is there a systematic way of finding out which filesystems are "bad" or should I take some shots in the, ah, insufficient light? Like I said, I can make informed guesses, but being certain would be even better.

FWIW, I tried to mount all filesystems individually and could mount all but one (attempting to mount that one caused the box to lock up). Does this mean that it (and only it) contains this specific corruption that was preventing the pool from being imported, even with -N? Is it safe to try to destroy it (will it not cause the rest of the pool to become corrupted)?

Also, all of my filesystems have many snapshots. Would it make sense to try to roll them all back to a previous snapshot? Since these are backups, I can easily lose a few.

@akorn
Copy link
Contributor Author

akorn commented Nov 27, 2014

Well, attempting to destroy that filesystem doesn't seem to affect the pool itself very much, but it also doesn't succeed:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000198
IP: [<ffffffffa02d46cd>] metaslab_check_free+0xa0/0x164 [zfs]
PGD c08989067 PUD c088eb067 PMD 0 
Oops: 0000 [#1] SMP 
Modules linked in: netconsole dummy w83795 cfq_iosched zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate i2c_piix4 acpi_cpufreq i2c_core k10temp mperf hwmon serio_raw sp5100_tco amd64_edac_mod edac_core button hid_generic pata_acpi e1000e mpt2sas pata_atiixp ptp raid_class ata_generic sg pps_core scsi_transport_sas [last unloaded: netconsole]
CPU: 6 PID: 5583 Comm: txg_sync Tainted: P           O 3.10.16-vs2.3.6.6-superguppy #1
Hardware name: Supermicro H8SGL/H8SGL, BIOS 3.0        08/31/2012
task: ffff8807f6b82440 ti: ffff8807e3fd8000 task.ti: ffff8807e3fd8000
RIP: 0010:[<ffffffffa02d46cd>]  [<ffffffffa02d46cd>] metaslab_check_free+0xa0/0x164 [zfs]
RSP: 0018:ffff8807e3fdb5b8  EFLAGS: 00010212
RAX: 0000000000000000 RBX: 1800000060000000 RCX: 0000000000000006
RDX: ffff8807f6b82440 RSI: 000000000000007f RDI: ffff880bf9c254a8
RBP: ffff8807e3fdb5f8 R08: 0000000000000000 R09: 0000000000000000
R10: 00000005ed45c240 R11: 000000000000b6b4 R12: ffff880bf149b180
R13: 702e6d6574737973 R14: ffff880bf149b180 R15: 0000000000000000
FS:  00007fc805428700(0000) GS:ffff880c1fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000198 CR3: 0000000c072c7000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
 ffffffffa02e9b71 ffff880bf9c24000 0000000000000000 ffff880bf149b180
 ffff880bf9c24000 ffff8807df0c9ad0 000000000018dc1f 0000000000000000
 ffff8807e3fdb638 ffffffffa0338099 0000000000000000 ffff880bf149b180
Call Trace:
 [<ffffffffa02e9b71>] ? spa_config_held+0x42/0x84 [zfs]
 [<ffffffffa0338099>] zio_free_sync+0xd8/0x19e [zfs]
 [<ffffffffa02c92fe>] dsl_scan_free_block_cb+0x4c/0x1ce [zfs]
 [<ffffffffa029a985>] bptree_visit_cb+0x38/0x115 [zfs]
 [<ffffffffa02aa733>] traverse_visitbp+0x152/0x722 [zfs]
 [<ffffffffa02ab3f5>] traverse_dnode+0x9b/0xaa [zfs]
 [<ffffffffa02aabb3>] traverse_visitbp+0x5d2/0x722 [zfs]
 [<ffffffffa02aaa87>] traverse_visitbp+0x4a6/0x722 [zfs]
 [<ffffffffa02aaa87>] traverse_visitbp+0x4a6/0x722 [zfs]
 [<ffffffffa02aaa87>] traverse_visitbp+0x4a6/0x722 [zfs]
 [<ffffffffa02aaa87>] traverse_visitbp+0x4a6/0x722 [zfs]
 [<ffffffffa02aaa87>] traverse_visitbp+0x4a6/0x722 [zfs]
 [<ffffffffa02aaa87>] traverse_visitbp+0x4a6/0x722 [zfs]
 [<ffffffffa02ab3b6>] traverse_dnode+0x5c/0xaa [zfs]
 [<ffffffffa02aac84>] traverse_visitbp+0x6a3/0x722 [zfs]
 [<ffffffffa00fa7da>] ? kmalloc_nofail+0x29/0x38 [spl]
 [<ffffffffa02ab00a>] traverse_impl+0x307/0x3f9 [zfs]
 [<ffffffffa02ab4e5>] traverse_dataset_destroyed+0x1f/0x21 [zfs]
 [<ffffffffa029a94d>] ? dbuf_stats_destroy+0x4e/0x4e [zfs]
 [<ffffffffa029b019>] bptree_iterate+0x1a7/0x434 [zfs]
 [<ffffffffa02c92b2>] ? dsl_scan_prefetch.isra.1+0x9f/0x9f [zfs]
 [<ffffffffa02cb276>] dsl_scan_sync+0x202/0xc27 [zfs]
 [<ffffffffa0338c11>] ? zio_wait+0x30e/0x31d [zfs]
 [<ffffffffa02dfd6d>] spa_sync+0x59e/0xa97 [zfs]
 [<ffffffff8109a5ec>] ? getrawmonotonic+0x20/0x8b
 [<ffffffffa02f036e>] txg_sync_thread+0x2a9/0x4f8 [zfs]
 [<ffffffff811207fb>] ? ac_put_obj.isra.39+0x12/0x47
 [<ffffffffa02f00c5>] ? txg_fini+0x1f0/0x1f0 [zfs]
 [<ffffffffa00fc4ad>] ? __thread_exit+0x12/0x12 [spl]
 [<ffffffffa00fc51f>] thread_generic_wrapper+0x72/0x7c [spl]
 [<ffffffff81078a90>] kthread+0x88/0x90
 [<ffffffff81078a08>] ? __kthread_parkme+0x60/0x60
 [<ffffffff814efcac>] ret_from_fork+0x7c/0xb0
 [<ffffffff81078a08>] ? __kthread_parkme+0x60/0x60
Code: 41 f7 44 24 20 ff ff ff 00 0f 95 c0 01 d0 41 39 c7 0f 8d b4 00 00 00 41 8b 76 04 48 8b 7d c8 e8 bd ce 01 00 4d 8b 6e 08 49 8b 1e <48> 8b 88 98 01 00 00 48 8b 80 b0 01 00 00 49 c1 e5 09 81 e3 ff 
RIP  [<ffffffffa02d46cd>] metaslab_check_free+0xa0/0x164 [zfs]
 RSP <ffff8807e3fdb5b8>
CR2: 0000000000000198
---[ end trace e4cc8c6988ad5c95 ]---
Kernel panic - not syncing: Fatal exception

Is there any way I can get rid of the corrupt dataset?

@dweeezil
Copy link
Contributor

Now that you've issued the zfs destroy, the filesystem is disconnected and it's not possible to track down the bogus dnode(s). Instead, you should have a list of to-be-freed blocks in the "sync_bplist".

If you want to work with the pool in this state, I'd suggest renaming or removing any existing cache file (zpool.cache) and rebooting and doing a little investigation with zdb and not to try importing the pool right away. In this state, you'll need to use zdb -e -p /dev/disk/by-id ... in order to operate on a non-imported pool with no cache file (replace the /dev path with one that's appropriate to find your pool). I'd start with zdb -e -p /dev -dddd tank 1 and verify that you see "sync_bplist = ". Zdb can display the list with a simple zdb -e -p /dev -ddddd tank | less. Search forward in less for the word "Deferred". It should look like this:

    Deferred frees: object 31, 50 blkptrs, 291K
        0:2c000000:800 4000L/800P F=0 B=43489/43489
        0:2a029400:4000 4000L/4000P F=0 B=43487/43487
        0:21db4200:200 4000L/200P F=0 B=43471/43471
        0:21db3a00:800 1000L/800P F=0 B=43471/43471
        0:2a02e400:4000 4000L/4000P F=0 B=43487/43487
        0:25c9d000:200 1000L/200P F=0 B=43472/43472

Any error you get while zdb produces this list might be useful in developing some sort of workaround. Worst case, I suppose, it might be possible to add a hack to the import code to ignore and clear the deferred free list which would result in the leakage of all the space but should otherwise result in a usable pool.

My recommendation, however, would be to try importing it to the txg prior to the destroy operation. You can use zdb -t <txg> to examine the pool prior to trying an import. If you can get it imported prior to the delete, it should be possible to try using normal filesystem operations to figure out which dnodes are corrupted. The xattr=sa/acltype=posixacl issue would normally only corrupt directories (they typically have a default acl which always requires a spill block). If you can find the corrupted files/directories, and figure out what sort of corruption they have (using my enhanced zdb used in other issues), it might be possible to come up with a workaround of some sort.

@akorn
Copy link
Contributor Author

akorn commented Nov 27, 2014

I issued the destroy but it didn't complete. The filesystem is still there (the kernel panicked before it could destory the fs).

@dweeezil
Copy link
Contributor

Good, then I'd suggest doing something like find <dir> -type f -print0 | xargs -0 md5sum or some such to traverse the whole thing and figure out which files/dirs are corrupted.

@akorn
Copy link
Contributor Author

akorn commented Nov 27, 2014

I can't mount it; mounting it leads to the same kind of panic that attempting to destroy it yields. Maybe that means the rootdir is corrupted?

@dweeezil
Copy link
Contributor

You could try running zdb ... -dddd tank/fs 4 (maybe more -d if necessary). Object 4 should be the root directory.

@akorn
Copy link
Contributor Author

akorn commented Nov 27, 2014

OK, that does work, but I'm not sure what I'm looking for.

The output looks like this:

Dataset tank/fs [ZPL], ID 6255, cr_txg 916955, 30.2G, 239720 objects, rootbp DVA[0]=<0:f00112d8000:4000> DVA[1]=<0:14400a454000:4000> [L0 DMU objset] fletcher4 lzjb LE contiguous unique double size=800L/200P birth=1627404L/1627404P fill=239720 cksum=1699e1264d:704d91ae539:1361227a0f7a5:26b2574327be26

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
         4    2    16K    16K  93.0K    32K  100.00  ZFS directory (K=inherit) (Z=inherit)
                                        292   bonus  System attributes
        dnode flags: USED_BYTES USERUSED_ACCOUNTED 
        dnode maxblkid: 1
        path    /
        uid     289
        gid     550
        atime   Wed Nov 19 07:38:13 2014
        mtime   Tue Nov 18 10:29:01 2014
        ctime   Wed Nov 19 07:38:13 2014
        crtime  Tue Mar 18 21:23:33 2014
        gen     916955
        mode    42770
        size    63
        parent  4
        links   16
        pflags  40800000044
        SA xattrs: 116 bytes, 1 entries

                system.posix_acl_default = \002\000\000\000\001\000\007\000\377\377\377\377\002\000\007\000!\001\000\000\004\000\007\000\377\377\377\377\010\000\007\000&\002\000\000\020\000\007\000\377\377\377\377 \000\000\000\377\377\377\377
        Fat ZAP stats:
                Pointer table:
                        1024 elements
                        zt_blk: 0
                        zt_numblks: 0
                        zt_shift: 10
                        zt_blks_copied: 0
                        zt_nextblk: 0
                ZAP entries: 61
                Leaf blocks: 1
                Total blocks: 2
                zap_block_type: 0x8000000000000001
                zap_magic: 0x2f52ab2ab
                zap_salt: 0xb24b82dc1
                Leafs with 2^n pointers:
                          9:      1 *
                Blocks with n*5 entries:
                          9:      1 *
                Blocks n/10 full:
                          4:      1 *
                Entries with n chunks:
                          3:     21 *********************
                          4:     20 ********************
                          5:     16 ****************
                          6:      4 ****
                Buckets with n entries:
                          0:    454 ****************************************
                          1:     55 *****
                          2:      3 *

                org.jenkinsci.plugins.valgrind.ValgrindPublisher.xml = 685 (type: Regular File)
                nodeMonitors.xml = 682 (type: Regular File)
                hudson.tasks.Mailer.xml = 672 (type: Regular File)
                org.jenkins_ci.plugins.flexible_publish.FlexiblePublisher.xml = 683 (type: Regular File)
                jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml = 680 (type: Regular File)
                Fingerprint cleanup.log = 586 (type: Regular File)
                email-templates = 7 (type: Directory)
                jobs = 11 (type: Directory)
                jenkins.diagnostics.ooom.OutOfOrderBuildMonitor = 9 (type: Directory)
                secrets = 16 (type: Directory)
                hudson.plugins.groovy.Groovy.xml = 360635 (type: Regular File)
                jenkins.model.ArtifactManagerConfiguration.xml = 677 (type: Regular File)
                updates = 17 (type: Directory)
                queue.xml.bak = 687 (type: Regular File)
                com.sonyericsson.rebuild.RebuildDescriptor.xml = 4952 (type: Regular File)
                com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveBuildWrapper.xml = 615 (type: Regular File)
                log = 12 (type: Directory)
                jenkins.security.RekeySecretAdminMonitor = 10 (type: Directory)
                hudson.plugins.ansicolor.AnsiColorBuildWrapper.xml = 131074 (type: Regular File)
                identity.key.enc = 131077 (type: Regular File)
                hudson.scm.SubversionSCM.xml = 670 (type: Regular File)
                hudson.tasks.Ant.xml = 671 (type: Regular File)
                com.foo.jenkins.plugins.dependencymanager.DependencyBuilder.xml = 360634 (type: Regular File)
                hudson.plugins.emailext.ExtendedEmailPublisher.xml = 668 (type: Regular File)
                config.xml = 627 (type: Regular File)
                hudson.tasks.Shell.xml = 674 (type: Regular File)
                .owner = 583 (type: Regular File)
                hudson.plugins.seleniumhq.SeleniumhqBuilder.xml = 311300 (type: Regular File)
                Download metadata.log = 122912 (type: Regular File)
                fingerprints = 8 (type: Directory)
                jenkins.model.DownloadSettings.xml = 131078 (type: Regular File)
                hudson.plugins.analysis.core.GlobalSettings.xml = 666 (type: Regular File)
                hudson.maven.MavenModuleSet.xml = 659 (type: Regular File)
                hudson.model.UpdateCenter.xml = 665 (type: Regular File)
                cache = 200705 (type: Directory)
                secret.key = 689 (type: Regular File)
                Gravatar periodic lookup.log = 592 (type: Regular File)
                Workspace clean-up.log = 603 (type: Regular File)
                jenkins.model.JenkinsLocationConfiguration.xml = 678 (type: Regular File)
                credentials.xml = 628 (type: Regular File)
                com.foo.jenkins.plugins.buildfarmcommon.BuildfarmConfiguration.xml = 360633 (type: Regular File)
                com.froglogic.squish.SquishBuilder.xml = 610 (type: Regular File)
                scm-sync-configuration.xml = 688 (type: Regular File)
                envInject.xml = 651 (type: Regular File)
                hudson.plugins.build_timeout.operations.BuildStepOperation.xml = 131075 (type: Regular File)
                userContent = 18 (type: Directory)
                Out of order build detection.log = 597 (type: Regular File)
                hudson.plugins.depgraph_view.DependencyGraphProperty.xml = 667 (type: Regular File)
                jenkins.security.QueueItemAuthenticatorConfiguration.xml = 681 (type: Regular File)
                Connection Activity monitoring to slaves.log = 584 (type: Regular File)
                hudson.plugins.copyartifact.TriggeredBuildSelector.xml = 131076 (type: Regular File)
                hudson.triggers.SCMTrigger.xml = 675 (type: Regular File)
                monitoring = 14 (type: Directory)
                plugins = 15 (type: Directory)
                hudson.scm.CVSSCM.xml = 669 (type: Regular File)
                jenkins.mvn.GlobalMavenConfig.xml = 679 (type: Regular File)
                hudson.tasks.Maven.xml = 673 (type: Regular File)
                secret.key.not-so-secret = 690 (type: Regular File)
                users = 19 (type: Directory)
                org.jenkinsci.plugins.conditionalbuildstep.singlestep.SingleConditionalBuilder.xml = 684 (type: Regular File)
                logs = 13 (type: Directory)
Indirect blocks:
               0 L1  0:1d8955964000:4000 0:1596225d0000:4000 4000L/400P F=2 B=1618786/1618786
               0  L0 0:1d8955950000:4000 0:1596225c0000:4000 4000L/400P F=1 B=1618786/1618786
            4000  L0 0:1d8955954000:8000 0:159575c20000:8000 4000L/1400P F=1 B=1618786/1618786

                segment [0000000000000000, 0000000000008000) size   32K

I suppose this is not the messed-up entry because it doesn't have a Spill blkptr?

@akorn
Copy link
Contributor Author

akorn commented Nov 27, 2014

Investigating the children of the rootdir, I found some that had SPILL_BLKPTR set; how do I recognize the corrupt one(s)? And what do I do once I find them?

Hmmm, I'm not using your enhanced zdb; I suppose I should try with that, maybe it'll tell me more.

@akorn
Copy link
Contributor Author

akorn commented Nov 27, 2014

@dweeezil , I'm looking at your zdb now (from https://raw.githubusercontent.com/dweeezil/zfs/6efc9d3256d2aec8cd34533776f4bc0e37fc87b0/cmd/zdb/zdb.c), and it appears to be some modifications behind the latest master (e.g. in zdb_leak_init, c and m are uint64_t in master and just int in your version).

Should I attempt to merge these versions somehow, or is it fine to just use your latest version as it is?

@dweeezil
Copy link
Contributor

Your root directory does not appear to be corrupted but I do find it odd that it only has a default ACL and not a regular ACL. Also, the bonus size of 292 is suspiciously close to that at which a spill would be necessary.

Generally speaking, a corrupted dnode will cause zdb to generate some sort of error.

It might be more expedient to put some telemetry in the module such as a printk() of objset and object in traverse_dnode() and then to run the failing import. That ought to help tracking down the corrupted object.

@akorn
Copy link
Contributor Author

akorn commented Nov 27, 2014

OK, assuming I find the corrupted dnode somehow -- what do I do with it then?

@dweeezil
Copy link
Contributor

Run the enhanced ZDB on it with 7-d and, depending on the type of corruption, we'll see if there's a way to craft a workaround.

@akorn
Copy link
Contributor Author

akorn commented Nov 28, 2014

I wrote a quick'n'dirty script to traverse the fs using zdb and GNU parallel.

It found one node where your zdb errors out like this (but it's still running, so there may be more):

zdb: ../../module/zfs/zio.c:247: Assertion `c < (1ULL << 17) >> 9 (0xffff < 0x100)' failed.
Dataset tank/fs [ZPL], ID 6255, cr_txg 916955, 30.2G, 239720 objects, rootbp DVA[0]=<0:f00112d8000:4000> DVA[1]=<0:14400a454000:4000> [L0 DMU objset] fletcher4 lzjb LE contiguous unique double size=800L/200P birth=1627404L/1627404P fill=239720 cksum=1699e1264d:704d91ae539:1361227a0f7a5:26b2574327be26

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
    161656    1    16K     2K  46.5K     2K  100.00  ZFS directory (K=inherit) (Z=inherit)
                                        284   bonus  System attributes
        dnode flags: USED_BYTES USERUSED_ACCOUNTED SPILL_BLKPTR
        dnode maxblkid: 0

        Spill blkptr:
                402653184:5cdacae8e6f2e600:0 2000000L/2000000P F=0 B=4294967295/141733920767
        Spill blkptr dump: \000\000\000\140\000\000\000\030\163\171\163\164\145\155\056\160\157\163\151\170\137\141\143\154\137\144\145\146\141\165\154\164\000\000\000\012\000\000\000\054\002\000\000\000\001\000\007\000\377\377\377\377\004\000\007\000\377\377\377\377\010\000\007\000\046\002\000\000\020\000\007\000\377\377\377\377\040\000\000\000\377\377\377\377\000\000\000\000\000\000\000\000\000\000\000\000\362\177\252\326\016\000\000\000\057\052\351\027\335\005\000\000\046\201\335\242\177\060\001\000\314\303\121\221\061\067\052\000
        SA hdrsize 16
        SA layout 4
        path    /userContent/foo-coverage-html
        uid     1116
        gid     550
        atime   Thu Mar 27 07:17:25 2014
        mtime   Wed Nov 19 08:26:14 2014
        ctime   Wed Nov 19 08:26:14 2014
        crtime  Tue Mar 25 11:42:03 2014
        gen     937659
        mode    42775
        size    3
        parent  18
        links   2
        pflags  40800000044
        ndacl   3

@dweeezil
Copy link
Contributor

I'll need a zdb -dddd 5 6 for the filesystems.

@akorn
Copy link
Contributor Author

akorn commented Nov 28, 2014

Dataset tank/fs [ZPL], ID 6255, cr_txg 916955, 30.2G, 239720 objects, rootbp DVA[0]=<0:f00112d8000:4000> DVA[1]=<0:14400a454000:4000> [L0 DMU objset] fletcher4 lzjb LE contiguous unique double size=800L/200P birth=1627404L/1627404P fill=239720 cksum=1699e1264d:704d91ae539:1361227a0f7a5:26b2574327be26

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
         5    1    16K  1.50K  23.5K  1.50K  100.00  SA attr registration
        dnode flags: USED_BYTES USERUSED_ACCOUNTED 
        dnode maxblkid: 0
        microzap: 1536 bytes, 21 entries

                ZPL_XATTR =  8000009 : [8:0:9]
                ZPL_SCANSTAMP =  20030012 : [32:3:18]
                ZPL_PAD =  2000000e : [32:0:14]
                ZPL_MODE =  8000005 : [8:0:5]
                ZPL_DACL_COUNT =  8000010 : [8:0:16]
                ZPL_GID =  800000d : [8:0:13]
                ZPL_FLAGS =  800000b : [8:0:11]
                ZPL_MTIME =  10000001 : [16:0:1]
                ZPL_CTIME =  10000002 : [16:0:2]
                ZPL_ZNODE_ACL =  5803000f : [88:3:15]
                ZPL_SIZE =  8000006 : [8:0:6]
                ZPL_GEN =  8000004 : [8:0:4]
                ZPL_DXATTR =  30014 : [0:3:20]
                ZPL_DACL_ACES =  40013 : [0:4:19]
                ZPL_CRTIME =  10000003 : [16:0:3]
                ZPL_ATIME =  10000000 : [16:0:0]
                ZPL_LINKS =  8000008 : [8:0:8]
                ZPL_PARENT =  8000007 : [8:0:7]
                ZPL_RDEV =  800000a : [8:0:10]
                ZPL_SYMLINK =  30011 : [0:3:17]
                ZPL_UID =  800000c : [8:0:12]
Dataset tank/fs [ZPL], ID 6255, cr_txg 916955, 30.2G, 239720 objects, rootbp DVA[0]=<0:f00112d8000:4000> DVA[1]=<0:14400a454000:4000> [L0 DMU objset] fletcher4 lzjb LE contiguous unique double size=800L/200P birth=1627404L/1627404P fill=239720 cksum=1699e1264d:704d91ae539:1361227a0f7a5:26b2574327be26

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
         6    1    16K    16K  46.5K    32K  100.00  SA attr layouts
        dnode flags: USED_BYTES USERUSED_ACCOUNTED 
        dnode maxblkid: 1
        Fat ZAP stats:
                Pointer table:
                        1024 elements
                        zt_blk: 0
                        zt_numblks: 0
                        zt_shift: 10
                        zt_blks_copied: 0
                        zt_nextblk: 0
                ZAP entries: 4
                Leaf blocks: 1
                Total blocks: 2
                zap_block_type: 0x8000000000000001
                zap_magic: 0x2f52ab2ab
                zap_salt: 0xe3d50b6e5
                Leafs with 2^n pointers:
                          9:      1 *
                Blocks with n*5 entries:
                          0:      1 *
                Blocks n/10 full:
                          1:      1 *
                Entries with n chunks:
                          3:      1 *
                          4:      3 ***
                Buckets with n entries:
                          0:    508 ****************************************
                          1:      4 *

                3 = [ 5  6  4  12  13  7  11  0  1  2  3  8  16  19  17 ]
                4 = [ 5  6  4  12  13  7  11  0  1  2  3  8  16  19  20 ]
                5 = [ 20 ]
                2 = [ 5  6  4  12  13  7  11  0  1  2  3  8  16  19 ]

Thanks for all the help, by the way!

@dweeezil
Copy link
Contributor

@akorn I'm thinking the best solution to allow you to delete your corrupted filesystems is going to be to simply ignore all spill blocks and live with a bit of leaked space (the spill blocks would be orphaned).

I've worked up one of the most gross hacks I can possibly imagine in https://github.com/dweeezil/zfs/tree/ignore-spill. It's extremely heavy-handed and is intended only for the purpose of importing a corrupted pool and running zpool destroy on an affected filesystem.

I ran did just that on a test filesystem containing a single spill block and afterward, a zdb shows:

leaked space: vdev 0, offset 0x3ee00, size 512
leaked space: vdev 0, offset 0x28038a00, size 512
block traversal size 663040 != alloc 664064 (leaked 1024)

which is exactly what I'd expect. You'll lose 2 blocks for every spill block (they're duped since they're considered to be metadata).

I'd like to know whether you've found any other corrupted dnodes before you try running this patch. If the corruption is all of the same type, then this may just do the trick.

@akorn
Copy link
Contributor Author

akorn commented Nov 29, 2014

So far there is only the one (corrupted dnode), and the recursive traversal is almost complete: 218137 objects examined of 231259 objects discovered so far -- of course it's just possible it'll stumble on a high-degree directory...

How safe would you estimate your gross hack is? I can certainly live with some leaked space, but hosing the rest of the pool would be bad. :)

Also, as of current master, is xattr=sa, acltype=posixacl believed to be safe?

@dweeezil
Copy link
Contributor

I'd suggest importing the pool with zpool import -o readonly=on and testing the broken directory. In particular, I'd expect getfattr -d -m - <dir> to produce a sensible result. I suppose you could use getfacl <dir> instead.

Given that the only SA layout on the filesystem with a small number of SA's is "5 = [20]" and that 20 is DXATTR, the only thing spill blocks are likely used for are xattrs. This means the gross hack should be pretty safe.

@dweeezil
Copy link
Contributor

dweeezil commented Dec 6, 2014

@akorn Just checking in. Any progress with this?

@akorn
Copy link
Contributor Author

akorn commented Dec 6, 2014

Thanks for keeping tabs on it; I got sidetracked but haven't forgotten. I'll try your gross hack in the following 3-4 days and I'll certainly report back.

Incidentally, as of current master, is xattr=sa, acltype=posixacl believed to be safe?

@dweeezil
Copy link
Contributor

dweeezil commented Dec 7, 2014

@akorn In my opinion, current master code which contains 4254acb should be safe. I'll note, however, that the bug fixed by this patch is not directly related to either xattr=sa or acltype=posix. That said, the problem it fixes is very very unlikely, if not impossible to happen without those settings because plain-old ZFS simply doesn't do very much with SAs.

@akorn
Copy link
Contributor Author

akorn commented Dec 8, 2014

@dweeezil I can't seem to compile your gross hack.

In file included from /usr/src/spl/include/sys/types.h:34:0,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/avl.h:38,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/spa.h:30,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/arc.c:130:
/usr/src/spl/include/linux/time_compat.h:35:1: error: redefinition of ‘timespec_sub’
 timespec_sub(struct timespec lhs, struct timespec rhs)
 ^
In file included from include/linux/stat.h:18:0,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/avl.h:38,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/spa.h:30,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/arc.c:130:
include/linux/time.h:78:31: note: previous definition of ‘timespec_sub’ was here
 static inline struct timespec timespec_sub(struct timespec lhs,
                               ^
In file included from /usr/src/spl/include/sys/types.h:35:0,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/avl.h:38,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/spa.h:30,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/arc.c:130:
/usr/src/spl/include/linux/bitops_compat.h:32:19: error: redefinition of ‘fls64’
 static inline int fls64(__u64 x)
                   ^
In file included from include/linux/bitops.h:22:0,
                 from include/linux/kernel.h:10,
                 from include/linux/cache.h:4,
                 from include/linux/time.h:4,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/avl.h:38,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/spa.h:30,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/arc.c:130:
/usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/bitops.h:489:28: note: previous definition of ‘fls64’ was here
 static __always_inline int fls64(__u64 x)
                            ^
In file included from /usr/src/spl/include/sys/types.h:34:0,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/zfs_context.h:37,
                 from /usr/src/dweeezil-hacked-zfs/module/zpios/../../module/zpios/pios.c:34:
/usr/src/spl/include/linux/time_compat.h:35:1: error: redefinition of ‘timespec_sub’
 timespec_sub(struct timespec lhs, struct timespec rhs)
 ^
In file included from include/linux/stat.h:18:0,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/zfs_context.h:37,
                 from /usr/src/dweeezil-hacked-zfs/module/zpios/../../module/zpios/pios.c:34:
include/linux/time.h:78:31: note: previous definition of ‘timespec_sub’ was here
 static inline struct timespec timespec_sub(struct timespec lhs,
                               ^
In file included from /usr/src/spl/include/sys/types.h:35:0,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/zfs_context.h:37,
                 from /usr/src/dweeezil-hacked-zfs/module/zpios/../../module/zpios/pios.c:34:
/usr/src/spl/include/linux/bitops_compat.h:32:19: error: redefinition of ‘fls64’
 static inline int fls64(__u64 x)
                   ^
In file included from include/linux/bitops.h:22:0,
                 from include/linux/kernel.h:10,
                 from include/linux/cache.h:4,
                 from include/linux/time.h:4,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/zfs_context.h:37,
                 from /usr/src/dweeezil-hacked-zfs/module/zpios/../../module/zpios/pios.c:34:
/usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/bitops.h:489:28: note: previous definition of ‘fls64’ was here
 static __always_inline int fls64(__u64 x)
                            ^
In file included from /usr/src/spl/include/sys/types.h:34:0,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/dmu.h:41,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/bpobj.h:29,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/bpobj.c:26:
/usr/src/spl/include/linux/time_compat.h:35:1: error: redefinition of ‘timespec_sub’
 timespec_sub(struct timespec lhs, struct timespec rhs)
 ^
In file included from include/linux/stat.h:18:0,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/dmu.h:41,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/bpobj.h:29,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/bpobj.c:26:
include/linux/time.h:78:31: note: previous definition of ‘timespec_sub’ was here
 static inline struct timespec timespec_sub(struct timespec lhs,
                               ^
In file included from /usr/src/spl/include/sys/types.h:35:0,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/dmu.h:41,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/bpobj.h:29,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/bpobj.c:26:
/usr/src/spl/include/linux/bitops_compat.h:32:19: error: redefinition of ‘fls64’
 static inline int fls64(__u64 x)
                   ^
In file included from include/linux/bitops.h:22:0,
                 from include/linux/kernel.h:10,
                 from include/linux/cache.h:4,
                 from include/linux/time.h:4,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/dmu.h:41,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/bpobj.h:29,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/bpobj.c:26:
/usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/bitops.h:489:28: note: previous definition of ‘fls64’ was here
 static __always_inline int fls64(__u64 x)
                            ^
In file included from /usr/src/spl/include/sys/types.h:34:0,
                 from /usr/src/dweeezil-hacked-zfs/module/unicode/../../module/unicode/u8_textprep.c:38:
/usr/src/spl/include/linux/time_compat.h:35:1: error: redefinition of ‘timespec_sub’
 timespec_sub(struct timespec lhs, struct timespec rhs)
 ^
In file included from include/linux/stat.h:18:0,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/module/unicode/../../module/unicode/u8_textprep.c:38:
include/linux/time.h:78:31: note: previous definition of ‘timespec_sub’ was here
 static inline struct timespec timespec_sub(struct timespec lhs,
                               ^
In file included from /usr/src/spl/include/sys/types.h:35:0,
                 from /usr/src/dweeezil-hacked-zfs/module/unicode/../../module/unicode/u8_textprep.c:38:
/usr/src/spl/include/linux/bitops_compat.h:32:19: error: redefinition of ‘fls64’
 static inline int fls64(__u64 x)
                   ^
In file included from include/linux/bitops.h:22:0,
                 from include/linux/kernel.h:10,
                 from include/linux/cache.h:4,
                 from include/linux/time.h:4,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/module/unicode/../../module/unicode/u8_textprep.c:38:
/usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/bitops.h:489:28: note: previous definition of ‘fls64’ was here
 static __always_inline int fls64(__u64 x)
                            ^
In file included from /usr/src/spl/include/sys/types.h:34:0,
                 from /usr/src/dweeezil-hacked-zfs/module/avl/../../module/avl/avl.c:90:
/usr/src/spl/include/linux/time_compat.h:35:1: error: redefinition of ‘timespec_sub’
 timespec_sub(struct timespec lhs, struct timespec rhs)
 ^
In file included from include/linux/stat.h:18:0,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/module/avl/../../module/avl/avl.c:90:
include/linux/time.h:78:31: note: previous definition of ‘timespec_sub’ was here
 static inline struct timespec timespec_sub(struct timespec lhs,
                               ^
In file included from /usr/src/spl/include/sys/types.h:35:0,
                 from /usr/src/dweeezil-hacked-zfs/module/avl/../../module/avl/avl.c:90:
/usr/src/spl/include/linux/bitops_compat.h:32:19: error: redefinition of ‘fls64’
 static inline int fls64(__u64 x)
                   ^
In file included from include/linux/bitops.h:22:0,
                 from include/linux/kernel.h:10,
                 from include/linux/cache.h:4,
                 from include/linux/time.h:4,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/module/avl/../../module/avl/avl.c:90:
/usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/bitops.h:489:28: note: previous definition of ‘fls64’ was here
 static __always_inline int fls64(__u64 x)
                            ^
In file included from /usr/src/spl/include/sys/types.h:34:0,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/nvpair.h:29,
                 from /usr/src/dweeezil-hacked-zfs/module/nvpair/../../module/nvpair/nvpair.c:30:
/usr/src/spl/include/linux/time_compat.h:35:1: error: redefinition of ‘timespec_sub’
 timespec_sub(struct timespec lhs, struct timespec rhs)
 ^
In file included from include/uapi/linux/timex.h:56:0,
                 from include/linux/timex.h:56,
                 from include/linux/sched.h:17,
                 from /usr/src/spl/include/spl-debug.h:46,
                 from /usr/src/spl/include/sys/debug.h:49,
                 from /usr/src/dweeezil-hacked-zfs/module/nvpair/../../module/nvpair/nvpair.c:27:
include/linux/time.h:78:31: note: previous definition of ‘timespec_sub’ was here
 static inline struct timespec timespec_sub(struct timespec lhs,
                               ^
In file included from /usr/src/spl/include/sys/types.h:35:0,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/nvpair.h:29,
                 from /usr/src/dweeezil-hacked-zfs/module/nvpair/../../module/nvpair/nvpair.c:30:
/usr/src/spl/include/linux/bitops_compat.h:32:19: error: redefinition of ‘fls64’
 static inline int fls64(__u64 x)
                   ^
In file included from include/linux/bitops.h:22:0,
                 from include/linux/kernel.h:10,
                 from include/linux/sched.h:15,
                 from /usr/src/spl/include/spl-debug.h:46,
                 from /usr/src/spl/include/sys/debug.h:49,
                 from /usr/src/dweeezil-hacked-zfs/module/nvpair/../../module/nvpair/nvpair.c:27:
/usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/bitops.h:489:28: note: previous definition of ‘fls64’ was here
 static __always_inline int fls64(__u64 x)
                            ^
In file included from /usr/src/spl/include/sys/kmem.h:38:0,
                 from /usr/src/spl/include/sys/condvar.h:31,
                 from /usr/src/spl/include/sys/t_lock.h:31,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/zfs_context.h:38,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/spa.h:31,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/arc.c:130:
/usr/src/spl/include/sys/vmsystm.h:79:8: error: redefinition of ‘struct vmalloc_info’
 struct vmalloc_info {
        ^
In file included from /usr/src/spl/include/sys/kmem.h:38:0,
                 from /usr/src/spl/include/sys/condvar.h:31,
                 from /usr/src/spl/include/sys/t_lock.h:31,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/zfs_context.h:38,
                 from /usr/src/dweeezil-hacked-zfs/module/zpios/../../module/zpios/pios.c:34:
/usr/src/spl/include/sys/vmsystm.h:79:8: error: redefinition of ‘struct vmalloc_info’
 struct vmalloc_info {
        ^
In file included from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/io.h:195:0,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/realmode.h:5,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/acpi.h:32,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/fixmap.h:19,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/apic.h:12,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/smp.h:13,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/mmzone_64.h:10,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/mmzone.h:4,
                 from include/linux/mmzone.h:919,
                 from include/linux/gfp.h:4,
                 from include/linux/kmod.h:22,
                 from include/linux/module.h:13,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/avl.h:38,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/spa.h:30,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/arc.c:130:
include/linux/vmalloc.h:173:8: note: originally defined here
 struct vmalloc_info {
        ^
In file included from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/io.h:195:0,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/realmode.h:5,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/acpi.h:32,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/fixmap.h:19,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/apic.h:12,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/smp.h:13,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/mmzone_64.h:10,
                 from /usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/mmzone.h:4,
                 from include/linux/mmzone.h:919,
                 from include/linux/gfp.h:4,
                 from include/linux/kmod.h:22,
                 from include/linux/module.h:13,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/zfs_context.h:37,
                 from /usr/src/dweeezil-hacked-zfs/module/zpios/../../module/zpios/pios.c:34:
include/linux/vmalloc.h:173:8: note: originally defined here
 struct vmalloc_info {
        ^
In file included from /usr/src/spl/include/sys/types.h:34:0,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/zfs_context.h:37,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/bplist.h:28,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/bplist.c:26:
/usr/src/spl/include/linux/time_compat.h:35:1: error: redefinition of ‘timespec_sub’
 timespec_sub(struct timespec lhs, struct timespec rhs)
 ^
In file included from include/linux/stat.h:18:0,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/zfs_context.h:37,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/bplist.h:28,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/bplist.c:26:
include/linux/time.h:78:31: note: previous definition of ‘timespec_sub’ was here
 static inline struct timespec timespec_sub(struct timespec lhs,
                               ^
In file included from /usr/src/spl/include/sys/types.h:35:0,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/zfs_context.h:37,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/bplist.h:28,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/bplist.c:26:
/usr/src/spl/include/linux/bitops_compat.h:32:19: error: redefinition of ‘fls64’
 static inline int fls64(__u64 x)
                   ^
In file included from include/linux/bitops.h:22:0,
                 from include/linux/kernel.h:10,
                 from include/linux/cache.h:4,
                 from include/linux/time.h:4,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from /usr/src/spl/include/sys/sysmacros.h:28,
                 from /usr/src/spl/include/sys/types.h:29,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/zfs_context.h:37,
                 from /usr/src/dweeezil-hacked-zfs/include/sys/bplist.h:28,
                 from /usr/src/dweeezil-hacked-zfs/module/zfs/../../module/zfs/bplist.c:26:
/usr/src/linux-headers-3.10.16-vs2.3.6.6-superguppy/arch/x86/include/asm/bitops.h:489:28: note: previous definition of ‘fls64’ was here
 static __always_inline int fls64(__u64 x)
                            ^
scripts/Makefile.build:308: recipe for target '/usr/src/dweeezil-hacked-zfs/module/avl/../../module/avl/avl.o' failed
make[5]: *** [/usr/src/dweeezil-hacked-zfs/module/avl/../../module/avl/avl.o] Error 1
scripts/Makefile.build:455: recipe for target '/usr/src/dweeezil-hacked-zfs/module/avl' failed
make[4]: *** [/usr/src/dweeezil-hacked-zfs/module/avl] Error 2
make[4]: *** Waiting for unfinished jobs....

(There are more redefinitions.)

Maybe this hacked version requires a specific kernel? Or I'm doing something wrong?

I configured it with

./configure --prefix=/usr/local/stow/dweeezil-hacked-zfs.issue2932 --with-spl=/usr/src/spl --with-spl-obj=/var/lib/dkms/spl/0.6.3/3.10.16-vs2.3.6.6-superguppy/x86_64

(/usr/src/spl is the source I built spl from)

@dweeezil
Copy link
Contributor

dweeezil commented Dec 8, 2014

@akorn How did you prepare the source code? Was it a straight checkout of the ignore-spill branch from https://github.com/dweeezil/zfs or did you try applying the dweeezil/zfs@35660b3 commit to some other tree? If the latter, I'd expect it not to work very well (nor to even apply cleanly for that matter). The former should work properly and appears to still be based on master code as of this moment.

@akorn
Copy link
Contributor Author

akorn commented Dec 8, 2014

@dweeezil Um. Alas, I suck at git. I went to the URL you provided (https://github.com/dweeezil/zfs/tree/ignore-spill) and copied the URL from the field on the right (https://github.com/dweeezil/zfs.git) into a git clone command line.

Looking at it now, it seems obvious that I didn't get the branch I should have.

I now went and did a git checkout ignore-spill.

git status showed:

~src/dweeezil-hacked-zfs (git)-[ignore-spill] % git status
On branch ignore-spill
Your branch is up-to-date with 'origin/ignore-spill'.
nothing to commit, working directory clean

Next, I ran ./autogen.sh (which printed a couple hundred warnings about subdir-objects being disabled).

Then, I ran configure again, followed by make.

And I got the same error message as before.

I imagine I'm missing something obvious, but I don't know what.

@dweeezil
Copy link
Contributor

@akorn I'm not sure at the moment why you can't compile the module but the errors you're seeing don't have anything to do with my patch. I suspect you'd not be able to compile a current master checkout, either. Is there a chance your currently-installed spl isn't new enough?

@akorn
Copy link
Contributor Author

akorn commented Dec 11, 2014

My spl is master as of 2014-11-25.

I suspect the problem arises somehow due to my mixing a debianized installation of spl-dkms (built from the pkg-spl git tree), with the checkout directory I built the .deb from, and a complete source installation of your zfs branch. I don't even begin to understand the build process well enough to see how and why it breaks.

FWIW, I have now (manually) applied your patch to the zfs master as of 25 November and am building a .deb from that. Applying the patch was straightforward, but I can't deny a certain amount of trepidation. :)

@akorn
Copy link
Contributor Author

akorn commented Dec 11, 2014

Well; destroying the fs worked.

I imported the pool with -N, destroyed the errant fs, then reverted to the zfs version without your patch, rebooted, imported the pool again, mounted all filesystems, then attempted a scrub.

I now have this in dmesg:

[ 1091.530116] VERIFY3(0 == dp->dp_free_dir->dd_phys->dd_used_bytes) failed (0 == 75852288)
[ 1091.537051] PANIC at dsl_scan.c:1564:dsl_scan_sync()
[ 1091.540798] Showing stack for process 6652
[ 1091.543661] CPU: 5 PID: 6652 Comm: txg_sync Tainted: P           O 3.10.16-vs2.3.6.6-superguppy #1
[ 1091.551449] Hardware name: Supermicro H8SGL/H8SGL, BIOS 3.0        08/31/2012
[ 1091.557358]  0000000000000000 ffff8807eaa5bb60 ffffffff814eb2db ffff8807eaa5bb70
[ 1091.563657]  ffffffffa00ef603 ffff8807eaa5bd08 ffffffffa00ef6cb ffff8807eaa5bd18
[ 1091.570012]  ffffffff81035426 ffff880700000030 ffff8807eaa5bd18 ffff8807eaa5bcb0
[ 1091.576331] Call Trace:
[ 1091.577558]  [<ffffffff814eb2db>] dump_stack+0x19/0x1b
[ 1091.581489]  [<ffffffffa00ef603>] spl_dumpstack+0x3d/0x3f [spl]
[ 1091.586235]  [<ffffffffa00ef6cb>] spl_panic+0xc6/0xf9 [spl]
[ 1091.590597]  [<ffffffff81035426>] ? read_tsc+0x9/0x19
[ 1091.594434]  [<ffffffff811207fb>] ? ac_put_obj.isra.39+0x12/0x47
[ 1091.599219]  [<ffffffff81121036>] ? __cache_free.isra.48+0x1c7/0x1d5
[ 1091.604388]  [<ffffffffa00ec414>] ? spl_kmem_cache_free+0x135/0x140 [spl]
[ 1091.610103]  [<ffffffffa02d95de>] dsl_scan_sync+0x56a/0xc27 [zfs]
[ 1091.615110]  [<ffffffffa0346c11>] ? zio_wait+0x30e/0x31d [zfs]
[ 1091.619860]  [<ffffffffa02edd6d>] spa_sync+0x59e/0xa97 [zfs]
[ 1091.624290]  [<ffffffff8109a5ec>] ? getrawmonotonic+0x20/0x8b
[ 1091.628942]  [<ffffffffa02fe36e>] txg_sync_thread+0x2a9/0x4f8 [zfs]
[ 1091.633980]  [<ffffffff811207fb>] ? ac_put_obj.isra.39+0x12/0x47
[ 1091.638887]  [<ffffffffa02fe0c5>] ? txg_fini+0x1f0/0x1f0 [zfs]
[ 1091.643505]  [<ffffffffa00ed4ad>] ? __thread_exit+0x12/0x12 [spl]
[ 1091.648385]  [<ffffffffa00ed51f>] thread_generic_wrapper+0x72/0x7c [spl]
[ 1091.653881]  [<ffffffff81078a90>] kthread+0x88/0x90
[ 1091.657532]  [<ffffffff81080000>] ? ftrace_raw_output_sched_kthread_stop+0x35/0x49
[ 1091.663890]  [<ffffffff81078a08>] ? __kthread_parkme+0x60/0x60
[ 1091.668517]  [<ffffffff814efcac>] ret_from_fork+0x7c/0xb0
[ 1091.672700]  [<ffffffff81078a08>] ? __kthread_parkme+0x60/0x60

Any suggestions?

@dweeezil
Copy link
Contributor

@akorn That's one of the groups of asserts my GH removed. There still may be a way to get this pool patched up. I guess I was a bit optimistic that a zfs destroy (which I erroneously referred to as zpool destroy in my previous note) would leave the pool in a sane state.

It would be interesting to see what the $FREE pseudo-filesystem contains and also whether you've got the $LEAK present by running zdb -dddd <pool> 4 (4 is usually correct but you may have to fish around for the first child map) and make note of the "$FREE =" object and whether there's a "$LEAK =" object. In each case, run a zdb -ddddd <pool> <obj> on the $FREE object and also on the $LEAK object if it exist. This should give us an idea of the state of things.

@akorn
Copy link
Contributor Author

akorn commented Dec 12, 2014

@dweeezil Yes, I realize I'm hitting an assertion your patch removed; I had hoped/thought removing the ASSERT would only be needed while I destroy the corrupt fs.

zdb output:

# zdb -dddd pool 4
Dataset mos [META], ID 0, cr_txg 4, 93.6G, 56759 objects, rootbp DVA[0]=<0:190014f2c000:4000> DVA[1]=<0:f0000798000:4000> DVA[2]=<0:1440084c4000:4000> [L0 DMU objset] fletcher4 lz4 LE contiguous unique triple size=800L/200P birth=1670983L/1670983P fill=56759 cksum=99cba611b:3f03eebc348:d218d235faa5:1db2157de33f7a

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
         4    1    16K     1K  35.0K     1K  100.00  DSL directory child map
        dnode flags: USED_BYTES 
        dnode maxblkid: 0
        microzap: 1024 bytes, 15 entries

[...]
                $MOS = 5 
[...]
                $ORIGIN = 12 
[...]
                $FREE = 8 
[...]
# zdb -ddddd pool 8
Dataset mos [META], ID 0, cr_txg 4, 93.6G, 56759 objects, rootbp DVA[0]=<0:190014f2c000:4000> DVA[1]=<0:f0000798000:4000> DVA[2]=<0:1440084c4000:4000> [L0 DMU objset] fletcher4 lz4 LE contiguous unique triple size=800L/200P birth=1670983L/1670983P fill=56759 cksum=99cba611b:3f03eebc348:d218d235faa5:1db2157de33f7a

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
         8    1    16K    512      0    512    0.00  DSL directory
                                        256   bonus  DSL directory
        dnode flags: 
        dnode maxblkid: 0
                creation_time = Mon May 27 13:54:40 2013
                head_dataset_obj = 0
                parent_dir_obj = 2
                origin_obj = 0
                child_dir_zapobj = 10
                used_bytes = 72.3M
                compressed_bytes = 1.56M
                uncompressed_bytes = 1.56M
                quota = 0
                reserved = 0
                props_zapobj = 9
                deleg_zapobj = 0
                flags = 1
                used_breakdown[HEAD] = 72.3M
                used_breakdown[SNAP] = 0
                used_breakdown[CHILD] = 0
                used_breakdown[CHILD_RSRV] = 0
                used_breakdown[REFRSRV] = 0
Indirect blocks:

Thanks for not giving up on this, btw. :)

@akorn
Copy link
Contributor Author

akorn commented Dec 29, 2014

I have a new error message on mounting a specific fs:

[  146.336516] VERIFY3(0 == (((dn->dn_phys)->dn_flags & (1<<0)) ? (dn->dn_phys)->dn_used : (dn->dn_phys)->dn_used << 9)) failed (0 == 23808)
[  146.347812] PANIC at dnode_sync.c:505:dnode_sync_free()
[  146.351862] Showing stack for process 5295
[  146.354806] CPU: 6 PID: 5295 Comm: txg_sync Tainted: P           O 3.10.16-vs2.3.6.6-superguppy #1
[  146.362749] Hardware name: Supermicro H8SGL/H8SGL, BIOS 3.0        08/31/2012
[  146.368775]  0000000000000000 ffff8807f73db9c0 ffffffff814eb2db ffff8807f73db9d0
[  146.375415]  ffffffffa0126603 ffff8807f73dbb68 ffffffffa01266cb ffff8807f73dbb78
[  146.382020]  ffff8807f73db9f8 ffffffff00000030 ffff8807f73dbb78 ffff8807f73dbb10
[  146.388624] Call Trace:
[  146.389901]  [<ffffffff814eb2db>] dump_stack+0x19/0x1b
[  146.393909]  [<ffffffffa0126603>] spl_dumpstack+0x3d/0x3f [spl]
[  146.398696]  [<ffffffffa01266cb>] spl_panic+0xc6/0xf9 [spl]
[  146.403139]  [<ffffffff81121036>] ? __cache_free.isra.48+0x1c7/0x1d5
[  146.408325]  [<ffffffff811207fb>] ? ac_put_obj.isra.39+0x12/0x47
[  146.413172]  [<ffffffff81121036>] ? __cache_free.isra.48+0x1c7/0x1d5
[  146.418361]  [<ffffffffa012274c>] ? kmem_free_debug+0x11/0x13 [spl]
[  146.423605]  [<ffffffffa02b9a97>] dnode_sync+0x6f1/0x10b1 [zfs]
[  146.428366]  [<ffffffff8108147b>] ? should_resched+0x9/0x28
[  146.432799]  [<ffffffff814ee800>] ? _cond_resched+0x9/0x19
[  146.437201]  [<ffffffffa02a86f3>] dmu_objset_sync_dnodes+0x148/0x158 [zfs]
[  146.443022]  [<ffffffffa02a891e>] dmu_objset_sync+0x21b/0x2e3 [zfs]
[  146.448273]  [<ffffffffa02bb1a3>] dsl_dataset_sync+0xd3/0xda [zfs]
[  146.453443]  [<ffffffffa02c8c06>] dsl_pool_sync+0xcc/0x584 [zfs]
[  146.458407]  [<ffffffffa02e374d>] spa_sync+0x505/0xa97 [zfs]
[  146.462909]  [<ffffffff8109a5ec>] ? getrawmonotonic+0x20/0x8b
[  146.467599]  [<ffffffffa02f3de7>] txg_sync_thread+0x2a9/0x4f8 [zfs]
[  146.472683]  [<ffffffff811207fb>] ? ac_put_obj.isra.39+0x12/0x47
[  146.477637]  [<ffffffffa02f3b3e>] ? txg_fini+0x1f0/0x1f0 [zfs]
[  146.482300]  [<ffffffffa01244ad>] ? __thread_exit+0x12/0x12 [spl]
[  146.487265]  [<ffffffffa012451f>] thread_generic_wrapper+0x72/0x7c [spl]
[  146.492838]  [<ffffffff81078a90>] kthread+0x88/0x90
[  146.496530]  [<ffffffff81080000>] ? ftrace_raw_output_sched_kthread_stop+0x35/0x49
[  146.502966]  [<ffffffff81078a08>] ? __kthread_parkme+0x60/0x60
[  146.507662]  [<ffffffff814efcac>] ret_from_fork+0x7c/0xb0
[  146.511878]  [<ffffffff81078a08>] ? __kthread_parkme+0x60/0x60

@cointer
Copy link

cointer commented Jan 19, 2015

@dweeezil Quick question regarding this bug. I have had this problem in the past (my post is referenced in the first post in this thread) but have subsequently destroyed any related pools. I am, however, still using xattr=sa and acltype=posixacl.

I do need posixacl set, but is this error mitigated by setting xattr to default instead of sa? Or is this problem still possible when acltype=posixacl is set without xattr=sa?

Just looking to stop this error from happening again altogether until a fix is out.

@dweeezil
Copy link
Contributor

@cointer AFAIK, the last of the SA problems should be fixed by 4254acb which is not in any current release tag. I suppose some packages might have included it in their point releases but I'd not know about that.

@cointer
Copy link

cointer commented Jan 19, 2015

@dweeezil Thanks, good to know. I'm not in a position where I will be able to update the ZFS build for a while though. So, in the meantime, will rsyncing to a dataset with acltype=posixacl and xattr=dir be sufficient to avoid this potential problem?

@dweeezil
Copy link
Contributor

@cointer Yes, but the correct option is xattr=on .

@behlendorf behlendorf modified the milestones: 0.6.5, 0.6.4 Feb 5, 2015
@behlendorf behlendorf modified the milestones: 0.7.0, 0.6.5 Jul 16, 2015
@alexanderhaensch
Copy link

Dont know if this is related, but i have a similar problem with 0.6.5 during a scrub.
The scrub is deadlocked and zfs commands do not work any more after this:

[64345.893387] VERIFY3(mintxg == dsl_dataset_phys(ds1)->ds_prev_snap_txg) failed (6614665 == 6614099)
[64345.893391] PANIC at dsl_scan.c:983:dsl_scan_ds_clone_swapped()
[64345.893392] Showing stack for process 6430
[64345.893395] CPU: 6 PID: 6430 Comm: txg_sync Tainted: P O 3.14.51-hardened #1
[64345.893396] Hardware name: Supermicro X9DR3-F/X9DR3-F, BIOS 3.0b 06/30/2014
[64345.893398] 00000000000003d7 ffff882024bdb918 ffffffff815f25a5 ffffffffa0ad68df
[64345.893401] ffff882024bdb928 ffffffffa07790ef ffff882024bdbaa8 ffffffffa07791a7
[64345.893402] ffff88201e672000 ffff882000000030 ffff882024bdbab8 ffff882024bdba58
[64345.893404] Call Trace:
[64345.893427] [] dump_stack+0x45/0x56
[64345.893473] [] ? _fini+0x2ce36/0x7b2a3 [zfs]
[64345.893480] [] spl_dumpstack+0x3f/0x50 [spl]
[64345.893486] [] spl_panic+0xa7/0xe0 [spl]
[64345.893495] [] ? dmu_buf_rele+0x9/0x10 [zfs]
[64345.893510] [] ? zap_unlockdir+0x5d/0x110 [zfs]
[64345.893524] [] ? zap_lookup_norm+0xbe/0x1b0 [zfs]
[64345.893537] [] ? zap_lookup+0x2e/0x30 [zfs]
[64345.893551] [] ? zap_lookup_int_key+0x4d/0x60 [zfs]
[64345.893566] [] dsl_scan_ds_clone_swapped+0xc0/0x470 [zfs]
[64345.893578] [] dsl_dataset_clone_swap_sync_impl+0x757/0x830 [zfs]
[64345.893590] [] dmu_recv_end_sync+0xbe/0x400 [zfs]
[64345.893605] [] dsl_sync_task_sync+0x12a/0x130 [zfs]
[64345.893618] [] dsl_pool_sync+0x493/0x5e0 [zfs]
[64345.893634] [] spa_sync+0x365/0xde0 [zfs]
[64345.893649] [] txg_sync_thread+0x3ba/0x690 [zfs]
[64345.893665] [] ? txg_thread_wait+0xd0/0xd0 [zfs]
[64345.893670] [] thread_generic_wrapper+0x75/0xb0 [spl]
[64345.893675] [] ? __thread_exit+0x20/0x20 [spl]
[64345.893681] [] kthread+0xc4/0xe0
[64345.893683] [] ? kthread_create_on_node+0x160/0x160
[64345.893687] [] ret_from_fork+0x49/0x80
[64345.893689] [] ? kthread_create_on_node+0x160/0x160

@behlendorf
Copy link
Contributor

Closing. This should be resolved in the current master branch.

@alexanderhaensch your issue looks a little different, can you open a new issue for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants