spl/zap object mismatch

[cwedgwood]

new shiny everything, aging off, etc

[  315.356717] SPLError: 4042:0:(zap_micro.c:464:zap_lockdir()) VERIFY3((((doi.doi_type) & 0x80) ? ((doi.doi_type) & 0x3f) : dmu_ot[(int)(doi.doi_type)].ot_byteswap) == DMU_BSWAP_ZAP) failed (0 == 4)
[  315.375286] SPLError: 4042:0:(zap_micro.c:464:zap_lockdir()) SPL PANIC
[  315.382200] SPL: Showing stack for process 4042
[  315.387114] Pid: 4042, comm: zfs Tainted: G           O 3.7.4.cw1 #8
[  315.393971] Call Trace:
[  315.396616]  [<ffffffffa00a1406>] spl_debug_dumpstack+0x30/0x32 [spl]
[  315.403603]  [<ffffffffa00a2742>] spl_debug_bug+0x8a/0xcf [spl]
[  315.410013]  [<ffffffffa059d2ef>] zap_lockdir+0xe0/0x867 [zfs]
[  315.416385]  [<ffffffffa059dcbb>] zap_cursor_retrieve+0x5c/0x2cc [zfs]
[  315.423417]  [<ffffffffa056b650>] ? sa_bulk_lookup+0xe3/0xf1 [zfs]
[  315.430053]  [<ffffffffa00a91be>] ? spl_kmem_cache_alloc+0x116b/0x11e4 [spl]
[  315.437654]  [<ffffffffa0598fd6>] zap_value_search+0x9f/0xce [zfs]
[  315.444327]  [<ffffffffa05c98f7>] zfs_obj_to_path_impl+0x126/0x2c3 [zfs]
[  315.451491]  [<ffffffffa056b650>] ? sa_bulk_lookup+0xe3/0xf1 [zfs]
[  315.458186]  [<ffffffffa056b3c1>] ? sa_handle_get+0x44/0x4f [zfs]
[  315.464786]  [<ffffffffa05c4275>] ? zfs_obj_to_stats_impl+0xa1/0xa3 [zfs]
[  315.472014]  [<ffffffffa05c9bb4>] zfs_obj_to_stats+0x95/0xc1 [zfs]
[  315.478754]  [<ffffffffa05ab56c>] zfs_ioc_obj_to_stats+0x6a/0x87 [zfs]
[  315.485782]  [<ffffffffa05b20f2>] zfsdev_ioctl+0xdf/0x175 [zfs]
[  315.492117]  [<ffffffff81118be6>] vfs_ioctl+0x26/0x39
[  315.497602]  [<ffffffff81119455>] do_vfs_ioctl+0x3ba/0x3fd
[  315.503620]  [<ffffffff81064967>] ? finish_task_switch+0x82/0xad
[  315.509966]  [<ffffffff810ae647>] ? trace_hardirqs_off_caller+0xe/0x23
[  315.517024]  [<ffffffff8126bf8a>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[  315.524030]  [<ffffffff811194dd>] sys_ioctl+0x45/0x70
[  315.529454]  [<ffffffff81511f99>] system_call_fastpath+0x16/0x1b

(more details on object to follow)

[cwedgwood]

update

seems specific to zfs diff on certain (not all datasets)

printk of the obj# shows something zdb can't find (odd)

[  502.692756] XXX bad type, obj = 0x9514

# zdb: dmu_bonus_hold(38164) failed, errno 2

removing that assert triggers another one later on:

[  502.696855] SPLError: 5332:0:(zap.c:579:zap_deref_leaf()) VERIFY3(zap->zap_u.zap_fat.zap_phys->zap_magic == 0x2F52AB2ABULL) failed (4402307930368 == 12703150763)
[  502.712172] SPLError: 5332:0:(zap.c:579:zap_deref_leaf()) SPL PANIC
[  502.718896] SPL: Showing stack for process 5332
[  502.723713] Pid: 5332, comm: zfs Tainted: G           O 3.7.4.cw1 #8
[  502.730513] Call Trace:
[  502.733196]  [<ffffffffa00a7406>] spl_debug_dumpstack+0x30/0x32 [spl]
[  502.740139]  [<ffffffffa00a8742>] spl_debug_bug+0x8a/0xcf [spl]
[  502.746625]  [<ffffffffa05c60fd>] zap_deref_leaf+0xe0/0x1ad [zfs]
[  502.753157]  [<ffffffffa05caefc>] ? spl_rw_clear_owner+0x2a/0x2f [zfs]
[  502.760210]  [<ffffffffa05cc6d7>] ? zap_lockdir+0x4c8/0x81b [zfs]
[  502.766767]  [<ffffffffa05c864f>] fzap_cursor_retrieve+0x83/0x298 [zfs]
[  502.773848]  [<ffffffffa05ccd26>] zap_cursor_retrieve+0x113/0x2cc [zfs]
[  502.781007]  [<ffffffffa059a650>] ? sa_bulk_lookup+0xe3/0xf1 [zfs]
[  502.787694]  [<ffffffffa00af1be>] ? spl_kmem_cache_alloc+0x116b/0x11e4 [spl]
[  502.795278]  [<ffffffffa05c7fd6>] zap_value_search+0x9f/0xce [zfs]
[  502.801883]  [<ffffffffa05f88ab>] zfs_obj_to_path_impl+0x126/0x2c3 [zfs]
[  502.809088]  [<ffffffffa059a650>] ? sa_bulk_lookup+0xe3/0xf1 [zfs]
[  502.815792]  [<ffffffffa059a3c1>] ? sa_handle_get+0x44/0x4f [zfs]
[  502.822274]  [<ffffffffa05f3229>] ? zfs_obj_to_stats_impl+0xa1/0xa3 [zfs]
[  502.829684]  [<ffffffffa05f8b68>] zfs_obj_to_stats+0x95/0xc1 [zfs]
[  502.836393]  [<ffffffffa05da520>] zfs_ioc_obj_to_stats+0x6a/0x87 [zfs]
[  502.843403]  [<ffffffffa05e10a6>] zfsdev_ioctl+0xdf/0x175 [zfs]
[  502.849774]  [<ffffffff81118be6>] vfs_ioctl+0x26/0x39
[  502.855263]  [<ffffffff81119455>] do_vfs_ioctl+0x3ba/0x3fd
[  502.861113]  [<ffffffff810ae647>] ? trace_hardirqs_off_caller+0xe/0x23
[  502.868206]  [<ffffffff8126bf8a>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[  502.875282]  [<ffffffff811194dd>] sys_ioctl+0x45/0x70
[  502.880689]  [<ffffffff81511f99>] system_call_fastpath+0x16/0x1b

[behlendorf]

Commit ceb49b0 will cleanly catch this case and return and error to the caller.