dmu_zfetch_run() panic

[ikozhukhov]

System information

Type	Version/Name
Distribution Name	DilOS
Distribution Version
Linux Kernel
Architecture
ZFS Version
SPL Version

Describe the problem you're observing

we have 4 vm as vmware guests under vmware esxi and 2 vm as bhyve guests with sporadic/periodic panics with the same stack trace below in different scenarios. we have no panic in the same place, but we can see instability and just revert of commit 891568c (issue #11652) fixed this issue.

Describe how to reproduce the problem

ZTS tests on several different vms in loop

Include any warning/errors/backtraces from the system logs

panic[cpu0]/thread=fffffe0c1836b7e0:
mutex_enter: bad mutex, lp=fffffe0d4bb6a340 owner=deadbeefdeadbee8 thread=fffffe0c1836b7e0


fffffe001116b920 unix:mutex_panic+4a ()
fffffe001116b990 unix:mutex_vector_enter+3a7 ()
fffffe001116ba50 zfs:dmu_zfetch_run+ad ()
fffffe001116bb10 zfs:dmu_buf_hold_array_by_dnode+482 ()
fffffe001116bbd0 zfs:dmu_read_uio_dnode+86 ()
fffffe001116bc40 zfs:dmu_read_uio_dbuf+6f ()
fffffe001116bd20 zfs:zfs_read+5a0 ()
fffffe001116bdc0 genunix:fop_read+fd ()
fffffe001116bef0 genunix:pread+1de ()
fffffe001116bf00 unix:brand_sys_syscall+32f ()

dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel + curproc

[ikozhukhov]

cc @behlendorf @ahrens

[behlendorf]

We haven't observed this on either Linux or FreeBSD to my knowledge. However, from the stack you've posted it looks like the zstream_t pointer being passed to dmu_zfetch_run() is referencing a freed zfetch_t structure. I'm not quite sure how that's possible, but if I had to make a guess I'd wager the issue is in dnode_move()->dnode_move_impl()->dmu_zfetch_fini(). This move functionality is disabled on both Linux and FreeBSD so this isn't a call path which would have been tested.

You could test this theory easily enough by commenting out the dnode_move() registration in dnode_init() for the purposes of a test.

cc: @amotin

[amotin]

The move functionality looks like a one big bug in context of zfetch. I've looked on it briefly while tried to debug previously reported issue, but then gave up, since the issue appeared to be different, while this code is indeed not use on either Linux or FreeBSD. It would be good to check whether the dnode_move() code is indeed related and then try to sort it out somehow.

[amotin]

@ikozhukhov Could you please test the linked PR?

[ikozhukhov]

@ikozhukhov Could you please test the linked PR?

will do it, thanks.
i have to finish some build fixes with zstd on sparc and will return to this and others my PRs