Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ZVOL minors related crash (i think) #4370

Closed
sempervictus opened this issue Feb 26, 2016 · 2 comments
Closed

ZVOL minors related crash (i think) #4370

sempervictus opened this issue Feb 26, 2016 · 2 comments

Comments

@sempervictus
Copy link
Contributor

At first glance, looks like zil replay isnt playing too well with the zvol_create_minor functionality from 4346, which i believe was recently merged into master.
Tried to import a pool after an SCST crash and subsequent reboot, and got this fine mess:

[Thu Feb 25 23:37:48 2016] INFO: task zpool:4651 blocked for more than 180 seconds.
[Thu Feb 25 23:37:48 2016]       Tainted: P           OE   4.4.2-sv-i7 #sv
[Thu Feb 25 23:37:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:37:48 2016] zpool           D ffff880c21003960     0  4651   4037 0x00000002
[Thu Feb 25 23:37:48 2016]  ffff880c21003960 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:37:48 2016]  ffff880c241d4000 ffff880c2139a000 ffff880c2139a3f8 0000000000000000
[Thu Feb 25 23:37:48 2016]  0000000000000003 ffff880c21003938 ffffffff820a1372 ffff880c21004000
[Thu Feb 25 23:37:48 2016] Call Trace:
[Thu Feb 25 23:37:48 2016]  [<ffffffff820a1372>] ? default_wake_function+0x12/0x20
[Thu Feb 25 23:37:48 2016]  [<ffffffff8279e635>] schedule+0x35/0xa0
[Thu Feb 25 23:37:48 2016]  [<ffffffffc00aa16d>] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:37:48 2016]  [<ffffffff820a7340>] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:37:48 2016]  [<ffffffffc00aa1b5>] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc03aecb9>] txg_wait_synced+0xe9/0x130 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc03f105a>] zil_replay+0xca/0x100 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc0400673>] __zvol_create_minor+0x663/0x670 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc0401193>] zvol_create_minor+0x33/0x70 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc04011de>] zvol_create_minors_cb+0xe/0x20 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc0362fea>] dmu_objset_find_impl+0xfa/0x3e0 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc03630a7>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc03630a7>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc0363319>] dmu_objset_find+0x49/0x70 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc0401397>] zvol_create_minors+0x47/0x60 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc03a2f54>] spa_import+0x464/0x730 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc03d1acc>] zfs_ioc_pool_import+0xdc/0x110 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc03d4328>] zfsdev_ioctl+0x458/0x490 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffff821ee81d>] do_vfs_ioctl+0x2cd/0x4b0
[Thu Feb 25 23:37:48 2016]  [<ffffffff821ea7c3>] ? putname+0x53/0x60
[Thu Feb 25 23:37:48 2016]  [<ffffffff821eea79>] SyS_ioctl+0x79/0x90
[Thu Feb 25 23:37:48 2016]  [<ffffffff827a2036>] entry_SYSCALL_64_fastpath+0x16/0x75
[Thu Feb 25 23:37:48 2016] INFO: task txg_quiesce:5854 blocked for more than 180 seconds.
[Thu Feb 25 23:37:48 2016]       Tainted: P           OE   4.4.2-sv-i7 #sv
[Thu Feb 25 23:37:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:37:48 2016] txg_quiesce     D ffff88060e26fd48     0  5854      2 0x00000000
[Thu Feb 25 23:37:48 2016]  ffff88060e26fd48 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:37:48 2016]  ffff880c241e0000 ffff880607446000 ffff8806074463f8 ffff880c2062ab40
[Thu Feb 25 23:37:48 2016]  0000000000000246 0000000000000003 0000000000000001 ffff88060e270000
[Thu Feb 25 23:37:48 2016] Call Trace:
[Thu Feb 25 23:37:48 2016]  [<ffffffff8279e635>] schedule+0x35/0xa0
[Thu Feb 25 23:37:48 2016]  [<ffffffffc00aa16d>] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:37:48 2016]  [<ffffffff820a7340>] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:37:48 2016]  [<ffffffffc00aa1b5>] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc03af21e>] txg_quiesce_thread+0x2ce/0x3e0 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc03aef50>] ? txg_delay+0x160/0x160 [zfs]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc00a52f1>] thread_generic_wrapper+0x71/0x80 [spl]
[Thu Feb 25 23:37:48 2016]  [<ffffffffc00a5280>] ? __thread_exit+0x20/0x20 [spl]
[Thu Feb 25 23:37:48 2016]  [<ffffffff82095a59>] kthread+0xc9/0xe0
[Thu Feb 25 23:37:48 2016]  [<ffffffff82095990>] ? kthread_park+0x60/0x60
[Thu Feb 25 23:37:48 2016]  [<ffffffff827a23cf>] ret_from_fork+0x3f/0x70
[Thu Feb 25 23:37:48 2016]  [<ffffffff82095990>] ? kthread_park+0x60/0x60
[Thu Feb 25 23:40:48 2016] INFO: task zpool:4651 blocked for more than 180 seconds.
[Thu Feb 25 23:40:48 2016]       Tainted: P           OE   4.4.2-sv-i7 #sv
[Thu Feb 25 23:40:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:40:48 2016] zpool           D ffff880c21003960     0  4651   4037 0x00000002
[Thu Feb 25 23:40:48 2016]  ffff880c21003960 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:40:48 2016]  ffff880c241d4000 ffff880c2139a000 ffff880c2139a3f8 0000000000000000
[Thu Feb 25 23:40:48 2016]  0000000000000003 ffff880c21003938 ffffffff820a1372 ffff880c21004000
[Thu Feb 25 23:40:48 2016] Call Trace:
[Thu Feb 25 23:40:48 2016]  [<ffffffff820a1372>] ? default_wake_function+0x12/0x20
[Thu Feb 25 23:40:48 2016]  [<ffffffff8279e635>] schedule+0x35/0xa0
[Thu Feb 25 23:40:48 2016]  [<ffffffffc00aa16d>] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:40:48 2016]  [<ffffffff820a7340>] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:40:48 2016]  [<ffffffffc00aa1b5>] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc03aecb9>] txg_wait_synced+0xe9/0x130 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc03f105a>] zil_replay+0xca/0x100 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc0400673>] __zvol_create_minor+0x663/0x670 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc0401193>] zvol_create_minor+0x33/0x70 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc04011de>] zvol_create_minors_cb+0xe/0x20 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc0362fea>] dmu_objset_find_impl+0xfa/0x3e0 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc03630a7>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc03630a7>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc0363319>] dmu_objset_find+0x49/0x70 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc0401397>] zvol_create_minors+0x47/0x60 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc03a2f54>] spa_import+0x464/0x730 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc03d1acc>] zfs_ioc_pool_import+0xdc/0x110 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc03d4328>] zfsdev_ioctl+0x458/0x490 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffff821ee81d>] do_vfs_ioctl+0x2cd/0x4b0
[Thu Feb 25 23:40:48 2016]  [<ffffffff821ea7c3>] ? putname+0x53/0x60
[Thu Feb 25 23:40:48 2016]  [<ffffffff821eea79>] SyS_ioctl+0x79/0x90
[Thu Feb 25 23:40:48 2016]  [<ffffffff827a2036>] entry_SYSCALL_64_fastpath+0x16/0x75
[Thu Feb 25 23:40:48 2016] INFO: task txg_quiesce:5854 blocked for more than 180 seconds.
[Thu Feb 25 23:40:48 2016]       Tainted: P           OE   4.4.2-sv-i7 #sv
[Thu Feb 25 23:40:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:40:48 2016] txg_quiesce     D ffff88060e26fd48     0  5854      2 0x00000000
[Thu Feb 25 23:40:48 2016]  ffff88060e26fd48 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:40:48 2016]  ffff880c241e0000 ffff880607446000 ffff8806074463f8 ffff880c2062ab40
[Thu Feb 25 23:40:48 2016]  0000000000000246 0000000000000003 0000000000000001 ffff88060e270000
[Thu Feb 25 23:40:48 2016] Call Trace:
[Thu Feb 25 23:40:48 2016]  [<ffffffff8279e635>] schedule+0x35/0xa0
[Thu Feb 25 23:40:48 2016]  [<ffffffffc00aa16d>] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:40:48 2016]  [<ffffffff820a7340>] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:40:48 2016]  [<ffffffffc00aa1b5>] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc03af21e>] txg_quiesce_thread+0x2ce/0x3e0 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc03aef50>] ? txg_delay+0x160/0x160 [zfs]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc00a52f1>] thread_generic_wrapper+0x71/0x80 [spl]
[Thu Feb 25 23:40:48 2016]  [<ffffffffc00a5280>] ? __thread_exit+0x20/0x20 [spl]
[Thu Feb 25 23:40:48 2016]  [<ffffffff82095a59>] kthread+0xc9/0xe0
[Thu Feb 25 23:40:48 2016]  [<ffffffff82095990>] ? kthread_park+0x60/0x60
[Thu Feb 25 23:40:48 2016]  [<ffffffff827a23cf>] ret_from_fork+0x3f/0x70
[Thu Feb 25 23:40:48 2016]  [<ffffffff82095990>] ? kthread_park+0x60/0x60
[Thu Feb 25 23:43:48 2016] INFO: task zpool:4651 blocked for more than 180 seconds.
[Thu Feb 25 23:43:48 2016]       Tainted: P           OE   4.4.2-sv-i7 #sv
[Thu Feb 25 23:43:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:43:48 2016] zpool           D ffff880c21003960     0  4651   4037 0x00000002
[Thu Feb 25 23:43:48 2016]  ffff880c21003960 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:43:48 2016]  ffff880c241d4000 ffff880c2139a000 ffff880c2139a3f8 0000000000000000
[Thu Feb 25 23:43:48 2016]  0000000000000003 ffff880c21003938 ffffffff820a1372 ffff880c21004000
[Thu Feb 25 23:43:48 2016] Call Trace:
[Thu Feb 25 23:43:48 2016]  [<ffffffff820a1372>] ? default_wake_function+0x12/0x20
[Thu Feb 25 23:43:48 2016]  [<ffffffff8279e635>] schedule+0x35/0xa0
[Thu Feb 25 23:43:48 2016]  [<ffffffffc00aa16d>] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:43:48 2016]  [<ffffffff820a7340>] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:43:48 2016]  [<ffffffffc00aa1b5>] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc03aecb9>] txg_wait_synced+0xe9/0x130 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc03f105a>] zil_replay+0xca/0x100 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc0400673>] __zvol_create_minor+0x663/0x670 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc0401193>] zvol_create_minor+0x33/0x70 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc04011de>] zvol_create_minors_cb+0xe/0x20 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc0362fea>] dmu_objset_find_impl+0xfa/0x3e0 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc03630a7>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc03630a7>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc0363319>] dmu_objset_find+0x49/0x70 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc0401397>] zvol_create_minors+0x47/0x60 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc03a2f54>] spa_import+0x464/0x730 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc03d1acc>] zfs_ioc_pool_import+0xdc/0x110 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc03d4328>] zfsdev_ioctl+0x458/0x490 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffff821ee81d>] do_vfs_ioctl+0x2cd/0x4b0
[Thu Feb 25 23:43:48 2016]  [<ffffffff821ea7c3>] ? putname+0x53/0x60
[Thu Feb 25 23:43:48 2016]  [<ffffffff821eea79>] SyS_ioctl+0x79/0x90
[Thu Feb 25 23:43:48 2016]  [<ffffffff827a2036>] entry_SYSCALL_64_fastpath+0x16/0x75
[Thu Feb 25 23:43:48 2016] INFO: task txg_quiesce:5854 blocked for more than 180 seconds.
[Thu Feb 25 23:43:48 2016]       Tainted: P           OE   4.4.2-sv-i7 #sv
[Thu Feb 25 23:43:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:43:48 2016] txg_quiesce     D ffff88060e26fd48     0  5854      2 0x00000000
[Thu Feb 25 23:43:48 2016]  ffff88060e26fd48 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:43:48 2016]  ffff880c241e0000 ffff880607446000 ffff8806074463f8 ffff880c2062ab40
[Thu Feb 25 23:43:48 2016]  0000000000000246 0000000000000003 0000000000000001 ffff88060e270000
[Thu Feb 25 23:43:48 2016] Call Trace:
[Thu Feb 25 23:43:48 2016]  [<ffffffff8279e635>] schedule+0x35/0xa0
[Thu Feb 25 23:43:48 2016]  [<ffffffffc00aa16d>] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:43:48 2016]  [<ffffffff820a7340>] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:43:48 2016]  [<ffffffffc00aa1b5>] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc03af21e>] txg_quiesce_thread+0x2ce/0x3e0 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc03aef50>] ? txg_delay+0x160/0x160 [zfs]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc00a52f1>] thread_generic_wrapper+0x71/0x80 [spl]
[Thu Feb 25 23:43:48 2016]  [<ffffffffc00a5280>] ? __thread_exit+0x20/0x20 [spl]
[Thu Feb 25 23:43:48 2016]  [<ffffffff82095a59>] kthread+0xc9/0xe0
[Thu Feb 25 23:43:48 2016]  [<ffffffff82095990>] ? kthread_park+0x60/0x60
[Thu Feb 25 23:43:48 2016]  [<ffffffff827a23cf>] ret_from_fork+0x3f/0x70
[Thu Feb 25 23:43:48 2016]  [<ffffffff82095990>] ? kthread_park+0x60/0x60
[Thu Feb 25 23:46:48 2016] INFO: task zpool:4651 blocked for more than 180 seconds.
[Thu Feb 25 23:46:48 2016]       Tainted: P           OE   4.4.2-sv-i7 #sv
[Thu Feb 25 23:46:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:46:48 2016] zpool           D ffff880c21003960     0  4651   4037 0x00000002
[Thu Feb 25 23:46:48 2016]  ffff880c21003960 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:46:48 2016]  ffff880c241d4000 ffff880c2139a000 ffff880c2139a3f8 0000000000000000
[Thu Feb 25 23:46:48 2016]  0000000000000003 ffff880c21003938 ffffffff820a1372 ffff880c21004000
[Thu Feb 25 23:46:48 2016] Call Trace:
[Thu Feb 25 23:46:48 2016]  [<ffffffff820a1372>] ? default_wake_function+0x12/0x20
[Thu Feb 25 23:46:48 2016]  [<ffffffff8279e635>] schedule+0x35/0xa0
[Thu Feb 25 23:46:48 2016]  [<ffffffffc00aa16d>] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:46:48 2016]  [<ffffffff820a7340>] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:46:48 2016]  [<ffffffffc00aa1b5>] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc03aecb9>] txg_wait_synced+0xe9/0x130 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc03f105a>] zil_replay+0xca/0x100 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc0400673>] __zvol_create_minor+0x663/0x670 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc0401193>] zvol_create_minor+0x33/0x70 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc04011de>] zvol_create_minors_cb+0xe/0x20 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc0362fea>] dmu_objset_find_impl+0xfa/0x3e0 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc03630a7>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc03630a7>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc0363319>] dmu_objset_find+0x49/0x70 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc0401397>] zvol_create_minors+0x47/0x60 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc03a2f54>] spa_import+0x464/0x730 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc03d1acc>] zfs_ioc_pool_import+0xdc/0x110 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc03d4328>] zfsdev_ioctl+0x458/0x490 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffff821ee81d>] do_vfs_ioctl+0x2cd/0x4b0
[Thu Feb 25 23:46:48 2016]  [<ffffffff821ea7c3>] ? putname+0x53/0x60
[Thu Feb 25 23:46:48 2016]  [<ffffffff821eea79>] SyS_ioctl+0x79/0x90
[Thu Feb 25 23:46:48 2016]  [<ffffffff827a2036>] entry_SYSCALL_64_fastpath+0x16/0x75
[Thu Feb 25 23:46:48 2016] INFO: task txg_quiesce:5854 blocked for more than 180 seconds.
[Thu Feb 25 23:46:48 2016]       Tainted: P           OE   4.4.2-sv-i7 #sv
[Thu Feb 25 23:46:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:46:48 2016] txg_quiesce     D ffff88060e26fd48     0  5854      2 0x00000000
[Thu Feb 25 23:46:48 2016]  ffff88060e26fd48 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:46:48 2016]  ffff880c241e0000 ffff880607446000 ffff8806074463f8 ffff880c2062ab40
[Thu Feb 25 23:46:48 2016]  0000000000000246 0000000000000003 0000000000000001 ffff88060e270000
[Thu Feb 25 23:46:48 2016] Call Trace:
[Thu Feb 25 23:46:48 2016]  [<ffffffff8279e635>] schedule+0x35/0xa0
[Thu Feb 25 23:46:48 2016]  [<ffffffffc00aa16d>] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:46:48 2016]  [<ffffffff820a7340>] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:46:48 2016]  [<ffffffffc00aa1b5>] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc03af21e>] txg_quiesce_thread+0x2ce/0x3e0 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc03aef50>] ? txg_delay+0x160/0x160 [zfs]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc00a52f1>] thread_generic_wrapper+0x71/0x80 [spl]
[Thu Feb 25 23:46:48 2016]  [<ffffffffc00a5280>] ? __thread_exit+0x20/0x20 [spl]
[Thu Feb 25 23:46:48 2016]  [<ffffffff82095a59>] kthread+0xc9/0xe0
[Thu Feb 25 23:46:48 2016]  [<ffffffff82095990>] ? kthread_park+0x60/0x60
[Thu Feb 25 23:46:48 2016]  [<ffffffff827a23cf>] ret_from_fork+0x3f/0x70
[Thu Feb 25 23:46:48 2016]  [<ffffffff82095990>] ? kthread_park+0x60/0x60
[Thu Feb 25 23:49:48 2016] INFO: task zpool:4651 blocked for more than 180 seconds.
[Thu Feb 25 23:49:48 2016]       Tainted: P           OE   4.4.2-sv-i7 #sv
[Thu Feb 25 23:49:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:49:48 2016] zpool           D ffff880c21003960     0  4651   4037 0x00000002
[Thu Feb 25 23:49:48 2016]  ffff880c21003960 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:49:48 2016]  ffff880c241d4000 ffff880c2139a000 ffff880c2139a3f8 0000000000000000
[Thu Feb 25 23:49:48 2016]  0000000000000003 ffff880c21003938 ffffffff820a1372 ffff880c21004000
[Thu Feb 25 23:49:48 2016] Call Trace:
[Thu Feb 25 23:49:48 2016]  [<ffffffff820a1372>] ? default_wake_function+0x12/0x20
[Thu Feb 25 23:49:48 2016]  [<ffffffff8279e635>] schedule+0x35/0xa0
[Thu Feb 25 23:49:48 2016]  [<ffffffffc00aa16d>] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:49:48 2016]  [<ffffffff820a7340>] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:49:48 2016]  [<ffffffffc00aa1b5>] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc03aecb9>] txg_wait_synced+0xe9/0x130 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc03f105a>] zil_replay+0xca/0x100 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc0400673>] __zvol_create_minor+0x663/0x670 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc0401193>] zvol_create_minor+0x33/0x70 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc04011de>] zvol_create_minors_cb+0xe/0x20 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc0362fea>] dmu_objset_find_impl+0xfa/0x3e0 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc03630a7>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc03630a7>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc04011d0>] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc0363319>] dmu_objset_find+0x49/0x70 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc0401397>] zvol_create_minors+0x47/0x60 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc03a2f54>] spa_import+0x464/0x730 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc03d1acc>] zfs_ioc_pool_import+0xdc/0x110 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc03d4328>] zfsdev_ioctl+0x458/0x490 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffff821ee81d>] do_vfs_ioctl+0x2cd/0x4b0
[Thu Feb 25 23:49:48 2016]  [<ffffffff821ea7c3>] ? putname+0x53/0x60
[Thu Feb 25 23:49:48 2016]  [<ffffffff821eea79>] SyS_ioctl+0x79/0x90
[Thu Feb 25 23:49:48 2016]  [<ffffffff827a2036>] entry_SYSCALL_64_fastpath+0x16/0x75
[Thu Feb 25 23:49:48 2016] INFO: task txg_quiesce:5854 blocked for more than 180 seconds.
[Thu Feb 25 23:49:48 2016]       Tainted: P           OE   4.4.2-sv-i7 #sv
[Thu Feb 25 23:49:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:49:48 2016] txg_quiesce     D ffff88060e26fd48     0  5854      2 0x00000000
[Thu Feb 25 23:49:48 2016]  ffff88060e26fd48 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:49:48 2016]  ffff880c241e0000 ffff880607446000 ffff8806074463f8 ffff880c2062ab40
[Thu Feb 25 23:49:48 2016]  0000000000000246 0000000000000003 0000000000000001 ffff88060e270000
[Thu Feb 25 23:49:48 2016] Call Trace:
[Thu Feb 25 23:49:48 2016]  [<ffffffff8279e635>] schedule+0x35/0xa0
[Thu Feb 25 23:49:48 2016]  [<ffffffffc00aa16d>] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:49:48 2016]  [<ffffffff820a7340>] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:49:48 2016]  [<ffffffffc00aa1b5>] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc03af21e>] txg_quiesce_thread+0x2ce/0x3e0 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc03aef50>] ? txg_delay+0x160/0x160 [zfs]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc00a52f1>] thread_generic_wrapper+0x71/0x80 [spl]
[Thu Feb 25 23:49:48 2016]  [<ffffffffc00a5280>] ? __thread_exit+0x20/0x20 [spl]
[Thu Feb 25 23:49:48 2016]  [<ffffffff82095a59>] kthread+0xc9/0xe0
[Thu Feb 25 23:49:48 2016]  [<ffffffff82095990>] ? kthread_park+0x60/0x60
[Thu Feb 25 23:49:48 2016]  [<ffffffff827a23cf>] ret_from_fork+0x3f/0x70
[Thu Feb 25 23:49:48 2016]  [<ffffffff82095990>] ? kthread_park+0x60/0x60

@behlendorf and @bprotopopov, any thoughts on whether this is related? My patch stack is:

zfs-linux (0.6.5-1~sv20160222) trusty; urgency=medium

  * origin/pr/4326
  ** Add l2arc_max_block_size tunable
  * origin/pr/4334
  ** FreeBSD r256956: Improve ZFS N-way mirror read performance by using load and locality information.
  * origin/pr/2668
  ** Allow for "zfs receive" to skip existing snapshots
  * origin/pr/3169
  ** Add dfree_zfs for changing how Samba reports space
  * origin/pr/4170
  ** Make src const to avoid future mistakes
  * origin/pr/4186
  ** Allow -F to regenerate missing vdev labels
  * origin/pr/4223
  ** Do not lose properties when creating a pool with a temporary name
  * origin/pr/4333
  ** Revert "Wrap smp_processor_id in kpreempt_[dis|en]able" and "Fix CPU_SEQID use in preemptible context"
  * origin/pr/4343
  ** Add -gLp to zpool subcommands for alt vdev names
  * origin/pr/4346
  ** Close the race window in zvol_open() to prevent removal of
  * origin/pr/4351
  ** Require libblkid
  * tuxoko/abd_next
  ** Add dmu_write_abd
  * master @ 4b9ed698b4838a36daa591d08728a692095f8f01
@bprotopopov
Copy link
Contributor

Hi, there,

it is not clear from the stack traces in your email why the txg_sync thread is stalled (and txg_wait_synced() is hung).

Try using 'crash' and see what txg_sync is up to.

The code in 4346 fixes a few bugs, and at first look, I don't think it has introduced this issue.

Assuming that there is some sort of a deadlock that txg_sync is involved in, this could be an example of an issue that is addressed by the #3830

#3830

If it is hard to see what is up, I suggest getting the above pull request and seeing if the problem goes away.

Or maybe backing out 4346 as a troubleshooting step, if that is easier ?

Boris.

[https://avatars3.githubusercontent.com/u/2488505?v=3&s=400]#3830

issues #2217, #3681 - set of commits dealing with zvol ...#3830
github.com
Two issues are being addressed: improved performance of dmu_objset_find() scans called from zvol__minor_() operations, and a deadlock resulting from lock order ...

From: RageLtMan [email protected]
Sent: Friday, February 26, 2016 12:06 AM
To: zfsonlinux/zfs
Cc: Boris Protopopov
Subject: [zfs] ZVOL minors related crash (i think) (#4370)

At first glance, looks like zil replay isnt playing too well with the zvol_create_minor functionality from 4346, which i believe was recently merged into master.
Tried to import a pool after an SCST crash and subsequent reboot, and got this fine mess:

[Thu Feb 25 23:37:48 2016] INFO: task zpool:4651 blocked for more than 180 seconds.
[Thu Feb 25 23:37:48 2016] Tainted: P OE 4.4.2-sv-i7 #sv
[Thu Feb 25 23:37:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:37:48 2016] zpool D ffff880c21003960 0 4651 4037 0x00000002
[Thu Feb 25 23:37:48 2016] ffff880c21003960 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:37:48 2016] ffff880c241d4000 ffff880c2139a000 ffff880c2139a3f8 0000000000000000
[Thu Feb 25 23:37:48 2016] 0000000000000003 ffff880c21003938 ffffffff820a1372 ffff880c21004000
[Thu Feb 25 23:37:48 2016] Call Trace:
[Thu Feb 25 23:37:48 2016] [] ? default_wake_function+0x12/0x20
[Thu Feb 25 23:37:48 2016] [] schedule+0x35/0xa0
[Thu Feb 25 23:37:48 2016] [] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:37:48 2016] [] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:37:48 2016] [] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:37:48 2016] [] txg_wait_synced+0xe9/0x130 [zfs]
[Thu Feb 25 23:37:48 2016] [] zil_replay+0xca/0x100 [zfs]
[Thu Feb 25 23:37:48 2016] [] __zvol_create_minor+0x663/0x670 [zfs]
[Thu Feb 25 23:37:48 2016] [] zvol_create_minor+0x33/0x70 [zfs]
[Thu Feb 25 23:37:48 2016] [] zvol_create_minors_cb+0xe/0x20 [zfs]
[Thu Feb 25 23:37:48 2016] [] dmu_objset_find_impl+0xfa/0x3e0 [zfs]
[Thu Feb 25 23:37:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:37:48 2016] [] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:37:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:37:48 2016] [] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:37:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:37:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:37:48 2016] [] dmu_objset_find+0x49/0x70 [zfs]
[Thu Feb 25 23:37:48 2016] [] zvol_create_minors+0x47/0x60 [zfs]
[Thu Feb 25 23:37:48 2016] [] spa_import+0x464/0x730 [zfs]
[Thu Feb 25 23:37:48 2016] [] zfs_ioc_pool_import+0xdc/0x110 [zfs]
[Thu Feb 25 23:37:48 2016] [] zfsdev_ioctl+0x458/0x490 [zfs]
[Thu Feb 25 23:37:48 2016] [] do_vfs_ioctl+0x2cd/0x4b0
[Thu Feb 25 23:37:48 2016] [] ? putname+0x53/0x60
[Thu Feb 25 23:37:48 2016] [] SyS_ioctl+0x79/0x90
[Thu Feb 25 23:37:48 2016] [] entry_SYSCALL_64_fastpath+0x16/0x75
[Thu Feb 25 23:37:48 2016] INFO: task txg_quiesce:5854 blocked for more than 180 seconds.
[Thu Feb 25 23:37:48 2016] Tainted: P OE 4.4.2-sv-i7 #sv
[Thu Feb 25 23:37:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:37:48 2016] txg_quiesce D ffff88060e26fd48 0 5854 2 0x00000000
[Thu Feb 25 23:37:48 2016] ffff88060e26fd48 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:37:48 2016] ffff880c241e0000 ffff880607446000 ffff8806074463f8 ffff880c2062ab40
[Thu Feb 25 23:37:48 2016] 0000000000000246 0000000000000003 0000000000000001 ffff88060e270000
[Thu Feb 25 23:37:48 2016] Call Trace:
[Thu Feb 25 23:37:48 2016] [] schedule+0x35/0xa0
[Thu Feb 25 23:37:48 2016] [] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:37:48 2016] [] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:37:48 2016] [] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:37:48 2016] [] txg_quiesce_thread+0x2ce/0x3e0 [zfs]
[Thu Feb 25 23:37:48 2016] [] ? txg_delay+0x160/0x160 [zfs]
[Thu Feb 25 23:37:48 2016] [] thread_generic_wrapper+0x71/0x80 [spl]
[Thu Feb 25 23:37:48 2016] [] ? __thread_exit+0x20/0x20 [spl]
[Thu Feb 25 23:37:48 2016] [] kthread+0xc9/0xe0
[Thu Feb 25 23:37:48 2016] [] ? kthread_park+0x60/0x60
[Thu Feb 25 23:37:48 2016] [] ret_from_fork+0x3f/0x70
[Thu Feb 25 23:37:48 2016] [] ? kthread_park+0x60/0x60
[Thu Feb 25 23:40:48 2016] INFO: task zpool:4651 blocked for more than 180 seconds.
[Thu Feb 25 23:40:48 2016] Tainted: P OE 4.4.2-sv-i7 #sv
[Thu Feb 25 23:40:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:40:48 2016] zpool D ffff880c21003960 0 4651 4037 0x00000002
[Thu Feb 25 23:40:48 2016] ffff880c21003960 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:40:48 2016] ffff880c241d4000 ffff880c2139a000 ffff880c2139a3f8 0000000000000000
[Thu Feb 25 23:40:48 2016] 0000000000000003 ffff880c21003938 ffffffff820a1372 ffff880c21004000
[Thu Feb 25 23:40:48 2016] Call Trace:
[Thu Feb 25 23:40:48 2016] [] ? default_wake_function+0x12/0x20
[Thu Feb 25 23:40:48 2016] [] schedule+0x35/0xa0
[Thu Feb 25 23:40:48 2016] [] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:40:48 2016] [] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:40:48 2016] [] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:40:48 2016] [] txg_wait_synced+0xe9/0x130 [zfs]
[Thu Feb 25 23:40:48 2016] [] zil_replay+0xca/0x100 [zfs]
[Thu Feb 25 23:40:48 2016] [] __zvol_create_minor+0x663/0x670 [zfs]
[Thu Feb 25 23:40:48 2016] [] zvol_create_minor+0x33/0x70 [zfs]
[Thu Feb 25 23:40:48 2016] [] zvol_create_minors_cb+0xe/0x20 [zfs]
[Thu Feb 25 23:40:48 2016] [] dmu_objset_find_impl+0xfa/0x3e0 [zfs]
[Thu Feb 25 23:40:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:40:48 2016] [] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:40:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:40:48 2016] [] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:40:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:40:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:40:48 2016] [] dmu_objset_find+0x49/0x70 [zfs]
[Thu Feb 25 23:40:48 2016] [] zvol_create_minors+0x47/0x60 [zfs]
[Thu Feb 25 23:40:48 2016] [] spa_import+0x464/0x730 [zfs]
[Thu Feb 25 23:40:48 2016] [] zfs_ioc_pool_import+0xdc/0x110 [zfs]
[Thu Feb 25 23:40:48 2016] [] zfsdev_ioctl+0x458/0x490 [zfs]
[Thu Feb 25 23:40:48 2016] [] do_vfs_ioctl+0x2cd/0x4b0
[Thu Feb 25 23:40:48 2016] [] ? putname+0x53/0x60
[Thu Feb 25 23:40:48 2016] [] SyS_ioctl+0x79/0x90
[Thu Feb 25 23:40:48 2016] [] entry_SYSCALL_64_fastpath+0x16/0x75
[Thu Feb 25 23:40:48 2016] INFO: task txg_quiesce:5854 blocked for more than 180 seconds.
[Thu Feb 25 23:40:48 2016] Tainted: P OE 4.4.2-sv-i7 #sv
[Thu Feb 25 23:40:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:40:48 2016] txg_quiesce D ffff88060e26fd48 0 5854 2 0x00000000
[Thu Feb 25 23:40:48 2016] ffff88060e26fd48 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:40:48 2016] ffff880c241e0000 ffff880607446000 ffff8806074463f8 ffff880c2062ab40
[Thu Feb 25 23:40:48 2016] 0000000000000246 0000000000000003 0000000000000001 ffff88060e270000
[Thu Feb 25 23:40:48 2016] Call Trace:
[Thu Feb 25 23:40:48 2016] [] schedule+0x35/0xa0
[Thu Feb 25 23:40:48 2016] [] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:40:48 2016] [] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:40:48 2016] [] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:40:48 2016] [] txg_quiesce_thread+0x2ce/0x3e0 [zfs]
[Thu Feb 25 23:40:48 2016] [] ? txg_delay+0x160/0x160 [zfs]
[Thu Feb 25 23:40:48 2016] [] thread_generic_wrapper+0x71/0x80 [spl]
[Thu Feb 25 23:40:48 2016] [] ? __thread_exit+0x20/0x20 [spl]
[Thu Feb 25 23:40:48 2016] [] kthread+0xc9/0xe0
[Thu Feb 25 23:40:48 2016] [] ? kthread_park+0x60/0x60
[Thu Feb 25 23:40:48 2016] [] ret_from_fork+0x3f/0x70
[Thu Feb 25 23:40:48 2016] [] ? kthread_park+0x60/0x60
[Thu Feb 25 23:43:48 2016] INFO: task zpool:4651 blocked for more than 180 seconds.
[Thu Feb 25 23:43:48 2016] Tainted: P OE 4.4.2-sv-i7 #sv
[Thu Feb 25 23:43:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:43:48 2016] zpool D ffff880c21003960 0 4651 4037 0x00000002
[Thu Feb 25 23:43:48 2016] ffff880c21003960 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:43:48 2016] ffff880c241d4000 ffff880c2139a000 ffff880c2139a3f8 0000000000000000
[Thu Feb 25 23:43:48 2016] 0000000000000003 ffff880c21003938 ffffffff820a1372 ffff880c21004000
[Thu Feb 25 23:43:48 2016] Call Trace:
[Thu Feb 25 23:43:48 2016] [] ? default_wake_function+0x12/0x20
[Thu Feb 25 23:43:48 2016] [] schedule+0x35/0xa0
[Thu Feb 25 23:43:48 2016] [] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:43:48 2016] [] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:43:48 2016] [] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:43:48 2016] [] txg_wait_synced+0xe9/0x130 [zfs]
[Thu Feb 25 23:43:48 2016] [] zil_replay+0xca/0x100 [zfs]
[Thu Feb 25 23:43:48 2016] [] __zvol_create_minor+0x663/0x670 [zfs]
[Thu Feb 25 23:43:48 2016] [] zvol_create_minor+0x33/0x70 [zfs]
[Thu Feb 25 23:43:48 2016] [] zvol_create_minors_cb+0xe/0x20 [zfs]
[Thu Feb 25 23:43:48 2016] [] dmu_objset_find_impl+0xfa/0x3e0 [zfs]
[Thu Feb 25 23:43:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:43:48 2016] [] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:43:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:43:48 2016] [] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:43:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:43:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:43:48 2016] [] dmu_objset_find+0x49/0x70 [zfs]
[Thu Feb 25 23:43:48 2016] [] zvol_create_minors+0x47/0x60 [zfs]
[Thu Feb 25 23:43:48 2016] [] spa_import+0x464/0x730 [zfs]
[Thu Feb 25 23:43:48 2016] [] zfs_ioc_pool_import+0xdc/0x110 [zfs]
[Thu Feb 25 23:43:48 2016] [] zfsdev_ioctl+0x458/0x490 [zfs]
[Thu Feb 25 23:43:48 2016] [] do_vfs_ioctl+0x2cd/0x4b0
[Thu Feb 25 23:43:48 2016] [] ? putname+0x53/0x60
[Thu Feb 25 23:43:48 2016] [] SyS_ioctl+0x79/0x90
[Thu Feb 25 23:43:48 2016] [] entry_SYSCALL_64_fastpath+0x16/0x75
[Thu Feb 25 23:43:48 2016] INFO: task txg_quiesce:5854 blocked for more than 180 seconds.
[Thu Feb 25 23:43:48 2016] Tainted: P OE 4.4.2-sv-i7 #sv
[Thu Feb 25 23:43:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:43:48 2016] txg_quiesce D ffff88060e26fd48 0 5854 2 0x00000000
[Thu Feb 25 23:43:48 2016] ffff88060e26fd48 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:43:48 2016] ffff880c241e0000 ffff880607446000 ffff8806074463f8 ffff880c2062ab40
[Thu Feb 25 23:43:48 2016] 0000000000000246 0000000000000003 0000000000000001 ffff88060e270000
[Thu Feb 25 23:43:48 2016] Call Trace:
[Thu Feb 25 23:43:48 2016] [] schedule+0x35/0xa0
[Thu Feb 25 23:43:48 2016] [] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:43:48 2016] [] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:43:48 2016] [] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:43:48 2016] [] txg_quiesce_thread+0x2ce/0x3e0 [zfs]
[Thu Feb 25 23:43:48 2016] [] ? txg_delay+0x160/0x160 [zfs]
[Thu Feb 25 23:43:48 2016] [] thread_generic_wrapper+0x71/0x80 [spl]
[Thu Feb 25 23:43:48 2016] [] ? __thread_exit+0x20/0x20 [spl]
[Thu Feb 25 23:43:48 2016] [] kthread+0xc9/0xe0
[Thu Feb 25 23:43:48 2016] [] ? kthread_park+0x60/0x60
[Thu Feb 25 23:43:48 2016] [] ret_from_fork+0x3f/0x70
[Thu Feb 25 23:43:48 2016] [] ? kthread_park+0x60/0x60
[Thu Feb 25 23:46:48 2016] INFO: task zpool:4651 blocked for more than 180 seconds.
[Thu Feb 25 23:46:48 2016] Tainted: P OE 4.4.2-sv-i7 #sv
[Thu Feb 25 23:46:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:46:48 2016] zpool D ffff880c21003960 0 4651 4037 0x00000002
[Thu Feb 25 23:46:48 2016] ffff880c21003960 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:46:48 2016] ffff880c241d4000 ffff880c2139a000 ffff880c2139a3f8 0000000000000000
[Thu Feb 25 23:46:48 2016] 0000000000000003 ffff880c21003938 ffffffff820a1372 ffff880c21004000
[Thu Feb 25 23:46:48 2016] Call Trace:
[Thu Feb 25 23:46:48 2016] [] ? default_wake_function+0x12/0x20
[Thu Feb 25 23:46:48 2016] [] schedule+0x35/0xa0
[Thu Feb 25 23:46:48 2016] [] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:46:48 2016] [] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:46:48 2016] [] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:46:48 2016] [] txg_wait_synced+0xe9/0x130 [zfs]
[Thu Feb 25 23:46:48 2016] [] zil_replay+0xca/0x100 [zfs]
[Thu Feb 25 23:46:48 2016] [] __zvol_create_minor+0x663/0x670 [zfs]
[Thu Feb 25 23:46:48 2016] [] zvol_create_minor+0x33/0x70 [zfs]
[Thu Feb 25 23:46:48 2016] [] zvol_create_minors_cb+0xe/0x20 [zfs]
[Thu Feb 25 23:46:48 2016] [] dmu_objset_find_impl+0xfa/0x3e0 [zfs]
[Thu Feb 25 23:46:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:46:48 2016] [] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:46:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:46:48 2016] [] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:46:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:46:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:46:48 2016] [] dmu_objset_find+0x49/0x70 [zfs]
[Thu Feb 25 23:46:48 2016] [] zvol_create_minors+0x47/0x60 [zfs]
[Thu Feb 25 23:46:48 2016] [] spa_import+0x464/0x730 [zfs]
[Thu Feb 25 23:46:48 2016] [] zfs_ioc_pool_import+0xdc/0x110 [zfs]
[Thu Feb 25 23:46:48 2016] [] zfsdev_ioctl+0x458/0x490 [zfs]
[Thu Feb 25 23:46:48 2016] [] do_vfs_ioctl+0x2cd/0x4b0
[Thu Feb 25 23:46:48 2016] [] ? putname+0x53/0x60
[Thu Feb 25 23:46:48 2016] [] SyS_ioctl+0x79/0x90
[Thu Feb 25 23:46:48 2016] [] entry_SYSCALL_64_fastpath+0x16/0x75
[Thu Feb 25 23:46:48 2016] INFO: task txg_quiesce:5854 blocked for more than 180 seconds.
[Thu Feb 25 23:46:48 2016] Tainted: P OE 4.4.2-sv-i7 #sv
[Thu Feb 25 23:46:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:46:48 2016] txg_quiesce D ffff88060e26fd48 0 5854 2 0x00000000
[Thu Feb 25 23:46:48 2016] ffff88060e26fd48 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:46:48 2016] ffff880c241e0000 ffff880607446000 ffff8806074463f8 ffff880c2062ab40
[Thu Feb 25 23:46:48 2016] 0000000000000246 0000000000000003 0000000000000001 ffff88060e270000
[Thu Feb 25 23:46:48 2016] Call Trace:
[Thu Feb 25 23:46:48 2016] [] schedule+0x35/0xa0
[Thu Feb 25 23:46:48 2016] [] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:46:48 2016] [] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:46:48 2016] [] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:46:48 2016] [] txg_quiesce_thread+0x2ce/0x3e0 [zfs]
[Thu Feb 25 23:46:48 2016] [] ? txg_delay+0x160/0x160 [zfs]
[Thu Feb 25 23:46:48 2016] [] thread_generic_wrapper+0x71/0x80 [spl]
[Thu Feb 25 23:46:48 2016] [] ? __thread_exit+0x20/0x20 [spl]
[Thu Feb 25 23:46:48 2016] [] kthread+0xc9/0xe0
[Thu Feb 25 23:46:48 2016] [] ? kthread_park+0x60/0x60
[Thu Feb 25 23:46:48 2016] [] ret_from_fork+0x3f/0x70
[Thu Feb 25 23:46:48 2016] [] ? kthread_park+0x60/0x60
[Thu Feb 25 23:49:48 2016] INFO: task zpool:4651 blocked for more than 180 seconds.
[Thu Feb 25 23:49:48 2016] Tainted: P OE 4.4.2-sv-i7 #sv
[Thu Feb 25 23:49:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:49:48 2016] zpool D ffff880c21003960 0 4651 4037 0x00000002
[Thu Feb 25 23:49:48 2016] ffff880c21003960 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:49:48 2016] ffff880c241d4000 ffff880c2139a000 ffff880c2139a3f8 0000000000000000
[Thu Feb 25 23:49:48 2016] 0000000000000003 ffff880c21003938 ffffffff820a1372 ffff880c21004000
[Thu Feb 25 23:49:48 2016] Call Trace:
[Thu Feb 25 23:49:48 2016] [] ? default_wake_function+0x12/0x20
[Thu Feb 25 23:49:48 2016] [] schedule+0x35/0xa0
[Thu Feb 25 23:49:48 2016] [] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:49:48 2016] [] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:49:48 2016] [] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:49:48 2016] [] txg_wait_synced+0xe9/0x130 [zfs]
[Thu Feb 25 23:49:48 2016] [] zil_replay+0xca/0x100 [zfs]
[Thu Feb 25 23:49:48 2016] [] __zvol_create_minor+0x663/0x670 [zfs]
[Thu Feb 25 23:49:48 2016] [] zvol_create_minor+0x33/0x70 [zfs]
[Thu Feb 25 23:49:48 2016] [] zvol_create_minors_cb+0xe/0x20 [zfs]
[Thu Feb 25 23:49:48 2016] [] dmu_objset_find_impl+0xfa/0x3e0 [zfs]
[Thu Feb 25 23:49:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:49:48 2016] [] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:49:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:49:48 2016] [] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]
[Thu Feb 25 23:49:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:49:48 2016] [] ? zvol_create_minor+0x70/0x70 [zfs]
[Thu Feb 25 23:49:48 2016] [] dmu_objset_find+0x49/0x70 [zfs]
[Thu Feb 25 23:49:48 2016] [] zvol_create_minors+0x47/0x60 [zfs]
[Thu Feb 25 23:49:48 2016] [] spa_import+0x464/0x730 [zfs]
[Thu Feb 25 23:49:48 2016] [] zfs_ioc_pool_import+0xdc/0x110 [zfs]
[Thu Feb 25 23:49:48 2016] [] zfsdev_ioctl+0x458/0x490 [zfs]
[Thu Feb 25 23:49:48 2016] [] do_vfs_ioctl+0x2cd/0x4b0
[Thu Feb 25 23:49:48 2016] [] ? putname+0x53/0x60
[Thu Feb 25 23:49:48 2016] [] SyS_ioctl+0x79/0x90
[Thu Feb 25 23:49:48 2016] [] entry_SYSCALL_64_fastpath+0x16/0x75
[Thu Feb 25 23:49:48 2016] INFO: task txg_quiesce:5854 blocked for more than 180 seconds.
[Thu Feb 25 23:49:48 2016] Tainted: P OE 4.4.2-sv-i7 #sv
[Thu Feb 25 23:49:48 2016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Feb 25 23:49:48 2016] txg_quiesce D ffff88060e26fd48 0 5854 2 0x00000000
[Thu Feb 25 23:49:48 2016] ffff88060e26fd48 ffff88060743a554 0000000300000000 ffff88060743a554
[Thu Feb 25 23:49:48 2016] ffff880c241e0000 ffff880607446000 ffff8806074463f8 ffff880c2062ab40
[Thu Feb 25 23:49:48 2016] 0000000000000246 0000000000000003 0000000000000001 ffff88060e270000
[Thu Feb 25 23:49:48 2016] Call Trace:
[Thu Feb 25 23:49:48 2016] [] schedule+0x35/0xa0
[Thu Feb 25 23:49:48 2016] [] cv_wait_common+0xed/0x120 [spl]
[Thu Feb 25 23:49:48 2016] [] ? prepare_to_wait_event+0xf0/0xf0
[Thu Feb 25 23:49:48 2016] [] __cv_wait+0x15/0x20 [spl]
[Thu Feb 25 23:49:48 2016] [] txg_quiesce_thread+0x2ce/0x3e0 [zfs]
[Thu Feb 25 23:49:48 2016] [] ? txg_delay+0x160/0x160 [zfs]
[Thu Feb 25 23:49:48 2016] [] thread_generic_wrapper+0x71/0x80 [spl]
[Thu Feb 25 23:49:48 2016] [] ? __thread_exit+0x20/0x20 [spl]
[Thu Feb 25 23:49:48 2016] [] kthread+0xc9/0xe0
[Thu Feb 25 23:49:48 2016] [] ? kthread_park+0x60/0x60
[Thu Feb 25 23:49:48 2016] [] ret_from_fork+0x3f/0x70
[Thu Feb 25 23:49:48 2016] [] ? kthread_park+0x60/0x60

@behlendorfhttps://github.com/behlendorf and @bprotopopovhttps://github.com/bprotopopov, any thoughts on whether this is related? My patch stack is:

zfs-linux (0.6.5-1~sv20160222) trusty; urgency=medium

  • origin/pr/4326
    ** Add l2arc_max_block_size tunable
  • origin/pr/4334
    ** FreeBSD r256956: Improve ZFS N-way mirror read performance by using load and locality information.
  • origin/pr/2668
    ** Allow for "zfs receive" to skip existing snapshots
  • origin/pr/3169
    ** Add dfree_zfs for changing how Samba reports space
  • origin/pr/4170
    ** Make src const to avoid future mistakes
  • origin/pr/4186
    ** Allow -F to regenerate missing vdev labels
  • origin/pr/4223
    ** Do not lose properties when creating a pool with a temporary name
  • origin/pr/4333
    ** Revert "Wrap smp_processor_id in kpreempt_[dis|en]able" and "Fix CPU_SEQID use in preemptible context"
  • origin/pr/4343
    ** Add -gLp to zpool subcommands for alt vdev names
  • origin/pr/4346
    ** Close the race window in zvol_open() to prevent removal of
  • origin/pr/4351
    ** Require libblkid
  • tuxoko/abd_next
    ** Add dmu_write_abd
  • master @ 4b9ed69

Reply to this email directly or view it on GitHubhttps://github.com//issues/4370.

@behlendorf
Copy link
Contributor

Closing. The finalized version of the async minor patches which were merged addressed this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@behlendorf @sempervictus @bprotopopov