nfsd bash z_unmount blocked for more than 120 seconds

[wonder]

Hello,

zfs-dkms 0.6.5.2-1~trusty

I'm trying to access a snapshot over nfs and sometimes it hangs, now i didn't even had any write operations via nfs.

After nfsd hanged, I wanted to copy some files from the snapshot directory directly from this server and entering in the diretory hanged as well.

Right now this box is very unstable with 0.6.5.2-1.

6480.144035] INFO: task nfsd:1794 blocked for more than 120 seconds.
[ 6480.144162] Tainted: P OE 3.16.0-51-generic #6914.04.1-Ubuntu
[ 6480.144266] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6480.144374] nfsd D ffff88045fc930c0 0 1794 2 0x00000000
[ 6480.144380] ffff88043e86ba20 0000000000000046 ffff88043dcef010 ffff88043e86bfd8
[ 6480.144383] 00000000000130c0 00000000000130c0 ffff880449659e90 ffffffffc0624560
[ 6480.144387] ffffffffc0624564 ffff88043dcef010 00000000ffffffff ffffffffc0624568
[ 6480.144390] Call Trace:
[ 6480.144412] [] schedule_preempt_disabled+0x29/0x70
[ 6480.144415] [] __mutex_lock_slowpath+0xd5/0x1d0
[ 6480.144419] [] mutex_lock+0x1f/0x2f
[ 6480.144487] [] zfsctl_snapshot_unmount_delay+0x26/0xa0 [zfs]
[ 6480.144528] [] zfsctl_lookup_objset+0x246/0x3a0 [zfs]
[ 6480.144532] [] ? mutex_lock+0x12/0x2f
[ 6480.144569] [] ? rrw_exit+0x68/0x170 [zfs]
[ 6480.144612] [] zfs_vget+0x176/0x3e0 [zfs]
[ 6480.144655] [] zpl_fh_to_dentry+0x72/0xb0 [zfs]
[ 6480.144668] [] ? nfsd_proc_getattr+0xa0/0xa0 [nfsd]
[ 6480.144673] [] exportfs_decode_fh+0x72/0x2e0
[ 6480.144697] [] ? cache_check+0x106/0x3d0 [sunrpc]
[ 6480.144705] [] ? exp_find+0xfc/0x1b0 [nfsd]
[ 6480.144710] [] ? kmem_cache_alloc_trace+0x1c6/0x1f0
[ 6480.144714] [] ? aa_alloc_task_context+0x27/0x40
[ 6480.144723] [] fh_verify+0x2f6/0x5e0 [nfsd]
[ 6480.144727] [] ? kfree+0xab/0x140
[ 6480.144736] [] nfsd4_putfh+0x49/0x50 [nfsd]
[ 6480.144746] [] nfsd4_proc_compound+0x4ff/0x820 [nfsd]
[ 6480.144754] [] nfsd_dispatch+0xbb/0x200 [nfsd]
[ 6480.144768] [] svc_process_common+0x440/0x6d0 [sunrpc]
[ 6480.144777] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6480.144790] [] svc_process+0x107/0x170 [sunrpc]
[ 6480.144798] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6480.144805] [] nfsd+0xc7/0x130 [nfsd]
[ 6480.144811] [] kthread+0xd2/0xf0
[ 6480.144814] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6480.144818] [] ret_from_fork+0x58/0x90
[ 6480.144821] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6480.144824] INFO: task nfsd:1797 blocked for more than 120 seconds.
[ 6480.144923] Tainted: P OE 3.16.0-51-generic #6914.04.1-Ubuntu
[ 6480.145026] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6480.145133] nfsd D ffff88045fc130c0 0 1797 2 0x00000000
[ 6480.145137] ffff88043e687a50 0000000000000046 ffff880440118a30 ffff88043e687fd8
[ 6480.145141] 00000000000130c0 00000000000130c0 ffff8804495c4750 ffffffffc0624560
[ 6480.145144] ffffffffc0624564 ffff880440118a30 00000000ffffffff ffffffffc0624568
[ 6480.145148] Call Trace:
[ 6480.145152] [] schedule_preempt_disabled+0x29/0x70
[ 6480.145155] [] __mutex_lock_slowpath+0xd5/0x1d0
[ 6480.145159] [] mutex_lock+0x1f/0x2f
[ 6480.145199] [] zfsctl_lookup_objset+0x34/0x3a0 [zfs]
[ 6480.145202] [] ? mutex_lock+0x12/0x2f
[ 6480.145239] [] ? rrw_exit+0x68/0x170 [zfs]
[ 6480.145281] [] zfs_vget+0x176/0x3e0 [zfs]
[ 6480.145285] [] ? check_preempt_curr+0x85/0xa0
[ 6480.145288] [] ? ttwu_do_wakeup+0x19/0xc0
[ 6480.145329] [] zpl_fh_to_dentry+0x72/0xb0 [zfs]
[ 6480.145337] [] ? nfsd_proc_getattr+0xa0/0xa0 [nfsd]
[ 6480.145340] [] exportfs_decode_fh+0x72/0x2e0
[ 6480.145344] [] ? default_wake_function+0x12/0x20
[ 6480.145349] [] ? autoremove_wake_function+0x12/0x40
[ 6480.145352] [] ? __wake_up_common+0x58/0x90
[ 6480.145355] [] ? __wake_up+0x44/0x50
[ 6480.145359] [] ? __call_rcu_nocb_enqueue+0xc8/0xd0
[ 6480.145367] [] fh_verify+0x2f6/0x5e0 [nfsd]
[ 6480.145370] [] ? kfree+0xab/0x140
[ 6480.145379] [] nfsd4_putfh+0x49/0x50 [nfsd]
[ 6480.145388] [] nfsd4_proc_compound+0x4ff/0x820 [nfsd]
[ 6480.145396] [] nfsd_dispatch+0xbb/0x200 [nfsd]
[ 6480.145410] [] svc_process_common+0x440/0x6d0 [sunrpc]
[ 6480.145418] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6480.145432] [] svc_process+0x107/0x170 [sunrpc]
[ 6480.145439] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6480.145447] [] nfsd+0xc7/0x130 [nfsd]
[ 6480.145450] [] kthread+0xd2/0xf0
[ 6480.145454] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6480.145457] [] ret_from_fork+0x58/0x90
[ 6480.145460] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6600.144028] INFO: task z_unmount:315 blocked for more than 120 seconds.
[ 6600.144133] Tainted: P OE 3.16.0-51-generic #6914.04.1-Ubuntu
[ 6600.144236] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6600.144344] z_unmount D ffff88045fcd30c0 0 315 2 0x00000000
[ 6600.144350] ffff8804464d7d40 0000000000000046 ffff88009e8fa8c0 ffff8804464d7fd8
[ 6600.144354] 00000000000130c0 00000000000130c0 ffff88044965a8c0 ffffffffc0624560
[ 6600.144357] ffffffffc0624564 ffff88009e8fa8c0 00000000ffffffff ffffffffc0624568
[ 6600.144361] Call Trace:
[ 6600.144382] [] schedule_preempt_disabled+0x29/0x70
[ 6600.144385] [] __mutex_lock_slowpath+0xd5/0x1d0
[ 6600.144389] [] mutex_lock+0x1f/0x2f
[ 6600.144457] [] zfsctl_snapshot_unmount+0x4a/0x170 [zfs]
[ 6600.144499] [] snapentry_expire+0x28/0xb0 [zfs]
[ 6600.144511] [] taskq_thread+0x20d/0x410 [spl]
[ 6600.144517] [] ? wake_up_state+0x20/0x20
[ 6600.144525] [] ? taskq_cancel_id+0x120/0x120 [spl]
[ 6600.144530] [] kthread+0xd2/0xf0
[ 6600.144533] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6600.144536] [] ret_from_fork+0x58/0x90
[ 6600.144539] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6600.144559] INFO: task nfsd:1794 blocked for more than 120 seconds.
[ 6600.144659] Tainted: P OE 3.16.0-51-generic #6914.04.1-Ubuntu
[ 6600.144764] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6600.144871] nfsd D ffff88045fc930c0 0 1794 2 0x00000004
[ 6600.144875] ffff88043e86ba20 0000000000000046 ffff88043dcef010 ffff88043e86bfd8
[ 6600.144878] 00000000000130c0 00000000000130c0 ffff880449659e90 ffffffffc0624560
[ 6600.144881] ffffffffc0624564 ffff88043dcef010 00000000ffffffff ffffffffc0624568
[ 6600.144885] Call Trace:
[ 6600.144890] [] schedule_preempt_disabled+0x29/0x70
[ 6600.144893] [] __mutex_lock_slowpath+0xd5/0x1d0
[ 6600.144897] [] mutex_lock+0x1f/0x2f
[ 6600.144937] [] zfsctl_snapshot_unmount_delay+0x26/0xa0 [zfs]
[ 6600.144978] [] zfsctl_lookup_objset+0x246/0x3a0 [zfs]
[ 6600.144982] [] ? mutex_lock+0x12/0x2f
[ 6600.145019] [] ? rrw_exit+0x68/0x170 [zfs]
[ 6600.145062] [] zfs_vget+0x176/0x3e0 [zfs]
[ 6600.145105] [] zpl_fh_to_dentry+0x72/0xb0 [zfs]
[ 6600.145118] [] ? nfsd_proc_getattr+0xa0/0xa0 [nfsd]
[ 6600.145123] [] exportfs_decode_fh+0x72/0x2e0
[ 6600.145147] [] ? cache_check+0x106/0x3d0 [sunrpc]
[ 6600.145156] [] ? exp_find+0xfc/0x1b0 [nfsd]
[ 6600.145160] [] ? kmem_cache_alloc_trace+0x1c6/0x1f0
[ 6600.145164] [] ? aa_alloc_task_context+0x27/0x40
[ 6600.145173] [] fh_verify+0x2f6/0x5e0 [nfsd]
[ 6600.145177] [] ? kfree+0xab/0x140
[ 6600.145187] [] nfsd4_putfh+0x49/0x50 [nfsd]
[ 6600.145196] [] nfsd4_proc_compound+0x4ff/0x820 [nfsd]
[ 6600.145204] [] nfsd_dispatch+0xbb/0x200 [nfsd]
[ 6600.145219] [] svc_process_common+0x440/0x6d0 [sunrpc]
[ 6600.145227] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6600.145240] [] svc_process+0x107/0x170 [sunrpc]
[ 6600.145248] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6600.145255] [] nfsd+0xc7/0x130 [nfsd]
[ 6600.145259] [] kthread+0xd2/0xf0
[ 6600.145262] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6600.145265] [] ret_from_fork+0x58/0x90
[ 6600.145269] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6600.145271] INFO: task nfsd:1797 blocked for more than 120 seconds.
[ 6600.145370] Tainted: P OE 3.16.0-51-generic #6914.04.1-Ubuntu
[ 6600.145474] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6600.145581] nfsd D ffff88045fc130c0 0 1797 2 0x00000004
[ 6600.145585] ffff88043e687a50 0000000000000046 ffff880440118a30 ffff88043e687fd8
[ 6600.145588] 00000000000130c0 00000000000130c0 ffff8804495c4750 ffffffffc0624560
[ 6600.145592] ffffffffc0624564 ffff880440118a30 00000000ffffffff ffffffffc0624568
[ 6600.145595] Call Trace:
[ 6600.145600] [] schedule_preempt_disabled+0x29/0x70
[ 6600.145603] [] __mutex_lock_slowpath+0xd5/0x1d0
[ 6600.145606] [] mutex_lock+0x1f/0x2f
[ 6600.145647] [] zfsctl_lookup_objset+0x34/0x3a0 [zfs]
[ 6600.145650] [] ? mutex_lock+0x12/0x2f
[ 6600.145686] [] ? rrw_exit+0x68/0x170 [zfs]
[ 6600.145728] [] zfs_vget+0x176/0x3e0 [zfs]
[ 6600.145731] [] ? check_preempt_curr+0x85/0xa0
[ 6600.145734] [] ? ttwu_do_wakeup+0x19/0xc0
[ 6600.145776] [] zpl_fh_to_dentry+0x72/0xb0 [zfs]
[ 6600.145784] [] ? nfsd_proc_getattr+0xa0/0xa0 [nfsd]
[ 6600.145787] [] exportfs_decode_fh+0x72/0x2e0
[ 6600.145790] [] ? default_wake_function+0x12/0x20
[ 6600.145795] [] ? autoremove_wake_function+0x12/0x40
[ 6600.145798] [] ? __wake_up_common+0x58/0x90
[ 6600.145801] [] ? __wake_up+0x44/0x50
[ 6600.145804] [] ? __call_rcu_nocb_enqueue+0xc8/0xd0
[ 6600.145813] [] fh_verify+0x2f6/0x5e0 [nfsd]
[ 6600.145816] [] ? kfree+0xab/0x140
[ 6600.145825] [] nfsd4_putfh+0x49/0x50 [nfsd]
[ 6600.145834] [] nfsd4_proc_compound+0x4ff/0x820 [nfsd]
[ 6600.145842] [] nfsd_dispatch+0xbb/0x200 [nfsd]
[ 6600.145856] [] svc_process_common+0x440/0x6d0 [sunrpc]
[ 6600.145864] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6600.145878] [] svc_process+0x107/0x170 [sunrpc]
[ 6600.145885] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6600.145893] [] nfsd+0xc7/0x130 [nfsd]
[ 6600.145896] [] kthread+0xd2/0xf0
[ 6600.145899] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6600.145903] [] ret_from_fork+0x58/0x90
[ 6600.145906] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6720.144029] INFO: task z_unmount:315 blocked for more than 120 seconds.
[ 6720.144135] Tainted: P OE 3.16.0-51-generic #6914.04.1-Ubuntu
[ 6720.144239] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6720.144347] z_unmount D ffff88045fcd30c0 0 315 2 0x00000000
[ 6720.144353] ffff8804464d7d40 0000000000000046 ffff88009e8fa8c0 ffff8804464d7fd8
[ 6720.144357] 00000000000130c0 00000000000130c0 ffff88044965a8c0 ffffffffc0624560
[ 6720.144360] ffffffffc0624564 ffff88009e8fa8c0 00000000ffffffff ffffffffc0624568
[ 6720.144364] Call Trace:
[ 6720.144386] [] schedule_preempt_disabled+0x29/0x70
[ 6720.144389] [] __mutex_lock_slowpath+0xd5/0x1d0
[ 6720.144393] [] mutex_lock+0x1f/0x2f
[ 6720.144461] [] zfsctl_snapshot_unmount+0x4a/0x170 [zfs]
[ 6720.144503] [] snapentry_expire+0x28/0xb0 [zfs]
[ 6720.144516] [] taskq_thread+0x20d/0x410 [spl]
[ 6720.144521] [] ? wake_up_state+0x20/0x20
[ 6720.144529] [] ? taskq_cancel_id+0x120/0x120 [spl]
[ 6720.144534] [] kthread+0xd2/0xf0
[ 6720.144538] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6720.144541] [] ret_from_fork+0x58/0x90
[ 6720.144544] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6720.144566] INFO: task nfsd:1794 blocked for more than 120 seconds.
[ 6720.144666] Tainted: P OE 3.16.0-51-generic #6914.04.1-Ubuntu
[ 6720.144769] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6720.144876] nfsd D ffff88045fc930c0 0 1794 2 0x00000004
[ 6720.144880] ffff88043e86ba20 0000000000000046 ffff88043dcef010 ffff88043e86bfd8
[ 6720.144883] 00000000000130c0 00000000000130c0 ffff880449659e90 ffffffffc0624560
[ 6720.144887] ffffffffc0624564 ffff88043dcef010 00000000ffffffff ffffffffc0624568
[ 6720.144890] Call Trace:
[ 6720.144895] [] schedule_preempt_disabled+0x29/0x70
[ 6720.144898] [] __mutex_lock_slowpath+0xd5/0x1d0
[ 6720.144902] [] mutex_lock+0x1f/0x2f
[ 6720.144943] [] zfsctl_snapshot_unmount_delay+0x26/0xa0 [zfs]
[ 6720.144984] [] zfsctl_lookup_objset+0x246/0x3a0 [zfs]
[ 6720.144987] [] ? mutex_lock+0x12/0x2f
[ 6720.145024] [] ? rrw_exit+0x68/0x170 [zfs]
[ 6720.145067] [] zfs_vget+0x176/0x3e0 [zfs]
[ 6720.145111] [] zpl_fh_to_dentry+0x72/0xb0 [zfs]
[ 6720.145123] [] ? nfsd_proc_getattr+0xa0/0xa0 [nfsd]
[ 6720.145129] [] exportfs_decode_fh+0x72/0x2e0
[ 6720.145152] [] ? cache_check+0x106/0x3d0 [sunrpc]
[ 6720.145161] [] ? exp_find+0xfc/0x1b0 [nfsd]
[ 6720.145165] [] ? kmem_cache_alloc_trace+0x1c6/0x1f0
[ 6720.145169] [] ? aa_alloc_task_context+0x27/0x40
[ 6720.145178] [] fh_verify+0x2f6/0x5e0 [nfsd]
[ 6720.145183] [] ? kfree+0xab/0x140
[ 6720.145192] [] nfsd4_putfh+0x49/0x50 [nfsd]
[ 6720.145201] [] nfsd4_proc_compound+0x4ff/0x820 [nfsd]
[ 6720.145210] [] nfsd_dispatch+0xbb/0x200 [nfsd]
[ 6720.145224] [] svc_process_common+0x440/0x6d0 [sunrpc]
[ 6720.145232] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6720.145246] [] svc_process+0x107/0x170 [sunrpc]
[ 6720.145253] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6720.145261] [] nfsd+0xc7/0x130 [nfsd]
[ 6720.145264] [] kthread+0xd2/0xf0
[ 6720.145267] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6720.145271] [] ret_from_fork+0x58/0x90
[ 6720.145274] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6720.145276] INFO: task nfsd:1797 blocked for more than 120 seconds.
[ 6720.145376] Tainted: P OE 3.16.0-51-generic #6914.04.1-Ubuntu
[ 6720.145478] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6720.145585] nfsd D ffff88045fc130c0 0 1797 2 0x00000004
[ 6720.145589] ffff88043e687a50 0000000000000046 ffff880440118a30 ffff88043e687fd8
[ 6720.145592] 00000000000130c0 00000000000130c0 ffff8804495c4750 ffffffffc0624560
[ 6720.145596] ffffffffc0624564 ffff880440118a30 00000000ffffffff ffffffffc0624568
[ 6720.145599] Call Trace:
[ 6720.145604] [] schedule_preempt_disabled+0x29/0x70
[ 6720.145607] [] __mutex_lock_slowpath+0xd5/0x1d0
[ 6720.145610] [] mutex_lock+0x1f/0x2f
[ 6720.145651] [] zfsctl_lookup_objset+0x34/0x3a0 [zfs]
[ 6720.145654] [] ? mutex_lock+0x12/0x2f
[ 6720.145690] [] ? rrw_exit+0x68/0x170 [zfs]
[ 6720.145732] [] zfs_vget+0x176/0x3e0 [zfs]
[ 6720.145735] [] ? check_preempt_curr+0x85/0xa0
[ 6720.145738] [] ? ttwu_do_wakeup+0x19/0xc0
[ 6720.145780] [] zpl_fh_to_dentry+0x72/0xb0 [zfs]
[ 6720.145788] [] ? nfsd_proc_getattr+0xa0/0xa0 [nfsd]
[ 6720.145791] [] exportfs_decode_fh+0x72/0x2e0
[ 6720.145794] [] ? default_wake_function+0x12/0x20
[ 6720.145799] [] ? autoremove_wake_function+0x12/0x40
[ 6720.145802] [] ? __wake_up_common+0x58/0x90
[ 6720.145805] [] ? __wake_up+0x44/0x50
[ 6720.145808] [] ? __call_rcu_nocb_enqueue+0xc8/0xd0
[ 6720.145817] [] fh_verify+0x2f6/0x5e0 [nfsd]
[ 6720.145820] [] ? kfree+0xab/0x140
[ 6720.145829] [] nfsd4_putfh+0x49/0x50 [nfsd]
[ 6720.145838] [] nfsd4_proc_compound+0x4ff/0x820 [nfsd]
[ 6720.145846] [] nfsd_dispatch+0xbb/0x200 [nfsd]
[ 6720.145860] [] svc_process_common+0x440/0x6d0 [sunrpc]
[ 6720.145868] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6720.145882] [] svc_process+0x107/0x170 [sunrpc]
[ 6720.145889] [] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 6720.145897] [] nfsd+0xc7/0x130 [nfsd]
[ 6720.145900] [] kthread+0xd2/0xf0
[ 6720.145904] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6720.145907] [] ret_from_fork+0x58/0x90
[ 6720.145910] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6720.145924] INFO: task bash:4802 blocked for more than 120 seconds.
[ 6720.146023] Tainted: P OE 3.16.0-51-generic #6914.04.1-Ubuntu
[ 6720.146126] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6720.146233] bash D ffff88045fcd30c0 0 4802 3466 0x00000000
[ 6720.146237] ffff880062b67b40 0000000000000086 ffff88043b478000 ffff880062b67fd8
[ 6720.146240] 00000000000130c0 00000000000130c0 ffff88044965a8c0 ffffffffc0624560
[ 6720.146243] ffffffffc0624564 ffff88043b478000 00000000ffffffff ffffffffc0624568
[ 6720.146247] Call Trace:
[ 6720.146251] [] schedule_preempt_disabled+0x29/0x70
[ 6720.146255] [] __mutex_lock_slowpath+0xd5/0x1d0
[ 6720.146258] [] mutex_lock+0x1f/0x2f
[ 6720.146299] [] zfsctl_snapshot_mount+0x1e1/0x440 [zfs]
[ 6720.146341] [] zpl_snapdir_automount+0x10/0x30 [zfs]
[ 6720.146346] [] follow_managed+0x13a/0x300
[ 6720.146350] [] ? __d_lookup+0x120/0x160
[ 6720.146353] [] lookup_fast+0x18b/0x2c0
[ 6720.146357] [] ? security_inode_permission+0x1c/0x30
[ 6720.146361] [] ? __inode_permission+0x52/0xc0
[ 6720.146364] [] do_last+0xa7/0x1210
[ 6720.146368] [] ? apparmor_file_alloc_security+0x5b/0x180
[ 6720.146371] [] ? security_file_alloc+0x16/0x20
[ 6720.146374] [] path_openat+0xbb/0x640
[ 6720.146377] [] ? putname+0x29/0x40
[ 6720.146380] [] ? user_path_at_empty+0x5f/0x90
[ 6720.146382] [] do_filp_open+0x3a/0x90
[ 6720.146386] [] ? __alloc_fd+0xa7/0x130
[ 6720.146391] [] do_sys_open+0x129/0x280
[ 6720.146394] [] SyS_openat+0x14/0x20
[ 6720.146397] [] system_call_fastpath+0x1a/0x1f
[ 6840.144027] INFO: task z_unmount:315 blocked for more than 120 seconds.
[ 6840.144132] Tainted: P OE 3.16.0-51-generic #6914.04.1-Ubuntu
[ 6840.144237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6840.144345] z_unmount D ffff88045fcd30c0 0 315 2 0x00000000
[ 6840.144350] ffff8804464d7d40 0000000000000046 ffff88009e8fa8c0 ffff8804464d7fd8
[ 6840.144354] 00000000000130c0 00000000000130c0 ffff88044965a8c0 ffffffffc0624560
[ 6840.144358] ffffffffc0624564 ffff88009e8fa8c0 00000000ffffffff ffffffffc0624568
[ 6840.144361] Call Trace:
[ 6840.144381] [] schedule_preempt_disabled+0x29/0x70
[ 6840.144384] [] __mutex_lock_slowpath+0xd5/0x1d0
[ 6840.144388] [] mutex_lock+0x1f/0x2f
[ 6840.144453] [] zfsctl_snapshot_unmount+0x4a/0x170 [zfs]
[ 6840.144496] [] snapentry_expire+0x28/0xb0 [zfs]
[ 6840.144507] [] taskq_thread+0x20d/0x410 [spl]
[ 6840.144513] [] ? wake_up_state+0x20/0x20
[ 6840.144521] [] ? taskq_cancel_id+0x120/0x120 [spl]
[ 6840.144526] [] kthread+0xd2/0xf0
[ 6840.144529] [] ? kthread_create_on_node+0x1c0/0x1c0
[ 6840.144533] [] ret_from_fork+0x58/0x90
[ 6840.144536] [] ? kthread_create_on_node+0x1c0/0x1c0

[behlendorf]

This is fixed in master by commit f22f900. It can be safely cherry-picked back to your version if needed to resolve the issue. Sorry about that.