zfs panics on, multiple lxc container migrations #5323
Labels
Comments
|
@upuvvad1 I'm not seeing a panic, however, the send process is clearly blocked. This is exactly what happens if the receive process isn't consuming any data. Are there other systems (on the network) involved with this replication? Why are there multiple receive processes to the same dataset? |
|
@dweeezil , And also the process state of ZFS is being set to D (uninterruptible sleep) which in the normal case should be in S(waiting for the data to receive on the network). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
I am using LXC containers, with ZFS file system as a backend.
At heavy computational load, zfs hangs up with kernel panic, and leaves processes without cleaning.
You can see the complete conversation with lxc team at https://github.com/lxc/lxd/issues/2543
ps aux | grep zfs
root 31730 0.0 0.0 34628 1788 ? S 14:24 0:00 zfs receive -F -u lxd/containers/cnt52
root 31775 0.0 0.0 34628 1764 ? S 14:24 0:00 zfs receive -F -u lxd/containers/cnt54
root 31787 0.0 0.0 34628 1804 ? S 14:26 0:00 zfs receive -F -u lxd/containers/cnt53
root 33030 0.0 0.0 34628 1696 ? S 14:36 0:00 zfs receive -F -u lxd/containers/cnt53
root 33308 0.0 0.0 34628 1800 ? S 14:47 0:00 zfs receive -F -u lxd/containers/cnt53
root 37162 0.0 0.0 34628 3552 ? D 17:50 0:01 zfs send lxd/containers/cnt53 migration-send-645df198-1868-43da-9f82-53755937b22e
Oct 21 17:58:46 r520-53 kernel: [25680.612721] INFO: task zfs:37162 blocked for more than 120 seconds.
Oct 21 17:58:46 r520-53 kernel: [25680.619937] Tainted: P O 4.4.0-43-generic #63-Ubuntu
Oct 21 17:58:46 r520-53 kernel: [25680.627537] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 21 17:58:46 r520-53 kernel: [25680.636530] zfs D ffff8806501e73a8 0 37162 22640 0x00000002
Oct 21 17:58:46 r520-53 kernel: [25680.636538] ffff8806501e73a8 ffffffff810c3712 ffff88080be42940 ffff880e3d3a3700
Oct 21 17:58:46 r520-53 kernel: [25680.636543] ffff8806501e8000 000000000006632f ffff88100b542bc8 0000000000000000
Oct 21 17:58:46 r520-53 kernel: [25680.636547] ffff88100afd1d00 ffff8806501e73c0 ffffffff8182d7c5 ffff88100b542b00
Oct 21 17:58:46 r520-53 kernel: [25680.636551] Call Trace:
Oct 21 17:58:46 r520-53 kernel: [25680.636579] [] ? __wake_up_common+0x52/0x90
Oct 21 17:58:46 r520-53 kernel: [25680.636597] [] schedule+0x35/0x80
Oct 21 17:58:46 r520-53 kernel: [25680.636621] [] taskq_wait_id+0x60/0xb0 [spl]
Oct 21 17:58:46 r520-53 kernel: [25680.636626] [] ? wake_atomic_t_function+0x60/0x60
Oct 21 17:58:46 r520-53 kernel: [25680.636692] [] ? dump_write+0x230/0x230 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.636752] [] spa_taskq_dispatch_sync+0x92/0xd0 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.636794] [] dump_bytes+0x51/0x70 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.636835] [] dump_write+0x11e/0x230 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.636875] [] backup_cb+0x633/0x850 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.636915] [] traverse_visitbp+0x47a/0x960 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.636923] [] ? spl_kmem_alloc+0xaf/0x190 [spl]
Oct 21 17:58:46 r520-53 kernel: [25680.636963] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637003] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637043] [] traverse_visitbp+0x6cc/0x960 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637082] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637121] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637161] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637200] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637239] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637278] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637318] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637356] [] traverse_visitbp+0x865/0x960 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637396] [] traverse_impl+0x1ae/0x410 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637435] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637475] [] traverse_dataset+0x52/0x60 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637514] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637552] [] dmu_send_impl+0x409/0x560 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637592] [] dmu_send_obj+0x172/0x1e0 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637651] [] zfs_ioc_send+0xe9/0x2c0 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637659] [] ? strdup+0x3b/0x60 [spl]
Oct 21 17:58:46 r520-53 kernel: [25680.637719] [] zfsdev_ioctl+0x44b/0x4e0 [zfs]
Oct 21 17:58:46 r520-53 kernel: [25680.637725] [] do_vfs_ioctl+0x29f/0x490
Oct 21 17:58:46 r520-53 kernel: [25680.637730] [] ? __do_page_fault+0x1b4/0x400
Oct 21 17:58:46 r520-53 kernel: [25680.637733] [] SyS_ioctl+0x79/0x90
Oct 21 17:58:46 r520-53 kernel: [25680.637738] [] entry_SYSCALL_64_fastpath+0x16/0x71
Oct 21 18:00:46 r520-53 kernel: [25800.633371] INFO: task zfs:37162 blocked for more than 120 seconds.
Oct 21 18:00:46 r520-53 kernel: [25800.640575] Tainted: P O 4.4.0-43-generic #63-Ubuntu
Oct 21 18:00:46 r520-53 kernel: [25800.648171] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 21 18:00:46 r520-53 kernel: [25800.657158] zfs D ffff8806501e73a8 0 37162 22640 0x00000002
Oct 21 18:00:46 r520-53 kernel: [25800.657165] ffff8806501e73a8 ffffffff810c3712 ffff88080be42940 ffff880e3d3a3700
Oct 21 18:00:46 r520-53 kernel: [25800.657169] ffff8806501e8000 000000000006632f ffff88100b542bc8 0000000000000000
Oct 21 18:00:46 r520-53 kernel: [25800.657173] ffff88100afd1d00 ffff8806501e73c0 ffffffff8182d7c5 ffff88100b542b00
Oct 21 18:00:46 r520-53 kernel: [25800.657176] Call Trace:
Oct 21 18:00:46 r520-53 kernel: [25800.657205] [] ? __wake_up_common+0x52/0x90
Oct 21 18:00:46 r520-53 kernel: [25800.657222] [] schedule+0x35/0x80
Oct 21 18:00:46 r520-53 kernel: [25800.657243] [] taskq_wait_id+0x60/0xb0 [spl]
Oct 21 18:00:46 r520-53 kernel: [25800.657247] [] ? wake_atomic_t_function+0x60/0x60
Oct 21 18:00:46 r520-53 kernel: [25800.657306] [] ? dump_write+0x230/0x230 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657360] [] spa_taskq_dispatch_sync+0x92/0xd0 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657398] [] dump_bytes+0x51/0x70 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657435] [] dump_write+0x11e/0x230 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657471] [] backup_cb+0x633/0x850 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657507] [] traverse_visitbp+0x47a/0x960 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657514] [] ? spl_kmem_alloc+0xaf/0x190 [spl]
Oct 21 18:00:46 r520-53 kernel: [25800.657550] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657586] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657621] [] traverse_visitbp+0x6cc/0x960 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657656] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657692] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657727] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657762] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657797] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657832] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657868] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657903] [] traverse_visitbp+0x865/0x960 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657938] [] traverse_impl+0x1ae/0x410 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.657974] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.658009] [] traverse_dataset+0x52/0x60 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.658044] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.658080] [] dmu_send_impl+0x409/0x560 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.658114] [] dmu_send_obj+0x172/0x1e0 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.658167] [] zfs_ioc_send+0xe9/0x2c0 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.658175] [] ? strdup+0x3b/0x60 [spl]
Oct 21 18:00:46 r520-53 kernel: [25800.658227] [] zfsdev_ioctl+0x44b/0x4e0 [zfs]
Oct 21 18:00:46 r520-53 kernel: [25800.658234] [] do_vfs_ioctl+0x29f/0x490
Oct 21 18:00:46 r520-53 kernel: [25800.658239] [] ? __do_page_fault+0x1b4/0x400
Oct 21 18:00:46 r520-53 kernel: [25800.658242] [] SyS_ioctl+0x79/0x90
Oct 21 18:00:46 r520-53 kernel: [25800.658247] [] entry_SYSCALL_64_fastpath+0x16/0x71
Oct 21 18:02:46 r520-53 kernel: [25920.654007] INFO: task zfs:37162 blocked for more than 120 seconds.
Oct 21 18:02:46 r520-53 kernel: [25920.661213] Tainted: P O 4.4.0-43-generic #63-Ubuntu
Oct 21 18:02:46 r520-53 kernel: [25920.668817] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 21 18:02:46 r520-53 kernel: [25920.677806] zfs D ffff8806501e73a8 0 37162 22640 0x00000002
Oct 21 18:02:46 r520-53 kernel: [25920.677815] ffff8806501e73a8 ffffffff810c3712 ffff88080be42940 ffff880e3d3a3700
Oct 21 18:02:46 r520-53 kernel: [25920.677819] ffff8806501e8000 000000000006632f ffff88100b542bc8 0000000000000000
Oct 21 18:02:46 r520-53 kernel: [25920.677823] ffff88100afd1d00 ffff8806501e73c0 ffffffff8182d7c5 ffff88100b542b00
Oct 21 18:02:46 r520-53 kernel: [25920.677840] Call Trace:
Oct 21 18:02:46 r520-53 kernel: [25920.677854] [] ? __wake_up_common+0x52/0x90
Oct 21 18:02:46 r520-53 kernel: [25920.677863] [] schedule+0x35/0x80
Oct 21 18:02:46 r520-53 kernel: [25920.677880] [] taskq_wait_id+0x60/0xb0 [spl]
Oct 21 18:02:46 r520-53 kernel: [25920.677884] [] ? wake_atomic_t_function+0x60/0x60
Oct 21 18:02:46 r520-53 kernel: [25920.677945] [] ? dump_write+0x230/0x230 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.677996] [] spa_taskq_dispatch_sync+0x92/0xd0 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678036] [] dump_bytes+0x51/0x70 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678073] [] dump_write+0x11e/0x230 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678110] [] backup_cb+0x633/0x850 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678148] [] traverse_visitbp+0x47a/0x960 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678156] [] ? spl_kmem_alloc+0xaf/0x190 [spl]
Oct 21 18:02:46 r520-53 kernel: [25920.678193] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678231] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678268] [] traverse_visitbp+0x6cc/0x960 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678305] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678342] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678379] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678416] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678453] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678490] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678527] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678563] [] traverse_visitbp+0x865/0x960 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678600] [] traverse_impl+0x1ae/0x410 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678637] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678674] [] traverse_dataset+0x52/0x60 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678711] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678747] [] dmu_send_impl+0x409/0x560 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678784] [] dmu_send_obj+0x172/0x1e0 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678840] [] zfs_ioc_send+0xe9/0x2c0 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678847] [] ? strdup+0x3b/0x60 [spl]
Oct 21 18:02:46 r520-53 kernel: [25920.678903] [] zfsdev_ioctl+0x44b/0x4e0 [zfs]
Oct 21 18:02:46 r520-53 kernel: [25920.678909] [] do_vfs_ioctl+0x29f/0x490
Oct 21 18:02:46 r520-53 kernel: [25920.678915] [] ? __do_page_fault+0x1b4/0x400
Oct 21 18:02:46 r520-53 kernel: [25920.678918] [] SyS_ioctl+0x79/0x90
Oct 21 18:02:46 r520-53 kernel: [25920.678922] [] entry_SYSCALL_64_fastpath+0x16/0x71
Oct 21 18:04:46 r520-53 kernel: [26040.674655] INFO: task zfs:37162 blocked for more than 120 seconds.
Oct 21 18:04:46 r520-53 kernel: [26040.681862] Tainted: P O 4.4.0-43-generic #63-Ubuntu
Oct 21 18:04:46 r520-53 kernel: [26040.689463] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 21 18:04:46 r520-53 kernel: [26040.698446] zfs D ffff8806501e73a8 0 37162 22640 0x00000002
Oct 21 18:04:46 r520-53 kernel: [26040.698454] ffff8806501e73a8 ffffffff810c3712 ffff88080be42940 ffff880e3d3a3700
Oct 21 18:04:46 r520-53 kernel: [26040.698458] ffff8806501e8000 000000000006632f ffff88100b542bc8 0000000000000000
Oct 21 18:04:46 r520-53 kernel: [26040.698462] ffff88100afd1d00 ffff8806501e73c0 ffffffff8182d7c5 ffff88100b542b00
Oct 21 18:04:46 r520-53 kernel: [26040.698466] Call Trace:
Oct 21 18:04:46 r520-53 kernel: [26040.698498] [] ? __wake_up_common+0x52/0x90
Oct 21 18:04:46 r520-53 kernel: [26040.698512] [] schedule+0x35/0x80
Oct 21 18:04:46 r520-53 kernel: [26040.698534] [] taskq_wait_id+0x60/0xb0 [spl]
Oct 21 18:04:46 r520-53 kernel: [26040.698538] [] ? wake_atomic_t_function+0x60/0x60
Oct 21 18:04:46 r520-53 kernel: [26040.698601] [] ? dump_write+0x230/0x230 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.698658] [] spa_taskq_dispatch_sync+0x92/0xd0 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.698699] [] dump_bytes+0x51/0x70 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.698738] [] dump_write+0x11e/0x230 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.698776] [] backup_cb+0x633/0x850 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.698815] [] traverse_visitbp+0x47a/0x960 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.698823] [] ? spl_kmem_alloc+0xaf/0x190 [spl]
Oct 21 18:04:46 r520-53 kernel: [26040.698861] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.698899] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.698952] [] traverse_visitbp+0x6cc/0x960 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.698992] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699033] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699073] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699113] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699153] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699194] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699234] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699275] [] traverse_visitbp+0x865/0x960 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699315] [] traverse_impl+0x1ae/0x410 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699355] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699396] [] traverse_dataset+0x52/0x60 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699436] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699476] [] dmu_send_impl+0x409/0x560 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699516] [] dmu_send_obj+0x172/0x1e0 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699577] [] zfs_ioc_send+0xe9/0x2c0 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699586] [] ? strdup+0x3b/0x60 [spl]
Oct 21 18:04:46 r520-53 kernel: [26040.699646] [] zfsdev_ioctl+0x44b/0x4e0 [zfs]
Oct 21 18:04:46 r520-53 kernel: [26040.699653] [] do_vfs_ioctl+0x29f/0x490
Oct 21 18:04:46 r520-53 kernel: [26040.699659] [] ? __do_page_fault+0x1b4/0x400
Oct 21 18:04:46 r520-53 kernel: [26040.699662] [] SyS_ioctl+0x79/0x90
Oct 21 18:04:46 r520-53 kernel: [26040.699666] [] entry_SYSCALL_64_fastpath+0x16/0x71
Oct 21 18:06:46 r520-53 kernel: [26160.695303] INFO: task zfs:37162 blocked for more than 120 seconds.
Oct 21 18:06:46 r520-53 kernel: [26160.702509] Tainted: P O 4.4.0-43-generic #63-Ubuntu
Oct 21 18:06:46 r520-53 kernel: [26160.710107] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 21 18:06:46 r520-53 kernel: [26160.719095] zfs D ffff8806501e73a8 0 37162 22640 0x00000002
Oct 21 18:06:46 r520-53 kernel: [26160.719103] ffff8806501e73a8 ffffffff810c3712 ffff88080be42940 ffff880e3d3a3700
Oct 21 18:06:46 r520-53 kernel: [26160.719107] ffff8806501e8000 000000000006632f ffff88100b542bc8 0000000000000000
Oct 21 18:06:46 r520-53 kernel: [26160.719110] ffff88100afd1d00 ffff8806501e73c0 ffffffff8182d7c5 ffff88100b542b00
Oct 21 18:06:46 r520-53 kernel: [26160.719114] Call Trace:
Oct 21 18:06:46 r520-53 kernel: [26160.719141] [] ? __wake_up_common+0x52/0x90
Oct 21 18:06:46 r520-53 kernel: [26160.719158] [] schedule+0x35/0x80
Oct 21 18:06:46 r520-53 kernel: [26160.719179] [] taskq_wait_id+0x60/0xb0 [spl]
Oct 21 18:06:46 r520-53 kernel: [26160.719183] [] ? wake_atomic_t_function+0x60/0x60
Oct 21 18:06:46 r520-53 kernel: [26160.719241] [] ? dump_write+0x230/0x230 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719297] [] spa_taskq_dispatch_sync+0x92/0xd0 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719338] [] dump_bytes+0x51/0x70 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719377] [] dump_write+0x11e/0x230 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719414] [] backup_cb+0x633/0x850 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719453] [] traverse_visitbp+0x47a/0x960 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719461] [] ? spl_kmem_alloc+0xaf/0x190 [spl]
Oct 21 18:06:46 r520-53 kernel: [26160.719499] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719537] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719574] [] traverse_visitbp+0x6cc/0x960 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719612] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719650] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719687] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719725] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719762] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719800] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719838] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719875] [] traverse_visitbp+0x865/0x960 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719912] [] traverse_impl+0x1ae/0x410 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719950] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.719988] [] traverse_dataset+0x52/0x60 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.720025] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.720063] [] dmu_send_impl+0x409/0x560 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.720100] [] dmu_send_obj+0x172/0x1e0 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.720157] [] zfs_ioc_send+0xe9/0x2c0 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.720165] [] ? strdup+0x3b/0x60 [spl]
Oct 21 18:06:46 r520-53 kernel: [26160.720221] [] zfsdev_ioctl+0x44b/0x4e0 [zfs]
Oct 21 18:06:46 r520-53 kernel: [26160.720228] [] do_vfs_ioctl+0x29f/0x490
Oct 21 18:06:46 r520-53 kernel: [26160.720233] [] ? __do_page_fault+0x1b4/0x400
Oct 21 18:06:46 r520-53 kernel: [26160.720236] [] SyS_ioctl+0x79/0x90
Oct 21 18:06:46 r520-53 kernel: [26160.720241] [] entry_SYSCALL_64_fastpath+0x16/0x71
Oct 21 18:08:46 r520-53 kernel: [26280.715932] INFO: task zfs:37162 blocked for more than 120 seconds.
Oct 21 18:08:46 r520-53 kernel: [26280.723137] Tainted: P O 4.4.0-43-generic #63-Ubuntu
Oct 21 18:08:46 r520-53 kernel: [26280.730743] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 21 18:08:46 r520-53 kernel: [26280.739735] zfs D ffff8806501e73a8 0 37162 22640 0x00000002
Oct 21 18:08:46 r520-53 kernel: [26280.739743] ffff8806501e73a8 ffffffff810c3712 ffff88080be42940 ffff880e3d3a3700
Oct 21 18:08:46 r520-53 kernel: [26280.739747] ffff8806501e8000 000000000006632f ffff88100b542bc8 0000000000000000
Oct 21 18:08:46 r520-53 kernel: [26280.739751] ffff88100afd1d00 ffff8806501e73c0 ffffffff8182d7c5 ffff88100b542b00
Oct 21 18:08:46 r520-53 kernel: [26280.739755] Call Trace:
Oct 21 18:08:46 r520-53 kernel: [26280.739784] [] ? __wake_up_common+0x52/0x90
Oct 21 18:08:46 r520-53 kernel: [26280.739792] [] schedule+0x35/0x80
Oct 21 18:08:46 r520-53 kernel: [26280.739813] [] taskq_wait_id+0x60/0xb0 [spl]
Oct 21 18:08:46 r520-53 kernel: [26280.739817] [] ? wake_atomic_t_function+0x60/0x60
Oct 21 18:08:46 r520-53 kernel: [26280.739881] [] ? dump_write+0x230/0x230 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.739931] [] spa_taskq_dispatch_sync+0x92/0xd0 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.739972] [] dump_bytes+0x51/0x70 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740011] [] dump_write+0x11e/0x230 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740049] [] backup_cb+0x633/0x850 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740087] [] traverse_visitbp+0x47a/0x960 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740096] [] ? spl_kmem_alloc+0xaf/0x190 [spl]
Oct 21 18:08:46 r520-53 kernel: [26280.740134] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740172] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740210] [] traverse_visitbp+0x6cc/0x960 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740247] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740285] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740322] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740359] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740397] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740434] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740471] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740509] [] traverse_visitbp+0x865/0x960 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740546] [] traverse_impl+0x1ae/0x410 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740584] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740621] [] traverse_dataset+0x52/0x60 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740659] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740696] [] dmu_send_impl+0x409/0x560 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740734] [] dmu_send_obj+0x172/0x1e0 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740790] [] zfs_ioc_send+0xe9/0x2c0 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740799] [] ? strdup+0x3b/0x60 [spl]
Oct 21 18:08:46 r520-53 kernel: [26280.740854] [] zfsdev_ioctl+0x44b/0x4e0 [zfs]
Oct 21 18:08:46 r520-53 kernel: [26280.740861] [] do_vfs_ioctl+0x29f/0x490
Oct 21 18:08:46 r520-53 kernel: [26280.740867] [] ? __do_page_fault+0x1b4/0x400
Oct 21 18:08:46 r520-53 kernel: [26280.740870] [] SyS_ioctl+0x79/0x90
Oct 21 18:08:46 r520-53 kernel: [26280.740875] [] entry_SYSCALL_64_fastpath+0x16/0x71
Oct 21 18:09:01 r520-53 CRON[37312]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
Oct 21 18:10:46 r520-53 kernel: [26400.736600] INFO: task zfs:37162 blocked for more than 120 seconds.
Oct 21 18:10:46 r520-53 kernel: [26400.743811] Tainted: P O 4.4.0-43-generic #63-Ubuntu
Oct 21 18:10:46 r520-53 kernel: [26400.756671] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 21 18:10:46 r520-53 kernel: [26400.776445] zfs D ffff8806501e73a8 0 37162 22640 0x00000002
Oct 21 18:10:46 r520-53 kernel: [26400.776453] ffff8806501e73a8 ffffffff810c3712 ffff88080be42940 ffff880e3d3a3700
Oct 21 18:10:46 r520-53 kernel: [26400.776457] ffff8806501e8000 000000000006632f ffff88100b542bc8 0000000000000000
Oct 21 18:10:46 r520-53 kernel: [26400.776460] ffff88100afd1d00 ffff8806501e73c0 ffffffff8182d7c5 ffff88100b542b00
Oct 21 18:10:46 r520-53 kernel: [26400.776476] Call Trace:
Oct 21 18:10:46 r520-53 kernel: [26400.776491] [] ? __wake_up_common+0x52/0x90
Oct 21 18:10:46 r520-53 kernel: [26400.776504] [] schedule+0x35/0x80
Oct 21 18:10:46 r520-53 kernel: [26400.776524] [] taskq_wait_id+0x60/0xb0 [spl]
Oct 21 18:10:46 r520-53 kernel: [26400.776529] [] ? wake_atomic_t_function+0x60/0x60
Oct 21 18:10:46 r520-53 kernel: [26400.776590] [] ? dump_write+0x230/0x230 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.776641] [] spa_taskq_dispatch_sync+0x92/0xd0 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.776680] [] dump_bytes+0x51/0x70 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.776715] [] dump_write+0x11e/0x230 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.776750] [] backup_cb+0x633/0x850 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.776786] [] traverse_visitbp+0x47a/0x960 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.776793] [] ? spl_kmem_alloc+0xaf/0x190 [spl]
Oct 21 18:10:46 r520-53 kernel: [26400.776828] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.776864] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.776899] [] traverse_visitbp+0x6cc/0x960 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.776934] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.776969] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777003] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777038] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777073] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777108] [] traverse_visitbp+0x5c0/0x960 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777143] [] traverse_dnode+0x7f/0xe0 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777177] [] traverse_visitbp+0x865/0x960 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777212] [] traverse_impl+0x1ae/0x410 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777247] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777281] [] traverse_dataset+0x52/0x60 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777315] [] ? dmu_recv_end_check+0x210/0x210 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777349] [] dmu_send_impl+0x409/0x560 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777383] [] dmu_send_obj+0x172/0x1e0 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777436] [] zfs_ioc_send+0xe9/0x2c0 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777443] [] ? strdup+0x3b/0x60 [spl]
Oct 21 18:10:46 r520-53 kernel: [26400.777495] [] zfsdev_ioctl+0x44b/0x4e0 [zfs]
Oct 21 18:10:46 r520-53 kernel: [26400.777501] [] do_vfs_ioctl+0x29f/0x490
Oct 21 18:10:46 r520-53 kernel: [26400.777506] [] ? __do_page_fault+0x1b4/0x400
Oct 21 18:10:46 r520-53 kernel: [26400.777509] [] SyS_ioctl+0x79/0x90
Oct 21 18:10:46 r520-53 kernel: [26400.777513] [] entry_SYSCALL_64_fastpath+0x16/0x71
Oct 21 18:12:46 r520-53 kernel: [26520.773238] INFO: task zfs:37162 blocked for more than 120 seconds.
Oct 21 18:12:46 r520-53 kernel: [26520.785873] Tainted: P O 4.4.0-43-generic #63-Ubuntu
Oct 21 18:12:46 r520-53 kernel: [26520.798862] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 21 18:12:46 r520-53 kernel: [26520.818747] zfs D ffff8806501e73a8 0 37162 22640 0x00000002
Oct 21 18:12:46 r520-53 kernel: [26520.818754] ffff8806501e73a8 ffffffff810c3712 ffff88080be42940 ffff880e3d3a3700
Oct 21 18:12:46 r520-53 kernel: [26520.818758] ffff8806501e8000 000000000006632f ffff88100b542bc8 0000000000000000
Oct 21 18:12:46 r520-53 kernel: [26520.818761] ffff88100afd1d00 ffff8806501e73c0 ffffffff8182d7c5 ffff88100b542b00
Oct 21 18:12:46 r520-53 kernel: [26520.818765] Call Trace:
Oct 21 18:12:46 r520-53 kernel: [26520.818779] [] ? __wake_up_common+0x52/0x90
Oct 21 18:12:46 r520-53 kernel: [26520.818796] [] schedule+0x35/0x80
Oct 21 18:12:46 r520-53 kernel: [26520.818831] [] taskq_wait_id+0x60/0xb0 [spl]
Oct 21 18:12:46 r520-53 kernel: [26520.818835] [] ? wake_atomic_t_function+0x60/0x60
Oct 21 18:12:46 r520-53 kernel: [26520.818897] [] ? dump_write+0x230/0x230 [zfs]
Oct 21 18:12:46 r520-53 kernel: [26520.818951] [] spa_taskq_dispatch_sync+0x92/0xd0 [zfs]
Oct 21 18:12:46 r520-53 kernel: [26520.818989] [] dump_bytes+0x51/0x70 [zfs]
Oct 21 18:12:46 r520-53 kernel: [26520.819025] [] dump_write+0x11e/0x230 [zfs]
Oct 21 18:12:46 r520-53 kernel: [26520.819060] [] backup_cb+0x633/0x850 [zfs]
Oct 21 18:12:46 r520-53 kernel: [26520.819097]
The text was updated successfully, but these errors were encountered: