Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deadlock/lockup under load #414

Closed
cdubbm opened this issue Sep 28, 2011 · 3 comments
Closed

Deadlock/lockup under load #414

cdubbm opened this issue Sep 28, 2011 · 3 comments
Milestone

Comments

@cdubbm
Copy link

cdubbm commented Sep 28, 2011

The machine in question is running 2.6.39-1 and has failed with this a few times when under load:

Sep 27 14:37:54 cheq-blz-01 kernel: INFO: task z_wr_iss/1:6318 blocked for more than 120 seconds.
Sep 27 14:37:54 cheq-blz-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 27 14:37:54 cheq-blz-01 kernel: z_wr_iss/1 D ffff88083fc519c0 0 6318 2 0x00000000
Sep 27 14:37:55 cheq-blz-01 kernel: ffff88081d657830 0000000000000046 ffff88080ab740c0 ffff880800000001
Sep 27 14:37:55 cheq-blz-01 kernel: ffff88083fd119c0 0000000000000003 ffff88083fc519c0 0000000000000004
Sep 27 14:37:55 cheq-blz-01 kernel: 0000000000000001 ffffffff8102a450 ffff88083fd119c0 ffff88080bbbc7b0
Sep 27 14:37:55 cheq-blz-01 kernel: Call Trace:
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? enqueue_task+0x51/0x5d
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? check_preempt_curr+0x27/0x66
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? try_to_wake_up+0x2f4/0x307
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? prepare_to_wait_exclusive+0x38/0x70
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? cv_wait_common+0x72/0xb8 [spl]
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? wake_up_bit+0x22/0x22
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? __wake_up+0x30/0x44
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? txg_wait_open+0x51/0x63 [zfs]
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? dmu_tx_assign+0xe5/0x331 [zfs]
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? zfs_inactive+0xa3/0x185 [zfs]
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? evict+0x67/0x106
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? dispose_list+0x25/0x33
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? shrink_icache_memory+0x22e/0x25d
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? shrink_slab+0xe1/0x152
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? try_to_free_pages+0x1ce/0x342
Sep 27 14:37:55 cheq-blz-01 kernel: [] ? __alloc_pages_nodemask+0x3bf/0x682
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? kmem_getpages+0x52/0x11a
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? fallback_alloc+0x10f/0x1a9
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? ____cache_alloc_node+0xad/0xef
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? __kmalloc+0xe4/0x154
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? kmem_alloc_debug+0x7c/0xbd [spl]
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? kmem_alloc_debug+0x7c/0xbd [spl]
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? vdev_raidz_io_start+0x10f/0x533 [zfs]
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? zio_create+0x296/0x2a8 [zfs]
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? zio_nowait+0xd6/0xf8 [zfs]
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? vdev_mirror_io_start+0x2c9/0x309 [zfs]
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? vdev_uberblock_sync_done+0x2d/0x2d [zfs]
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? zio_vdev_io_start+0x3f/0x235 [zfs]
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? zio_execute+0xb4/0xd5 [zfs]
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? taskq_thread+0x1b8/0x2a1 [spl]
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? try_to_wake_up+0x307/0x307
Sep 27 14:37:56 cheq-blz-01 kernel: [] ? spl_taskq_init+0x4c/0x4c [spl]
Sep 27 14:37:57 cheq-blz-01 kernel: [] ? kthread+0x7e/0x86
Sep 27 14:37:57 cheq-blz-01 kernel: [] ? kernel_thread_helper+0x4/0x10
Sep 27 14:37:57 cheq-blz-01 kernel: [] ? kthread_stop+0xa6/0xa6
Sep 27 14:37:57 cheq-blz-01 kernel: [] ? gs_change+0xb/0xb

@randomei
Copy link

randomei commented Oct 9, 2011

Simillar problem here, running kubuntu 11.04 with 2.6.38-11 kernel, zfs from ubuntu repo - 0.6.0.33, 4Gb RAM. System hangs on disk load (downloading torrent) with this in syslog:

Oct 9 04:50:13 drcomp kernel: [ 9715.172576] TCP: Possible SYN flooding on port 6881. Sending cookies.
Oct 9 04:50:19 drcomp kernel: [ 9720.790063] INFO: task kthreadd:2 blocked for more than 120 seconds.
Oct 9 04:50:19 drcomp kernel: [ 9720.790072] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 9 04:50:19 drcomp kernel: [ 9720.790079] kthreadd D 0000000000000001 0 2 0 0x00000000
Oct 9 04:50:19 drcomp kernel: [ 9720.790090] ffff8801276d16a0 0000000000000046 ffff8801276d1fd8 ffff8801276d0000
Oct 9 04:50:19 drcomp kernel: [ 9720.790100] 0000000000013d00 ffff8801276c1a98 ffff8801276d1fd8 0000000000013d00
Oct 9 04:50:19 drcomp kernel: [ 9720.790109] ffff880094c896e0 ffff8801276c16e0 ffff8801276d16c0 ffff880107267b30
Oct 9 04:50:19 drcomp kernel: [ 9720.790118] Call Trace:
Oct 9 04:50:19 drcomp kernel: [ 9720.790162] [] cv_wait_common+0x77/0xd0 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.790176] [] ? autoremove_wake_function+0x0/0x40
Oct 9 04:50:19 drcomp kernel: [ 9720.790198] [] __cv_wait+0x13/0x20 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.790303] [] txg_wait_open+0x73/0xb0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.790360] [] dmu_tx_wait+0xed/0xf0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.790432] [] zfs_putpage+0x25b/0x270 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.790443] [] ? rmap_walk+0x107/0x250
Oct 9 04:50:19 drcomp kernel: [ 9720.790509] [] zpl_putpage+0x2f/0x40 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.790572] [] zpl_writepage+0x12/0x20 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.790583] [] fallback_migrate_page+0xc7/0x130
Oct 9 04:50:19 drcomp kernel: [ 9720.790592] [] move_to_new_page+0x103/0x120
Oct 9 04:50:19 drcomp kernel: [ 9720.790599] [] unmap_and_move+0x2f6/0x350
Oct 9 04:50:19 drcomp kernel: [ 9720.790609] [] ? __mod_zone_page_state+0x49/0x50
Oct 9 04:50:19 drcomp kernel: [ 9720.790619] [] ? compaction_alloc+0x0/0x70
Oct 9 04:50:19 drcomp kernel: [ 9720.790626] [] migrate_pages+0xa2/0x150
Oct 9 04:50:19 drcomp kernel: [ 9720.790635] [] compact_zone+0xa4/0x220
Oct 9 04:50:19 drcomp kernel: [ 9720.790643] [] compact_zone_order+0xac/0xf0
Oct 9 04:50:19 drcomp kernel: [ 9720.790652] [] try_to_compact_pages+0xcd/0x100
Oct 9 04:50:19 drcomp kernel: [ 9720.790662] [] __alloc_pages_direct_compact+0xcf/0x190
Oct 9 04:50:19 drcomp kernel: [ 9720.790698] [] __alloc_pages_nodemask+0x43d/0x840
Oct 9 04:50:19 drcomp kernel: [ 9720.790709] [] ? _raw_spin_lock_irqsave+0x2f/0x40
Oct 9 04:50:19 drcomp kernel: [ 9720.790718] [] alloc_pages_current+0xa5/0x110
Oct 9 04:50:19 drcomp kernel: [ 9720.790725] [] __get_free_pages+0xe/0x50
Oct 9 04:50:19 drcomp kernel: [ 9720.790734] [] dup_task_struct+0x46/0x190
Oct 9 04:50:19 drcomp kernel: [ 9720.790742] [] copy_process+0xbe/0xe80
Oct 9 04:50:19 drcomp kernel: [ 9720.790750] [] do_fork+0x59/0x340
Oct 9 04:50:19 drcomp kernel: [ 9720.790759] [] ? perf_event_task_sched_out+0x2e/0xa0
Oct 9 04:50:19 drcomp kernel: [ 9720.790768] [] ? reschedule_interrupt+0xe/0x20
Oct 9 04:50:19 drcomp kernel: [ 9720.790777] [] kernel_thread+0x76/0x80
Oct 9 04:50:19 drcomp kernel: [ 9720.790785] [] ? kthread+0x0/0xa0
Oct 9 04:50:19 drcomp kernel: [ 9720.790792] [] ? kernel_thread_helper+0x0/0x10
Oct 9 04:50:19 drcomp kernel: [ 9720.790799] [] ? _raw_spin_lock+0xe/0x20
Oct 9 04:50:19 drcomp kernel: [ 9720.790806] [] kthreadd+0x108/0x140
Oct 9 04:50:19 drcomp kernel: [ 9720.790816] [] ? __switch_to+0xc0/0x2f0
Oct 9 04:50:19 drcomp kernel: [ 9720.790825] [] ? finish_task_switch+0x41/0xe0
Oct 9 04:50:19 drcomp kernel: [ 9720.790832] [] kernel_thread_helper+0x4/0x10
Oct 9 04:50:19 drcomp kernel: [ 9720.790840] [] ? kthreadd+0x0/0x140
Oct 9 04:50:19 drcomp kernel: [ 9720.790847] [] ? kernel_thread_helper+0x0/0x10
Oct 9 04:50:19 drcomp kernel: [ 9720.790858] INFO: task kswapd0:52 blocked for more than 120 seconds.
Oct 9 04:50:19 drcomp kernel: [ 9720.790863] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 9 04:50:19 drcomp kernel: [ 9720.790868] kswapd0 D 0000000000000000 0 52 2 0x00000000
Oct 9 04:50:19 drcomp kernel: [ 9720.790878] ffff8801232dd940 0000000000000046 ffff8801232ddfd8 ffff8801232dc000
Oct 9 04:50:19 drcomp kernel: [ 9720.790887] 0000000000013d00 ffff880124391a98 ffff8801232ddfd8 0000000000013d00
Oct 9 04:50:19 drcomp kernel: [ 9720.790895] ffff88009e57adc0 ffff8801243916e0 ffff8801232dd960 ffff880107267b30
Oct 9 04:50:19 drcomp kernel: [ 9720.790904] Call Trace:
Oct 9 04:50:19 drcomp kernel: [ 9720.790925] [] cv_wait_common+0x77/0xd0 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.790933] [] ? autoremove_wake_function+0x0/0x40
Oct 9 04:50:19 drcomp kernel: [ 9720.790952] [] __cv_wait+0x13/0x20 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.791020] [] txg_wait_open+0x73/0xb0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.791076] [] dmu_tx_wait+0xed/0xf0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.791142] [] zfs_putpage+0x25b/0x270 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.791152] [] ? ____pagevec_lru_add+0x1ab/0x1d0
Oct 9 04:50:19 drcomp kernel: [ 9720.791215] [] zpl_putpage+0x2f/0x40 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.791278] [] zpl_writepage+0x12/0x20 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.791287] [] pageout.clone.33+0x131/0x290
Oct 9 04:50:19 drcomp kernel: [ 9720.791297] [] ? __mem_cgroup_uncharge_common+0xdb/0x190
Oct 9 04:50:19 drcomp kernel: [ 9720.791305] [] shrink_page_list+0x396/0x5c0
Oct 9 04:50:19 drcomp kernel: [ 9720.791313] [] shrink_inactive_list+0x11a/0x450
Oct 9 04:50:19 drcomp kernel: [ 9720.791319] [] ? _raw_spin_lock_irq+0x15/0x20
Oct 9 04:50:19 drcomp kernel: [ 9720.791328] [] ? get_scan_count+0x1ab/0x290
Oct 9 04:50:19 drcomp kernel: [ 9720.791335] [] shrink_zone+0x1f2/0x2b0
Oct 9 04:50:19 drcomp kernel: [ 9720.791342] [] balance_pgdat+0x50a/0x6e0
Oct 9 04:50:19 drcomp kernel: [ 9720.791350] [] kswapd+0xea/0x1d0
Oct 9 04:50:19 drcomp kernel: [ 9720.791357] [] ? kswapd+0x0/0x1d0
Oct 9 04:50:19 drcomp kernel: [ 9720.791363] [] kthread+0x96/0xa0
Oct 9 04:50:19 drcomp kernel: [ 9720.791371] [] kernel_thread_helper+0x4/0x10
Oct 9 04:50:19 drcomp kernel: [ 9720.791378] [] ? kthread+0x0/0xa0
Oct 9 04:50:19 drcomp kernel: [ 9720.791385] [] ? kernel_thread_helper+0x0/0x10
Oct 9 04:50:19 drcomp kernel: [ 9720.791409] INFO: task Xorg:1135 blocked for more than 120 seconds.
Oct 9 04:50:19 drcomp kernel: [ 9720.791413] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 9 04:50:19 drcomp kernel: [ 9720.791418] Xorg D 0000000000000000 0 1135 1128 0x00400004
Oct 9 04:50:19 drcomp kernel: [ 9720.791427] ffff880121b474e8 0000000000000082 ffff880121b47fd8 ffff880121b46000
Oct 9 04:50:19 drcomp kernel: [ 9720.791436] 0000000000013d00 ffff880121b51a98 ffff880121b47fd8 0000000000013d00
Oct 9 04:50:19 drcomp kernel: [ 9720.791444] ffff8801243916e0 ffff880121b516e0 ffff880121b47508 ffff880107267b30
Oct 9 04:50:19 drcomp kernel: [ 9720.791452] Call Trace:
Oct 9 04:50:19 drcomp kernel: [ 9720.791472] [] cv_wait_common+0x77/0xd0 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.791480] [] ? autoremove_wake_function+0x0/0x40
Oct 9 04:50:19 drcomp kernel: [ 9720.791498] [] __cv_wait+0x13/0x20 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.791567] [] txg_wait_open+0x73/0xb0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.791623] [] dmu_tx_wait+0xed/0xf0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.791689] [] zfs_putpage+0x25b/0x270 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.791697] [] ? rmap_walk+0x107/0x250
Oct 9 04:50:19 drcomp kernel: [ 9720.791760] [] zpl_putpage+0x2f/0x40 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.791823] [] zpl_writepage+0x12/0x20 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.791832] [] fallback_migrate_page+0xc7/0x130
Oct 9 04:50:19 drcomp kernel: [ 9720.791840] [] move_to_new_page+0x103/0x120
Oct 9 04:50:19 drcomp kernel: [ 9720.791848] [] unmap_and_move+0x2f6/0x350
Oct 9 04:50:19 drcomp kernel: [ 9720.791856] [] ? __mod_zone_page_state+0x49/0x50
Oct 9 04:50:19 drcomp kernel: [ 9720.791864] [] ? compaction_alloc+0x0/0x70
Oct 9 04:50:19 drcomp kernel: [ 9720.791872] [] migrate_pages+0xa2/0x150
Oct 9 04:50:19 drcomp kernel: [ 9720.791880] [] compact_zone+0xa4/0x220
Oct 9 04:50:19 drcomp kernel: [ 9720.791889] [] compact_zone_order+0xac/0xf0
Oct 9 04:50:19 drcomp kernel: [ 9720.791896] [] ? finish_task_switch+0x41/0xe0
Oct 9 04:50:19 drcomp kernel: [ 9720.791905] [] try_to_compact_pages+0xcd/0x100
Oct 9 04:50:19 drcomp kernel: [ 9720.791913] [] __alloc_pages_direct_compact+0xcf/0x190
Oct 9 04:50:19 drcomp kernel: [ 9720.791922] [] __alloc_pages_nodemask+0x43d/0x840
Oct 9 04:50:19 drcomp kernel: [ 9720.791931] [] alloc_pages_current+0xa5/0x110
Oct 9 04:50:19 drcomp kernel: [ 9720.791939] [] new_slab+0x1f5/0x290
Oct 9 04:50:19 drcomp kernel: [ 9720.791946] [] __slab_alloc+0x1b2/0x390
Oct 9 04:50:19 drcomp kernel: [ 9720.792655] [] ? os_alloc_mem+0xaf/0x100 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.792664] [] __kmalloc+0x14d/0x160
Oct 9 04:50:19 drcomp kernel: [ 9720.793090] [] ? os_alloc_mem+0xaf/0x100 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.793512] [] os_alloc_mem+0xaf/0x100 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.793940] [] _nv023258rm+0xb2/0xc2 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.794454] [] ? _nv013688rm+0xb3/0x5c7 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.794969] [] ? _nv012930rm+0x1e3/0x257 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.795485] [] ? _nv013310rm+0xf4/0x387 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.796004] [] ? _nv013311rm+0xcbf/0xf94 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.796336] [] ? _nv002287rm+0x404/0x485 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.796667] [] ? _nv001903rm+0x84/0xc1 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.796995] [] ? _nv001903rm+0x47/0xc1 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.797419] [] ? _nv002308rm+0x475/0x751 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.797842] [] ? rm_ioctl+0x6d/0x177 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.798266] [] ? nv_kern_ioctl+0x155/0x460 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.798687] [] ? nv_kern_unlocked_ioctl+0x21/0x30 [nvidia]
Oct 9 04:50:19 drcomp kernel: [ 9720.798698] [] ? do_vfs_ioctl+0x8f/0x360
Oct 9 04:50:19 drcomp kernel: [ 9720.798707] [] ? vfs_read+0x120/0x180
Oct 9 04:50:19 drcomp kernel: [ 9720.798715] [] ? sys_ioctl+0x91/0xa0
Oct 9 04:50:19 drcomp kernel: [ 9720.798722] [] ? system_call_fastpath+0x16/0x1b
Oct 9 04:50:19 drcomp kernel: [ 9720.798767] INFO: task txg_sync:1492 blocked for more than 120 seconds.
Oct 9 04:50:19 drcomp kernel: [ 9720.798771] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 9 04:50:19 drcomp kernel: [ 9720.798776] txg_sync D 0000000000000000 0 1492 2 0x00000000
Oct 9 04:50:19 drcomp kernel: [ 9720.798785] ffff8801006d5260 0000000000000046 ffff8801006d5fd8 ffff8801006d4000
Oct 9 04:50:19 drcomp kernel: [ 9720.798794] 0000000000013d00 ffff8801028b5f38 ffff8801006d5fd8 0000000000013d00
Oct 9 04:50:19 drcomp kernel: [ 9720.798802] ffff880121b516e0 ffff8801028b5b80 ffff8801006d5280 ffff880107267b30
Oct 9 04:50:19 drcomp kernel: [ 9720.798810] Call Trace:
Oct 9 04:50:19 drcomp kernel: [ 9720.798832] [] cv_wait_common+0x77/0xd0 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.798840] [] ? autoremove_wake_function+0x0/0x40
Oct 9 04:50:19 drcomp kernel: [ 9720.798859] [] __cv_wait+0x13/0x20 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.798932] [] txg_wait_open+0x73/0xb0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.798989] [] dmu_tx_wait+0xed/0xf0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799055] [] zfs_putpage+0x25b/0x270 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799064] [] ? rmap_walk+0x107/0x250
Oct 9 04:50:19 drcomp kernel: [ 9720.799128] [] zpl_putpage+0x2f/0x40 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799191] [] zpl_writepage+0x12/0x20 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799200] [] fallback_migrate_page+0xc7/0x130
Oct 9 04:50:19 drcomp kernel: [ 9720.799209] [] move_to_new_page+0x103/0x120
Oct 9 04:50:19 drcomp kernel: [ 9720.799216] [] unmap_and_move+0x2f6/0x350
Oct 9 04:50:19 drcomp kernel: [ 9720.799225] [] ? compaction_alloc+0x0/0x70
Oct 9 04:50:19 drcomp kernel: [ 9720.799232] [] migrate_pages+0xa2/0x150
Oct 9 04:50:19 drcomp kernel: [ 9720.799241] [] compact_zone+0xa4/0x220
Oct 9 04:50:19 drcomp kernel: [ 9720.799249] [] compact_zone_order+0xac/0xf0
Oct 9 04:50:19 drcomp kernel: [ 9720.799258] [] try_to_compact_pages+0xcd/0x100
Oct 9 04:50:19 drcomp kernel: [ 9720.799267] [] __alloc_pages_direct_compact+0xcf/0x190
Oct 9 04:50:19 drcomp kernel: [ 9720.799275] [] __alloc_pages_nodemask+0x43d/0x840
Oct 9 04:50:19 drcomp kernel: [ 9720.799293] [] ? kmem_alloc_debug+0xeb/0x130 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.799304] [] ? cpumask_next_and+0x36/0x50
Oct 9 04:50:19 drcomp kernel: [ 9720.799312] [] alloc_pages_current+0xa5/0x110
Oct 9 04:50:19 drcomp kernel: [ 9720.799320] [] new_slab+0x1f5/0x290
Oct 9 04:50:19 drcomp kernel: [ 9720.799328] [] __slab_alloc+0x1b2/0x390
Oct 9 04:50:19 drcomp kernel: [ 9720.799344] [] ? kmem_alloc_debug+0xbb/0x130 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.799352] [] __kmalloc+0x14d/0x160
Oct 9 04:50:19 drcomp kernel: [ 9720.799369] [] ? kmem_alloc_debug+0xbb/0x130 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.799377] [] ? cpumask_next_and+0x36/0x50
Oct 9 04:50:19 drcomp kernel: [ 9720.799394] [] kmem_alloc_debug+0xbb/0x130 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.799444] [] dbuf_hold_impl+0x58/0xc0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799494] [] dbuf_hold+0x20/0x30 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799551] [] dnode_hold_impl+0xd2/0x5c0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799560] [] ? dequeue_task_fair+0x8f/0xa0
Oct 9 04:50:19 drcomp kernel: [ 9720.799569] [] ? __switch_to+0xc0/0x2f0
Oct 9 04:50:19 drcomp kernel: [ 9720.799577] [] ? finish_task_switch+0x41/0xe0
Oct 9 04:50:19 drcomp kernel: [ 9720.799635] [] dnode_hold+0x19/0x20 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799687] [] dmu_buf_hold_array+0x39/0x90 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799705] [] ? kmem_free_debug+0x16/0x20 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.799757] [] dmu_write+0x60/0x180 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799774] [] ? kmem_free_debug+0x16/0x20 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.799843] [] space_map_sync+0x21a/0x310 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799911] [] metaslab_sync+0x111/0x370 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.799921] [] ? mutex_lock+0x1e/0x50
Oct 9 04:50:19 drcomp kernel: [ 9720.799989] [] vdev_sync+0x6f/0x140 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.800123] [] spa_sync+0x41b/0x9b0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.800131] [] ? autoremove_wake_function+0x16/0x40
Oct 9 04:50:19 drcomp kernel: [ 9720.800139] [] ? __wake_up+0x53/0x70
Oct 9 04:50:19 drcomp kernel: [ 9720.800208] [] txg_sync_thread+0x241/0x3c0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.800278] [] ? txg_sync_thread+0x0/0x3c0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.800296] [] thread_generic_wrapper+0x78/0x90 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.800314] [] ? thread_generic_wrapper+0x0/0x90 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.800322] [] kthread+0x96/0xa0
Oct 9 04:50:19 drcomp kernel: [ 9720.800330] [] kernel_thread_helper+0x4/0x10
Oct 9 04:50:19 drcomp kernel: [ 9720.800337] [] ? kthread+0x0/0xa0
Oct 9 04:50:19 drcomp kernel: [ 9720.800345] [] ? kernel_thread_helper+0x0/0x10
Oct 9 04:50:19 drcomp kernel: [ 9720.800399] INFO: task amarok:2346 blocked for more than 120 seconds.
Oct 9 04:50:19 drcomp kernel: [ 9720.800403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 9 04:50:19 drcomp kernel: [ 9720.800408] amarok D 0000000000000000 0 2346 1 0x00000000
Oct 9 04:50:19 drcomp kernel: [ 9720.800416] ffff8800b26f9378 0000000000000082 ffff8800b26f9fd8 ffff8800b26f8000
Oct 9 04:50:19 drcomp kernel: [ 9720.800425] 0000000000013d00 ffff8800b25d1a98 ffff8800b26f9fd8 0000000000013d00
Oct 9 04:50:19 drcomp kernel: [ 9720.800433] ffff8800af605b80 ffff8800b25d16e0 ffff8800b26f9398 ffff880107267b30
Oct 9 04:50:19 drcomp kernel: [ 9720.800441] Call Trace:
Oct 9 04:50:19 drcomp kernel: [ 9720.800461] [] cv_wait_common+0x77/0xd0 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.800470] [] ? autoremove_wake_function+0x0/0x40
Oct 9 04:50:19 drcomp kernel: [ 9720.800489] [] __cv_wait+0x13/0x20 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.800557] [] txg_wait_open+0x73/0xb0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.800614] [] dmu_tx_wait+0xed/0xf0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.800680] [] zfs_putpage+0x25b/0x270 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.800688] [] ? rmap_walk+0x107/0x250
Oct 9 04:50:19 drcomp kernel: [ 9720.800752] [] zpl_putpage+0x2f/0x40 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.800815] [] zpl_writepage+0x12/0x20 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.800824] [] fallback_migrate_page+0xc7/0x130
Oct 9 04:50:19 drcomp kernel: [ 9720.800833] [] move_to_new_page+0x103/0x120
Oct 9 04:50:19 drcomp kernel: [ 9720.800841] [] unmap_and_move+0x2f6/0x350
Oct 9 04:50:19 drcomp kernel: [ 9720.800849] [] ? compaction_alloc+0x0/0x70
Oct 9 04:50:19 drcomp kernel: [ 9720.800857] [] migrate_pages+0xa2/0x150
Oct 9 04:50:19 drcomp kernel: [ 9720.800865] [] compact_zone+0xa4/0x220
Oct 9 04:50:19 drcomp kernel: [ 9720.800874] [] ? check_preempt_wakeup+0x1c7/0x280
Oct 9 04:50:19 drcomp kernel: [ 9720.800882] [] compact_zone_order+0xac/0xf0
Oct 9 04:50:19 drcomp kernel: [ 9720.800890] [] ? prep_new_page+0x142/0x1b0
Oct 9 04:50:19 drcomp kernel: [ 9720.800899] [] try_to_compact_pages+0xcd/0x100
Oct 9 04:50:19 drcomp kernel: [ 9720.800907] [] __alloc_pages_direct_compact+0xcf/0x190
Oct 9 04:50:19 drcomp kernel: [ 9720.800916] [] __alloc_pages_nodemask+0x43d/0x840
Oct 9 04:50:19 drcomp kernel: [ 9720.800926] [] ? __switch_to+0x20e/0x2f0
Oct 9 04:50:19 drcomp kernel: [ 9720.800934] [] alloc_pages_current+0xa5/0x110
Oct 9 04:50:19 drcomp kernel: [ 9720.800941] [] new_slab+0x1f5/0x290
Oct 9 04:50:19 drcomp kernel: [ 9720.800948] [] __slab_alloc+0x1b2/0x390
Oct 9 04:50:19 drcomp kernel: [ 9720.800959] [] ? sock_alloc_send_pskb+0xf2/0x260
Oct 9 04:50:19 drcomp kernel: [ 9720.800967] [] __kmalloc_node_track_caller+0x9b/0x1a0
Oct 9 04:50:19 drcomp kernel: [ 9720.800976] [] ? sock_alloc_send_pskb+0xf2/0x260
Oct 9 04:50:19 drcomp kernel: [ 9720.800985] [] __alloc_skb+0x83/0x170
Oct 9 04:50:19 drcomp kernel: [ 9720.800994] [] sock_alloc_send_pskb+0xf2/0x260
Oct 9 04:50:19 drcomp kernel: [ 9720.801002] [] sock_alloc_send_skb+0x15/0x20
Oct 9 04:50:19 drcomp kernel: [ 9720.801013] [] unix_stream_sendmsg+0x330/0x470
Oct 9 04:50:19 drcomp kernel: [ 9720.801022] [] sock_sendmsg+0xf3/0x130
Oct 9 04:50:19 drcomp kernel: [ 9720.801031] [] ? release_pages+0x284/0x2b0
Oct 9 04:50:19 drcomp kernel: [ 9720.801039] [] ? verify_iovec+0x82/0xf0
Oct 9 04:50:19 drcomp kernel: [ 9720.801046] [] sys_sendmsg+0x266/0x3b0
Oct 9 04:50:19 drcomp kernel: [ 9720.801055] [] ? perf_event_task_sched_out+0x2e/0xa0
Oct 9 04:50:19 drcomp kernel: [ 9720.801063] [] ? cpumask_any_but+0x2d/0x40
Oct 9 04:50:19 drcomp kernel: [ 9720.801073] [] ? read_tsc+0x9/0x20
Oct 9 04:50:19 drcomp kernel: [ 9720.801081] [] ? ktime_get_ts+0xb1/0xf0
Oct 9 04:50:19 drcomp kernel: [ 9720.801089] [] ? sys_recvmsg+0x75/0x90
Oct 9 04:50:19 drcomp kernel: [ 9720.801097] [] system_call_fastpath+0x16/0x1b
Oct 9 04:50:19 drcomp kernel: [ 9720.801106] INFO: task ktorrent:2395 blocked for more than 120 seconds.
Oct 9 04:50:19 drcomp kernel: [ 9720.801110] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 9 04:50:19 drcomp kernel: [ 9720.801115] ktorrent D 0000000000000000 0 2395 1 0x00000000
Oct 9 04:50:19 drcomp kernel: [ 9720.801123] ffff8800ad8b5b88 0000000000000082 ffff8800ad8b5fd8 ffff8800ad8b4000
Oct 9 04:50:19 drcomp kernel: [ 9720.801132] 0000000000013d00 ffff8800af605f38 ffff8800ad8b5fd8 0000000000013d00
Oct 9 04:50:19 drcomp kernel: [ 9720.801140] ffff880082552dc0 ffff8800af605b80 ffff8800ad8b5ba8 ffff880107267b30
Oct 9 04:50:19 drcomp kernel: [ 9720.801148] Call Trace:
Oct 9 04:50:19 drcomp kernel: [ 9720.801168] [] cv_wait_common+0x77/0xd0 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.801176] [] ? autoremove_wake_function+0x0/0x40
Oct 9 04:50:19 drcomp kernel: [ 9720.801195] [] __cv_wait+0x13/0x20 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.801263] [] txg_wait_open+0x73/0xb0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.801320] [] dmu_tx_wait+0xed/0xf0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.801385] [] zfs_write+0x38e/0xca0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.801393] [] ? _raw_spin_lock+0xe/0x20
Oct 9 04:50:19 drcomp kernel: [ 9720.801400] [] ? dput+0x12e/0x1d0
Oct 9 04:50:19 drcomp kernel: [ 9720.801408] [] ? path_put+0x22/0x30
Oct 9 04:50:19 drcomp kernel: [ 9720.801416] [] ? putname+0x35/0x50
Oct 9 04:50:19 drcomp kernel: [ 9720.801480] [] zpl_write_common+0x52/0x80 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.801544] [] zpl_write+0x68/0xa0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.801552] [] vfs_write+0xc6/0x180
Oct 9 04:50:19 drcomp kernel: [ 9720.801559] [] sys_write+0x51/0x90
Oct 9 04:50:19 drcomp kernel: [ 9720.801566] [] system_call_fastpath+0x16/0x1b
Oct 9 04:50:19 drcomp kernel: [ 9720.801574] INFO: task ktorrent:2453 blocked for more than 120 seconds.
Oct 9 04:50:19 drcomp kernel: [ 9720.801580] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 9 04:50:19 drcomp kernel: [ 9720.801587] ktorrent D 0000000000000000 0 2453 1 0x00000000
Oct 9 04:50:19 drcomp kernel: [ 9720.801595] ffff88008142b428 0000000000000082 ffff88008142bfd8 ffff88008142a000
Oct 9 04:50:19 drcomp kernel: [ 9720.801603] 0000000000013d00 ffff880082553178 ffff88008142bfd8 0000000000013d00
Oct 9 04:50:19 drcomp kernel: [ 9720.801612] ffffffff81a0b020 ffff880082552dc0 ffff88008142b448 ffff880107267b30
Oct 9 04:50:19 drcomp kernel: [ 9720.801620] Call Trace:
Oct 9 04:50:19 drcomp kernel: [ 9720.801639] [] cv_wait_common+0x77/0xd0 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.801648] [] ? autoremove_wake_function+0x0/0x40
Oct 9 04:50:19 drcomp kernel: [ 9720.801667] [] __cv_wait+0x13/0x20 [spl]
Oct 9 04:50:19 drcomp kernel: [ 9720.801735] [] txg_wait_open+0x73/0xb0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.801791] [] dmu_tx_wait+0xed/0xf0 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.801857] [] zfs_putpage+0x25b/0x270 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.801865] [] ? rmap_walk+0x107/0x250
Oct 9 04:50:19 drcomp kernel: [ 9720.801929] [] zpl_putpage+0x2f/0x40 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.801992] [] zpl_writepage+0x12/0x20 [zfs]
Oct 9 04:50:19 drcomp kernel: [ 9720.802001] [] fallback_migrate_page+0xc7/0x130
Oct 9 04:50:19 drcomp kernel: [ 9720.802010] [] move_to_new_page+0x103/0x120
Oct 9 04:50:19 drcomp kernel: [ 9720.802017] [] unmap_and_move+0x2f6/0x350
Oct 9 04:50:19 drcomp kernel: [ 9720.802026] [] ? compaction_alloc+0x0/0x70
Oct 9 04:50:19 drcomp kernel: [ 9720.802034] [] migrate_pages+0xa2/0x150
Oct 9 04:50:19 drcomp kernel: [ 9720.802043] [] compact_zone+0xa4/0x220
Oct 9 04:50:19 drcomp kernel: [ 9720.802051] [] compact_zone_order+0xac/0xf0
Oct 9 04:50:19 drcomp kernel: [ 9720.802058] [] ? try_to_wake_up+0x244/0x3e0
Oct 9 04:50:19 drcomp kernel: [ 9720.802068] [] try_to_compact_pages+0xcd/0x100
Oct 9 04:50:19 drcomp kernel: [ 9720.802076] [] __alloc_pages_direct_compact+0xcf/0x190
Oct 9 04:50:19 drcomp kernel: [ 9720.802084] [] __alloc_pages_nodemask+0x43d/0x840
Oct 9 04:50:19 drcomp kernel: [ 9720.802094] [] alloc_pages_current+0xa5/0x110
Oct 9 04:50:19 drcomp kernel: [ 9720.802101] [] new_slab+0x1f5/0x290
Oct 9 04:50:19 drcomp kernel: [ 9720.802109] [] __slab_alloc+0x1b2/0x390
Oct 9 04:50:19 drcomp kernel: [ 9720.802120] [] ? sk_stream_alloc_skb+0x41/0x130
Oct 9 04:50:19 drcomp kernel: [ 9720.802128] [] __kmalloc_node_track_caller+0x9b/0x1a0
Oct 9 04:50:19 drcomp kernel: [ 9720.802137] [] ? sk_stream_alloc_skb+0x41/0x130
Oct 9 04:50:19 drcomp kernel: [ 9720.802146] [] __alloc_skb+0x83/0x170
Oct 9 04:50:19 drcomp kernel: [ 9720.802155] [] sk_stream_alloc_skb+0x41/0x130
Oct 9 04:50:19 drcomp kernel: [ 9720.802163] [] tcp_sendmsg+0x3dc/0xc80
Oct 9 04:50:19 drcomp kernel: [ 9720.802171] [] ? skb_dequeue+0x67/0x90
Oct 9 04:50:19 drcomp kernel: [ 9720.802181] [] inet_sendmsg+0x64/0xb0
Oct 9 04:50:19 drcomp kernel: [ 9720.802192] [] ? apparmor_socket_sendmsg+0x17/0x20
Oct 9 04:50:19 drcomp kernel: [ 9720.802200] [] sock_sendmsg+0xf3/0x130
Oct 9 04:50:19 drcomp kernel: [ 9720.802208] [] ? sock_aio_read+0x164/0x170
Oct 9 04:50:19 drcomp kernel: [ 9720.802216] [] ? futex_wait+0x204/0x300
Oct 9 04:50:19 drcomp kernel: [ 9720.802224] [] ? do_sync_read+0xd2/0x110
Oct 9 04:50:19 drcomp kernel: [ 9720.802231] [] ? sockfd_lookup_light+0x29/0x90
Oct 9 04:50:19 drcomp kernel: [ 9720.802239] [

@randomei
Copy link

After upgrade to kubuntu 11.10 with kernel 3.0.x and zfs 0.6.0.34 I don't have this problem any more...

@behlendorf
Copy link
Contributor

This is believed to be resolved in the latest code. Please reopen if you observe the failure again.

kernelOfTruth pushed a commit to kernelOfTruth/zfs that referenced this issue Mar 1, 2015
The core motivation behind these changes is to minimize the
memory management differences between ZFS on Linux and other
platforms.  This simplifies the process of porting changes to
Linux from other platforms.  This is good for code quality
and is expected to reduce the number of defects accidentally
introduced due to porting.

The key reason this is now possible is due to the addition of
Linux features such as the thread-specific PF_FSTRANS bit which
was introduced for XFS.

This patch stack also performs some refactoring and cleanup
designed to make the code more maintainable and understandable.
Finally, in the context of making and testing these changes
several bugs were identified and resolved resulting in a
more robust implementation.

Signed-off-by: Brian Behlendorf <[email protected]>
Signed-off-by: Richard Yao <[email protected]>
Signed-off-by: Tim Chase <[email protected]>
Closes openzfs#414
sdimitro pushed a commit to sdimitro/zfs that referenced this issue May 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants