'BUG: scheduling while atomic' #688
Comments
|
I have removed #669 from my server and the stability issues persist. I am still looking into the cause. |
|
I do not know what is wrong, but I seem to be able to trigger deadlocks whenever I stop my virtual machines. I am not quite sure why. I was able to capture this before things went south this time: [ 331.670385] stack segment: 0000 [#1] SMP |
|
I can no longer reproduce this issue. |
…class (openzfs#688) Instead of overriding the normal metaslab class with the object store logic and ops methods, create a separate metaslab class for the object store. Besides making the code a bit simpler this also lays the groundwork for having hybrid pools which is a requirement for the DOSE In-Place Migration project.
I encountered this on my server while using CONFIG_PREEMPT_VOLUNTARY=y:
[ 788.283267] stack segment: 0000 [#1] SMP
[ 788.283287] CPU 3
[ 788.283293] Modules linked in: bridge ipv6 stp llc softdog vhost_net macvtap macvlan tun powernow_k8 mperf microcode kvm_amd kvm e1000e i2c_piix4 r8168(O) i2c_core k10temp rtc_cmos hwmon xhci_hcd sg evdev zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ohci_hcd ehci_hcd usbcore usb_common firewire_ohci firewire_core crc_itu_t
[ 788.283437]
[ 788.283446] Pid: 1823, comm: txg_sync Tainted: P O 3.3.1 #3 System manufacturer System Product Name/M4A89TD PRO USB3
[ 788.283474] RIP: 0010:[] [] __kmalloc+0xa6/0x190
[ 788.283499] RSP: 0018:ffff8803d97599c0 EFLAGS: 00010286
[ 788.283510] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000049780
[ 788.283523] RDX: 000000000004977f RSI: 00000000000080d0 RDI: 0000000000013840
[ 788.283536] RBP: ff8803d7c9980000 R08: 0000000000000000 R09: 0000000000000000
[ 788.283548] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8803ee802900
[ 788.283561] R13: 00000000000080d0 R14: ffffffffa0074cdb R15: 00000000000080d0
[ 788.283575] FS: 00007feba58a1700(0000) GS:ffff8803ffcc0000(0000) knlGS:0000000000000000
[ 788.283588] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 788.283633] CR2: 00007feba4c7bff2 CR3: 0000000001605000 CR4: 00000000000006e0
[ 788.283677] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 788.283722] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 788.283767] Process txg_sync (pid: 1823, threadinfo ffff8803d9758000, task ffff8803e8802170)
[ 788.283843] Stack:
[ 788.283881] 0000000000000000 0000000000000780 00000000000080d0 00000000000007e6
[ 788.283966] ffffffffa01a3fe2 ffffffffa0074cdb 2222222222222222 2222222222222222
[ 788.284050] 0000000000000003 ffff8803e212cd18 ffff8803e212cc28 0000000000000001
[ 788.284135] Call Trace:
[ 788.284186] [] ? kmem_alloc_debug+0x20b/0x360 [spl]
[ 788.284194] [] ? dbuf_hold_impl+0x54/0x340 [zfs]
[ 788.284194] [] ? dbuf_hold+0x18/0x30 [zfs]
[ 788.284194] [] ? down_read+0x9/0x20
[ 788.284194] [] ? dnode_hold_impl+0xbb/0x5b0 [zfs]
[ 788.284194] [] ? dnode_setdirty+0x13f/0x1c0 [zfs]
[ 788.284194] [] ? dmu_buf_hold+0x4f/0x180 [zfs]
[ 788.284194] [] ? bpobj_enqueue_subobj+0x4ea/0xa40 [zfs]
[ 788.284194] [] ? spa_sync_allpools+0x480/0x4e0 [zfs]
[ 788.284194] [] ? sched_clock_local+0x15/0x80
[ 788.284194] [] ? zio_add_child+0x4db/0x5c0 [zfs]
[ 788.284194] [] ? zio_null+0x5b/0x60 [zfs]
[ 788.284194] [] ? spa_sync+0x250/0x10f0 [zfs]
[ 788.284194] [] ? check_preempt_curr+0x6d/0x90
[ 788.284194] [] ? ttwu_do_wakeup+0x11/0x90
[ 788.284194] [] ? try_to_wake_up+0xd3/0x290
[ 788.284194] [] ? txg_sync_start+0x4a9/0xa50 [zfs]
[ 788.284194] [] ? txg_sync_start+0x1c0/0xa50 [zfs]
[ 788.284194] [] ? __thread_create+0x320/0x3b0 [spl]
[ 788.284194] [] ? __thread_create+0x395/0x3b0 [spl]
[ 788.284194] [] ? kthread+0x9e/0xb0
[ 788.284194] [] ? kernel_thread_helper+0x4/0x10
[ 788.284194] [] ? flush_kthread_worker+0xc0/0xc0
[ 788.284194] [] ? gs_change+0xb/0xb
[ 788.284194] Code: 00 49 8b 0c 24 65 48 03 0c 25 58 cb 00 00 48 8b 51 08 48 8b 29 48 85 ed 0f 84 cb 00 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <48> 8b 5c 05 00 48 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 c2 48
[ 788.284194] RIP [] __kmalloc+0xa6/0x190
[ 788.284194] RSP
[ 788.286018] ---[ end trace f252a438dabde006 ]---
[ 788.286077] note: txg_sync[1823] exited with preempt_count 1
[ 794.281150] stack segment: 0000 [#2] SMP
[ 794.281241] CPU 3
[ 794.281261] Modules linked in: bridge ipv6 stp llc softdog vhost_net macvtap macvlan tun powernow_k8 mperf microcode kvm_amd kvm e1000e i2c_piix4 r8168(O) i2c_core k10temp rtc_cmos hwmon xhci_hcd sg evdev zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ohci_hcd ehci_hcd usbcore usb_common firewire_ohci firewire_core crc_itu_t
[ 794.282097]
[ 794.282097] Pid: 8317, comm: vim Tainted: P D O 3.3.1 #3 System manufacturer System Product Name/M4A89TD PRO USB3
[ 794.282097] RIP: 0010:[] [] __kmalloc+0xa6/0x190
[ 794.282097] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000049780
[ 794.282097] RDX: 000000000004977f RSI: 00000000000080d0 RDI: 0000000000013840
[ 794.282097] RBP: ff8803d7c9980000 R08: 0000000000000000 R09: 0000000000000000
[ 794.282097] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8803ee802900
[ 794.282097] R13: 00000000000080d0 R14: ffffffffa0074cdb R15: 00000000000080d0
[ 794.282097] FS: 00007fe1686d7700(0000) GS:ffff8803ffcc0000(0000) knlGS:0000000000000000
[ 794.282097] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 794.282097] CR2: 00000000fe7e0ecc CR3: 00000002f0c66000 CR4: 00000000000006e0
[ 794.282097] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 794.282097] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 794.282097] Process vim (pid: 8317, threadinfo ffff8803d9758000, task ffff8803e8e93580)
[ 794.282097] Stack:
[ 794.282097] 0000000000000000 0000000000000780 00000000000080d0 00000000000007e6
[ 794.282097] ffffffffa01a3fe2 ffffffffa0074cdb ffff8802f0c728e0 ffffffff81312ac3
[ 794.282097] ffff880300000000 ffff8803d9759878 2222222222222222 ffffffffa016762c
[ 794.282097] Call Trace:
[ 794.282097] [] ? kmem_alloc_debug+0x20b/0x360 [spl]
[ 794.282097] [] ? __mutex_lock_slowpath+0x1e3/0x2a0
[ 794.282097] [] ? zap_match+0x3c/0x180 [zfs]
[ 794.282097] [] ? __kmalloc+0x7c/0x190
[ 794.282097] [] ? dbuf_hold_impl+0x54/0x340 [zfs]
[ 794.282097] [] ? dbuf_hold+0x18/0x30 [zfs]
[ 794.282097] [] ? down_read+0x9/0x20
[ 794.282097] [] ? dnode_hold_impl+0xbb/0x5b0 [zfs]
[ 794.282097] [] ? __mutex_lock_slowpath+0x1e3/0x2a0
[ 794.282097] [] ? dmu_bonus_hold+0x31/0x980 [zfs]
[ 794.282097] [] ? zfs_zget+0x113/0x2c0 [zfs]
[ 794.282097] [] ? zap_lookup_norm+0xe2/0x1b0 [zfs]
[ 794.282097] [] ? zfs_dirent_lock+0x49e/0x5b0 [zfs]
[ 794.282097] [] ? zfs_acl_free+0x545/0x800 [zfs]
[ 794.282097] [] ? zfs_dirlook+0x92/0x2d0 [zfs]
[ 794.282097] [] ? zfs_zaccess+0xa4/0x250 [zfs]
[ 794.282097] [] ? zfs_lookup+0x2f4/0x380 [zfs]
[ 794.282097] [] ? zpl_fallocate_common+0x262/0x700 [zfs]
[ 794.282097] [] ? d_alloc_and_lookup+0x33/0x70
[ 794.282097] [] ? do_lookup+0x293/0x390
[ 794.282097] [] ? path_lookupat+0x12d/0x6e0
[ 794.282097] [] ? do_lookup+0x189/0x390
[ 794.282097] [] ? do_path_lookup+0x2b/0x90
[ 794.282097] [] ? user_path_at_empty+0x5e/0xc0
[ 794.282097] [] ? do_path_lookup+0x2b/0x90
[ 794.282097] [] ? kmem_cache_free+0x14/0x90
[ 798.362148] RSP: 0018:ffff880165c7fb78 EFLAGS: 00010286
[ 798.362148] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000049780
[ 798.362148] RDX: 000000000004977f RSI: 00000000000080d0 RDI: 0000000000013840
[ 798.362148] RBP: ff8803d7c9980000 R08: 0000000000000000 R09: 0000000000000000
[ 798.362148] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8803ee802900
[ 798.362148] R13: 00000000000080d0 R14: ffffffffa0074cdb R15: 00000000000080d0
[ 798.362148] FS: 00007fa921d37700(0000) GS:ffff8803ffcc0000(0000) knlGS:0000000000000000
[ 798.362148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 798.362148] CR2: 000000000047f8c0 CR3: 00000002f3edb000 CR4: 00000000000006e0
[ 798.362148] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 798.362148] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 798.362148] Process bash (pid: 8330, threadinfo ffff880165c7e000, task ffff8803e8e95040)
[ 798.362148] Stack:
[ 798.362148] 0000000000000000 0000000000000780 00000000000080d0 00000000000007e6
[ 798.362148] ffffffffa01a3fe2 ffffffffa0074cdb 0000000000000001 ffffffff8163d2c0
[ 798.362148] ffff880165c7ffd8 0000000000000002 0000000000000003 ffff8803e8e95040
[ 798.362148] Call Trace:
[ 798.362148] [] ? kmem_alloc_debug+0x20b/0x360 [spl]
[ 798.362148] [] ? dbuf_hold_impl+0x54/0x340 [zfs]
[ 798.362148] [] ? dbuf_hold+0x18/0x30 [zfs]
[ 798.362148] [] ? down_read+0x9/0x20
[ 798.362148] [] ? dnode_hold_impl+0xbb/0x5b0 [zfs]
[ 798.362148] [] ? __kmalloc+0x7c/0x190
[ 798.362148] [] ? kmem_alloc_debug+0x283/0x360 [spl]
[ 798.362148] [] ? dmu_bonus_hold+0x915/0x980 [zfs]
[ 798.362148] [] ? avl_add+0x33/0x50 [zavl]
[ 798.362148] [] ? dmu_read_uio+0x39/0xd0 [zfs]
[ 798.362148] [] ? zfs_read+0x3b2/0x490 [zfs]
[ 798.362148] [] ? zpl_read_common+0x51/0x110 [zfs]
[ 798.362148] [] ? zpl_read_common+0xd4/0x110 [zfs]
[ 798.362148] [] ? vfs_read+0xc3/0x180
[ 798.362148] [] ? sys_read+0x4e/0x90
[ 798.362148] [] ? page_fault+0x1f/0x30
[ 798.362148] [] ? system_call_fastpath+0x16/0x1b
[ 798.362148] Code: 00 49 8b 0c 24 65 48 03 0c 25 58 cb 00 00 48 8b 51 08 48 8b 29 48 85 ed 0f 84 cb 00 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <48> 8b 5c 05 00 48 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 c2 48
[ 798.362148] RIP [] __kmalloc+0xa6/0x190
[ 798.362148] RSP
[ 798.366659] ---[ end trace f252a438dabde008 ]---
[ 798.366718] BUG: scheduling while atomic: bash/8330/0x10000001
[ 798.366779] Modules linked in: bridge ipv6 stp llc softdog vhost_net macvtap macvlan tun powernow_k8 mperf microcode kvm_amd kvm e1000e i2c_piix4 r8168(O) i2c_core k10temp rtc_cmos hwmon xhci_hcd sg evdev zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ohci_hcd ehci_hcd usbcore usb_common firewire_ohci firewire_core crc_itu_t
[ 798.367683] Pid: 8330, comm: bash Tainted: P D O 3.3.1 #3
[ 798.367687] Call Trace:
[ 798.367690] <#SS> [] ? __schedule+0x4ea/0x640
[ 798.367707] [] ? kmem_alloc_debug+0x20b/0x360 [spl]
[ 798.367718] [] ? __cond_resched+0x1d/0x30
[ 798.367724] [] ? _cond_resched+0x2f/0x40
[ 798.367731] [] ? down_read+0x9/0x20
[ 798.367737] [] ? exit_signals+0x1b/0x130
[ 798.367745] [] ? do_exit+0xac/0x860
[ 798.367754] [] ? kmem_alloc_debug+0x20b/0x360 [spl]
[ 798.367762] [] ? printk+0x40/0x47
[ 798.367768] [] ? kmsg_dump+0x52/0x110
[ 798.367778] [] ? kmem_alloc_debug+0x20b/0x360 [spl]
[ 798.367785] [] ? oops_end+0x9b/0xe0
[ 798.367791] [] ? do_stack_segment+0x6e/0xa0
[ 798.367797] [] ? stack_segment+0x1f/0x30
[ 798.367806] [] ? kmem_alloc_debug+0x20b/0x360 [spl]
[ 798.367812] [] ? __kmalloc+0xa6/0x190
[ 798.367816] <> [] ? kmem_alloc_debug+0x20b/0x360 [spl]
[ 798.367845] [] ? dbuf_hold_impl+0x54/0x340 [zfs]
[ 798.367864] [] ? dbuf_hold+0x18/0x30 [zfs]
[ 798.367872] [] ? down_read+0x9/0x20
[ 798.367896] [] ? dnode_hold_impl+0xbb/0x5b0 [zfs]
[ 798.367902] [] ? __kmalloc+0x7c/0x190
[ 798.367912] [] ? kmem_alloc_debug+0x283/0x360 [spl]
[ 798.367934] [] ? dmu_bonus_hold+0x915/0x980 [zfs]
[ 798.367945] [] ? avl_add+0x33/0x50 [zavl]
[ 798.367967] [] ? dmu_read_uio+0x39/0xd0 [zfs]
[ 798.367988] [] ? zfs_read+0x3b2/0x490 [zfs]
[ 798.368005] [] ? zpl_read_common+0x51/0x110 [zfs]
[ 798.368033] [] ? zpl_read_common+0xd4/0x110 [zfs]
[ 798.368039] [] ? vfs_read+0xc3/0x180
[ 798.368045] [] ? sys_read+0x4e/0x90
[ 798.368050] [] ? page_fault+0x1f/0x30
[ 798.368057] [] ? system_call_fastpath+0x16/0x1b
[ 798.368075] note: bash[8330] exited with preempt_count 1
My screen session locked up shortly after getting that from dmesg.
Here is my pool configuration:
zdb
bpool:
version: 28
name: 'bpool'
state: 0
txg: 381988
pool_guid: 10197376284787005672
hostname: 'vserver'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 10197376284787005672
children[0]:
type: 'mirror'
id: 0
guid: 17431331604726921311
metaslab_array: 31
metaslab_shift: 20
ashift: 12
asize: 129499136
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 15907417882988350642
path: '/dev/sdf1'
whole_disk: 0
DTL: 111
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 15582814262976156480
path: '/dev/sde1'
whole_disk: 0
DTL: 110
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 9323895700972417586
path: '/dev/sdd1'
whole_disk: 0
DTL: 109
create_txg: 4
children[3]:
type: 'disk'
id: 3
guid: 7305042815279891748
path: '/dev/sdb1'
whole_disk: 0
DTL: 108
create_txg: 4
children[4]:
type: 'disk'
id: 4
guid: 1149656489871005278
path: '/dev/sdc1'
whole_disk: 0
DTL: 107
create_txg: 4
children[5]:
type: 'disk'
id: 5
guid: 8053081386639308922
path: '/dev/sda1'
whole_disk: 0
DTL: 106
create_txg: 4
rpool:
version: 28
name: 'rpool'
state: 0
txg: 1768
pool_guid: 854433711871625000
hostname: '(none)'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 854433711871625000
children[0]:
type: 'raidz'
id: 0
guid: 11086353909113716591
nparity: 2
metaslab_array: 31
metaslab_shift: 36
ashift: 12
asize: 12001553154048
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 4403236062912303989
path: '/dev/sdf2'
whole_disk: 0
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 13773692824674393969
path: '/dev/sde2'
whole_disk: 0
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 4552351019414441823
path: '/dev/sdd2'
whole_disk: 0
create_txg: 4
children[3]:
type: 'disk'
id: 3
guid: 5391404027882106468
path: '/dev/sdb2'
whole_disk: 0
create_txg: 4
children[4]:
type: 'disk'
id: 4
guid: 6762155564963459645
path: '/dev/sdc2'
whole_disk: 0
create_txg: 4
children[5]:
type: 'disk'
id: 5
guid: 2858552060245597587
path: '/dev/sda2'
whole_disk: 0
create_txg: 4
This is with patches from pull requests #618, #651, #660, #669 and #674 applied to my system. No swap was being used, but I had initiated the shutdown process on all KVM virtual machines running on the system.
I tried swithing to PREEMPT_NONE=y, but that did not seem to address the issues. I had done
zfs set sync=disabled rpool/KVM/win7before this started, so I suspect it might be related. I am still looking into this.The text was updated successfully, but these errors were encountered: