Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RIP [<ffffffff8150fca7>] __mutex_lock_slowpath+0x47/0x1c0 #3940

Closed
Gintaras1 opened this issue Oct 21, 2015 · 4 comments
Closed

RIP [<ffffffff8150fca7>] __mutex_lock_slowpath+0x47/0x1c0 #3940

Gintaras1 opened this issue Oct 21, 2015 · 4 comments

Comments

@Gintaras1
Copy link

Debian 8.2, zfs 0.6.5.2-2. raidz1-0 3x SSD

dmesg:
[256678.875157] general protection fault: 0000 [#1] SMP
[256678.875173] Modules linked in: nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc 8021q garp mrp bridge stp llc bonding snd_hda_codec_hdmi zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) x86_pkg_temp_thermal intel_powerclamp intel_rapl coretemp kvm_intel kvm snd_hda_codec_realtek iTCO_wdt crc32_pclmul eeepc_wmi iTCO_vendor_support asus_wmi snd_hda_codec_generic sparse_keymap rfkill evdev ghash_clmulni_intel nouveau aesni_intel aes_x86_64 lrw mxm_wmi gf128mul video glue_helper ablk_helper cryptd ttm psmouse sb_edac drm_kms_helper snd_hda_intel pcspkr snd_hda_controller serio_raw drm edac_core snd_hda_codec snd_hwdep i2c_algo_bit snd_pcm snd_timer snd lpc_ich soundcore i2c_i801 mei_me i2c_core shpchp mfd_core mei wmi tpm_infineon processor tpm_tis thermal_sys tpm button fuse
[256678.875304] autofs4 ext4 crc16 mbcache jbd2 dm_mod raid1 md_mod sg sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci ehci_pci xhci_hcd firewire_ohci ehci_hcd libata e1000e scsi_mod usbcore ptp firewire_core crc_itu_t usb_common pps_core
[256678.875354] CPU: 5 PID: 20122 Comm: systemd-udevd Tainted: P O 3.16.0-4-amd64 #1 Debian 3.16.7-ckt11-1+deb8u5
[256678.875361] Hardware name: System manufacturer System Product Name/P9X79, BIOS 4608 12/24/2013
[256678.875367] task: ffff8810388a4050 ti: ffff880f530ac000 task.ti: ffff880f530ac000
[256678.875372] RIP: 0010:[] [] __mutex_lock_slowpath+0x47/0x1c0
[256678.875383] RSP: 0018:ffff880f530af800 EFLAGS: 00010206
[256678.875387] RAX: 15e6d2c84df29899 RBX: ffff880ffbdd71d8 RCX: 00000000000000e0
[256678.875392] RDX: 000000000000002f RSI: ffff881038438000 RDI: ffff880ffbdd71d8
[256678.875398] RBP: ffff880f530af850 R08: 0000000000000008 R09: ffff880ffadbc848
[256678.875402] R10: ffff880e864b31c8 R11: 0000000000001000 R12: ffff880ffbdd7210
[256678.875407] R13: ffff8810388a4050 R14: ffff880ffbdd71d8 R15: 0000000000000000
[256678.875413] FS: 00007f0d0505c880(0000) GS:ffff88103fd40000(0000) knlGS:0000000000000000
[256678.875418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[256678.875422] CR2: 00007f3bbd890000 CR3: 0000000f290b8000 CR4: 00000000000407e0
[256678.875427] Stack:
[256678.875430] ffff880febe04008 0000000000004210 0000000000000000 00000000000000e0
[256678.875441] 0000000000000000 ffff880ffbdd71d8 ffff880ffbdd7210 0000000000001000
[256678.875450] ffff880ffbdd71d8 0000000000000000 ffff880ffbdd7130 ffffffff8150fe3b
[256678.875458] Call Trace:
[256678.875463] [] ? mutex_lock+0x1b/0x2a
[256678.875482] [] ? zfs_range_lock+0x88/0x600 [zfs]
[256678.875496] [] ? zvol_request+0x276/0x5f0 [zfs]
[256678.875503] [] ? generic_make_request_checks+0x264/0x390
[256678.875508] [] ? generic_make_request+0xb0/0x100
[256678.875514] [] ? submit_bio+0x71/0x150
[256678.875521] [] ? bio_alloc_bioset+0x198/0x290
[256678.875527] [] ? _submit_bh+0x12a/0x1f0
[256678.875533] [] ? block_read_full_page+0x1cf/0x2c0
[256678.875538] [] ? I_BDEV+0x10/0x10
[256678.875544] [] ? __add_to_page_cache_locked+0x13b/0x1f0
[256678.875550] [] ? blkdev_write_begin+0x20/0x20
[256678.875556] [] ? do_read_cache_page+0x7e/0x1c0
[256678.875562] [] ? string.isra.7+0x36/0xe0
[256678.875568] [] ? read_dev_sector+0x28/0xa0
[256678.875573] [] ? check_partition+0x210/0x210
[256678.875578] [] ? adfspart_check_ICS+0x71/0x270
[256678.875856] [] ? snprintf+0x39/0x40
[256678.876479] [] ? check_partition+0x210/0x210
[256678.877101] [] ? check_partition+0xff/0x210
[256678.877713] [] ? rescan_partitions+0xad/0x280
[256678.878310] [] ? bdi_lock_two+0x1e/0x50
[256678.878890] [] ? __blkdev_get+0x35b/0x480
[256678.879454] [] ? blkdev_get_by_dev+0x40/0x40
[256678.880001] [] ? blkdev_get+0x1b6/0x310
[256678.880531] [] ? blkdev_get_by_dev+0x40/0x40
[256678.881043] [] ? do_dentry_open+0x1f2/0x330
[256678.881529] [] ? finish_open+0x2d/0x40
[256678.882002] [] ? do_last+0x654/0x11b0
[256678.882451] [] ? path_openat+0xbb/0x680
[256678.882892] [] ? free_pages_and_swap_cache+0x95/0xb0
[256678.883317] [] ? do_filp_open+0x3a/0x90
[256678.883732] [] ? __alloc_fd+0x7c/0x120
[256678.884140] [] ? do_sys_open+0x129/0x220
[256678.884535] [] ? system_call_fast_compare_end+0x10/0x15
[256678.884928] Code: 89 fb 48 83 e4 f0 48 83 ec 20 65 48 8b 04 25 88 b8 00 00 48 8b 80 38 c0 ff ff a8 08 75 28 48 8b 47 18 48 85 c0 0f 84 5e 01 00 00 <8b> 40 28 85 c0 74 14 4c 8d 7b 20 4c 89 ff e8 c6 c5 b9 ff 84 c0
[256678.885838] RIP [] __mutex_lock_slowpath+0x47/0x1c0
[256678.886246] RSP
[256678.888792] ---[ end trace 3fbb330e1ffc2fc2 ]---

@behlendorf
Copy link
Contributor

What version of ZFS and is this still an issue?

@Gintaras1
Copy link
Author

After ~36 days uptime crash again (2015-12-18)

modinfo zfs |grep version

version: 0.6.5.2-2

uname -a

Linux ohv1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u5 (2015-10-09) x86_64 GNU/Linux

Process D state:
26822 ? Ds 0:00 zfs create -s -V 5368840192 home/zfs-2004/disk.0

zpool list

NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
home 2.17T 953G 1.24T - 48% 42% 1.39x ONLINE -

dedup off
compression lz4

dmesg:
[3118890.119403] zd3584: p1 p2 p3
[3118917.276547] BUG: unable to handle kernel NULL pointer dereference at 0000000000000050
[3118917.276710] IP: [] zvol_revalidate_disk+0x28/0x30 [zfs]
[3118917.276864] PGD 0
[3118917.277024] Oops: 0002 [#1] SMP
[3118917.277206] Modules linked in: nfnetlink_queue nfnetlink_log bluetooth 6lowpan_iphc dm_snapshot dm_bufio dm_zero vhost_net vhost macvtap macvlan nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ip6table_filter ip6_tables ebt_arp ebt_ip ebtable_nat ebtables tun xt_physdev ip_set nfnetlink iptable_filter ip_tables x_tables rpcsec_gss_krb5 nfsv4 dns_resolver nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc 8021q garp mrp bridge stp llc bonding snd_hda_codec_hdmi zfs(PO) zunicode(PO) x86_pkg_temp_thermal intel_powerclamp intel_rapl coretemp zcommon(PO) znvpair(PO) kvm_intel kvm spl(O) zavl(PO) crc32_pclmul ghash_clmulni_intel eeepc_wmi asus_wmi iTCO_wdt sparse_keymap iTCO_vendor_support rfkill snd_hda_codec_realtek evdev snd_hda_codec_generic nouveau aesni_intel aes_x86_64 lrw
[3118917.279528] gf128mul mxm_wmi glue_helper ablk_helper video cryptd psmouse ttm sb_edac pcspkr snd_hda_intel drm_kms_helper snd_hda_controller serio_raw drm snd_hda_codec edac_core snd_hwdep snd_pcm i2c_algo_bit snd_timer snd i2c_i801 soundcore mei_me i2c_core lpc_ich mei mfd_core shpchp wmi tpm_infineon tpm_tis processor button tpm thermal_sys autofs4 ext4 crc16 mbcache jbd2 dm_mod raid1 md_mod sg sd_mod crc_t10dif crct10dif_generic ahci crct10dif_pclmul libahci crct10dif_common crc32c_intel xhci_hcd ehci_pci e1000e ehci_hcd libata firewire_ohci ptp firewire_core crc_itu_t usbcore scsi_mod pps_core usb_common
[3118917.282550] CPU: 7 PID: 21825 Comm: systemd-udevd Tainted: P O 3.16.0-4-amd64 #1 Debian 3.16.7-ckt11-1+deb8u5
[3118917.283130] Hardware name: System manufacturer System Product Name/P9X79, BIOS 4608 12/24/2013
[3118917.283740] task: ffff880ffb76e210 ti: ffff880e0d40c000 task.ti: ffff880e0d40c000
[3118917.284335] RIP: 0010:[] [] zvol_revalidate_disk+0x28/0x30 [zfs]
[3118917.284939] RSP: 0018:ffff880e0d40fb90 EFLAGS: 00010246
[3118917.285556] RAX: 0000000000000000 RBX: ffff880122641b80 RCX: 000000000000f670
[3118917.286185] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff880fd2075c00
[3118917.286816] RBP: 0000000000000001 R08: ffff880e0d40c000 R09: 000000000000bb70
[3118917.287458] R10: 0000000000004ebd R11: 0000000000000000 R12: ffffffffa0e064e0
[3118917.288110] R13: 0000000000000001 R14: 0000000000000000 R15: ffff88037721b710
[3118917.288756] FS: 00007f2acc804880(0000) GS:ffff88103fdc0000(0000) knlGS:0000000000000000
[3118917.289407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3118917.290060] CR2: 0000000000000050 CR3: 000000013ff14000 CR4: 00000000000427e0
[3118917.290716] Stack:
[3118917.291380] ffffffff811dbfcb ffff8801c585b800 ffff880122641b80 0000000000000000
[3118917.292038] ffffffffa0dd0651 000000000e600de0 ffff880e0d40fc1c ffff880122641b98
[3118917.292711] ffff880122641b80 ffff880fd2075c00 ffff880122641b98 ffffffffa0e06580
[3118917.293389] Call Trace:
[3118917.294051] [] ? check_disk_change+0x5b/0x70
[3118917.294742] [] ? zvol_open+0xb1/0x2f0 [zfs]
[3118917.295388] [] ? __blkdev_get+0xcc/0x480
[3118917.296065] [] ? blkdev_get_by_dev+0x40/0x40
[3118917.296747] [] ? blkdev_get+0x1b6/0x310
[3118917.297429] [] ? blkdev_get_by_dev+0x40/0x40
[3118917.298095] [] ? do_dentry_open+0x1f2/0x330
[3118917.298760] [] ? finish_open+0x2d/0x40
[3118917.299426] [] ? do_last+0x654/0x11b0
[3118917.300096] [] ? path_openat+0xbb/0x680
[3118917.300758] [] ? free_pages_and_swap_cache+0x95/0xb0
[3118917.301428] [] ? do_filp_open+0x3a/0x90
[3118917.302102] [] ? __alloc_fd+0x7c/0x120
[3118917.302785] [] ? do_sys_open+0x129/0x220
[3118917.303454] [] ? system_call_fast_compare_end+0x10/0x15
[3118917.304121] Code: 00 00 00 66 66 66 66 90 48 8b 87 80 03 00 00 c7 80 20 01 00 00 00 00 00 00 48 8b 90 98 05 00 00 48 8b 80 00 01 00 00 48 c1 e8 09 <48> 89 42 50 31 c0 c3 90 66 66 66 66 90 48 8b 87 98 00 00 00 48
[3118917.305561] RIP [] zvol_revalidate_disk+0x28/0x30 [zfs]
[3118917.306239] RSP
[3118917.306897] CR2: 0000000000000050
[3118917.309754] ---[ end trace fa396bd9d5c3005f ]---
[3118923.313687] general protection fault: 0000 [#2] SMP
[3118923.313947] Modules linked in: nfnetlink_queue nfnetlink_log bluetooth 6lowpan_iphc dm_snapshot dm_bufio dm_zero vhost_net vhost macvtap macvlan nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ip6table_filter ip6_tables ebt_arp ebt_ip ebtable_nat ebtables tun xt_physdev ip_set nfnetlink iptable_filter ip_tables x_tables rpcsec_gss_krb5 nfsv4 dns_resolver nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc 8021q garp mrp bridge stp llc bonding snd_hda_codec_hdmi zfs(PO) zunicode(PO) x86_pkg_temp_thermal intel_powerclamp intel_rapl coretemp zcommon(PO) znvpair(PO) kvm_intel kvm spl(O) zavl(PO) crc32_pclmul ghash_clmulni_intel eeepc_wmi asus_wmi iTCO_wdt sparse_keymap iTCO_vendor_support rfkill snd_hda_codec_realtek evdev snd_hda_codec_generic nouveau aesni_intel aes_x86_64 lrw
[3118923.317480] gf128mul mxm_wmi glue_helper ablk_helper video cryptd psmouse ttm sb_edac pcspkr snd_hda_intel drm_kms_helper snd_hda_controller serio_raw drm snd_hda_codec edac_core snd_hwdep snd_pcm i2c_algo_bit snd_timer snd i2c_i801 soundcore mei_me i2c_core lpc_ich mei mfd_core shpchp wmi tpm_infineon tpm_tis processor button tpm thermal_sys autofs4 ext4 crc16 mbcache jbd2 dm_mod raid1 md_mod sg sd_mod crc_t10dif crct10dif_generic ahci crct10dif_pclmul libahci crct10dif_common crc32c_intel xhci_hcd ehci_pci e1000e ehci_hcd libata firewire_ohci ptp firewire_core crc_itu_t usbcore scsi_mod pps_core usb_common
[3118923.321272] CPU: 0 PID: 21839 Comm: systemd-udevd Tainted: P D O 3.16.0-4-amd64 #1 Debian 3.16.7-ckt11-1+deb8u5
[3118923.321937] Hardware name: System manufacturer System Product Name/P9X79, BIOS 4608 12/24/2013
[3118923.322605] task: ffff88058cfca3d0 ti: ffff880713270000 task.ti: ffff880713270000
[3118923.323278] RIP: 0010:[] [] skb_dequeue+0x40/0x70
[3118923.323960] RSP: 0018:ffff880713273e58 EFLAGS: 00010097
[3118923.324651] RAX: 0000000000000246 RBX: ffff880100000000 RCX: fff5fff7fff0ffef
[3118923.325357] RDX: 000a0009fffefff5 RSI: 0000000000000246 RDI: ffff8801c585b934
[3118923.326047] RBP: ffff8801c585b920 R08: 0000000000000000 R09: 0000000000000000
[3118923.326733] R10: ffff880dd048d410 R11: 0000000000000246 R12: ffff8801c585b934
[3118923.327408] R13: ffff881038fd57e0 R14: ffff880d70dc60d8 R15: ffff880fd380f2f0
[3118923.328088] FS: 00007f2acc804880(0000) GS:ffff88103fc00000(0000) knlGS:0000000000000000
[3118923.328774] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3118923.329455] CR2: 00007f2ace0da050 CR3: 0000000deed4d000 CR4: 00000000000427f0
[3118923.330136] Stack:
[3118923.330800] ffff8801c585b920 ffff880fd380f2c0 ffff8801c585b998 ffffffff8140d2b0
[3118923.331502] ffff8801c585b800 ffffffff81449d5b ffffea0000000000 0000000000000000
[3118923.332195] ffff880fd380f2c0 0000000000000000 ffff880fd380f2f0 ffffffff814032da
[3118923.332891] Call Trace:
[3118923.333580] [] ? skb_queue_purge+0x20/0x30
[3118923.334284] [] ? netlink_release+0xfb/0x320
[3118923.334990] [] ? sock_release+0x1a/0x90
[3118923.335695] [] ? sock_close+0xe/0x20
[3118923.336403] [] ? __fput+0xca/0x1d0
[3118923.337100] [] ? task_work_run+0x97/0xd0
[3118923.337798] [] ? do_notify_resume+0x69/0xa0
[3118923.338499] [] ? int_signal+0x12/0x17
[3118923.339196] Code: 50 10 00 48 8b 5d 00 48 39 eb 74 3f 48 85 db 74 21 83 6d 10 01 48 8b 0b 48 8b 53 08 48 c7 03 00 00 00 00 48 c7 43 08 00 00 00 00 <48> 89 51 08 48 89 0a 48 89 c6 4c 89 e7 e8 6e 4d 10 00 48 89 d8
[3118923.340680] RIP [] skb_dequeue+0x40/0x70
[3118923.341401] RSP
[3118923.342103] ---[ end trace fa396bd9d5c30060 ]---

@dweeezil
Copy link
Contributor

That appears to be the same issue as #3678.

@behlendorf
Copy link
Contributor

Closing as duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants