Skip to content
This repository has been archived by the owner on Feb 26, 2020. It is now read-only.

Kernel oops with splat on 2.6.35-rc3 #22

Closed
sehe opened this issue Jul 6, 2010 · 6 comments
Closed

Kernel oops with splat on 2.6.35-rc3 #22

sehe opened this issue Jul 6, 2010 · 6 comments

Comments

@sehe
Copy link

sehe commented Jul 6, 2010

It seems that splat may overdraw vmap/vmalloc and have an unchecked NULL reference:

[   48.505423] SPL: Loaded Solaris Porting Layer v0.5.0
[   48.521175] SPLAT: Loaded Solaris Porting LAyer Tests v0.5.0
[  557.027296] vmap allocation for size 4198400 failed: use vmalloc= to increase size.
[  557.054972] BUG: unable to handle kernel NULL pointer dereference at (null)
[  557.054979] IP: [] splat_kmem_test4+0x7f/0x1d0 [splat]
[  557.054985] *pdpt = 000000002e729001 *pde = 0000000000000000 
[  557.054989] Oops: 0000 [#1] SMP 
[  557.054991] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/local_cpus
[  557.054994] Modules linked in: splat spl crc32c binfmt_misc ppdev nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables kvm_intel kvm snd_hda_codec_analog autofs4 snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm ib_iser rdma_cm ib_cm iw_cm snd_seq_dummy snd_seq_oss ib_sa snd_seq_midi ib_mad snd_rawmidi ib_core ib_addr snd_seq_midi_event snd_seq snd_timer iscsi_tcp libiscsi_tcp fbcon tileblit font bitblit softcursor bridge snd_seq_device stp vga16fb libiscsi vgastate scsi_transport_iscsi snd nvidia(P) usblp asus_atk0110 lp usbhid soundcore ohci1394 snd_page_alloc parport intel_agp agpgart hid sky2 ieee1394 ahci pata_jmicron skge libahci
[  557.055044] 
[  557.055047] Pid: 4984, comm: splat Tainted: P            2.6.35-rc3splice+ #5 P5B-Deluxe/System Product Name
[  557.055049] EIP: 0060:[] EFLAGS: 00010246 CPU: 3
[  557.055052] EIP is at splat_kmem_test4+0x7f/0x1d0 [splat]
[  557.055054] EAX: 00000000 EBX: 00000000 ECX: edc87260 EDX: 00000009
[  557.055055] ESI: 00000000 EDI: 00400000 EBP: edcf9e98 ESP: edcf9e30
[  557.055057]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[  557.055059] Process splat (pid: 4984, ti=edcf8000 task=ed67e580 task.ti=edcf8000)
[  557.055061] Stack:
[  557.055062]  000000cd f87c1b48 00000014 f87c5885 00200000 0000000a 0000000a edcf9e70
[  557.055067] <0> 00000000 00000009 00400000 ec41fb00 00000009 ff03c000 fcd82000 fd184000
[  557.055072] <0> fd586000 fd988000 fdd8a000 fe18c000 fe58e000 fe990000 00000000 ec41fb00
[  557.055077] Call Trace:
[  557.055082]  [] ? splat_ioctl+0x3e9/0x640 [splat]
[  557.055087]  [] ? _raw_spin_lock_irqsave+0x2f/0x50
[  557.055091]  [] ? remove_wait_queue+0x41/0x50
[  557.055093]  [] ? _lock_kernel+0x3f/0x8a
[  557.055097]  [] ? xen_flush_tlb_others+0x6/0xe0
[  557.055099]  [] ? vfs_ioctl+0x8c/0xb0
[  557.055101]  [] ? do_vfs_ioctl+0x72/0x590
[  557.055105]  [] ? tty_ldisc_deref+0xd/0x10
[  557.055107]  [] ? tty_write+0x1b8/0x210
[  557.055109]  [] ? n_tty_write+0x0/0x3d0
[  557.055113]  [] ? vfs_write+0x12f/0x1a0
[  557.055115]  [] ? tty_write+0x0/0x210
[  557.055117]  [] ? sys_ioctl+0x67/0x80
[  557.055120]  [] ? sysenter_do_call+0x12/0x28
[  557.055123]  [] ? xen_flush_tlb_others+0x6/0xe0
[  557.055124] Code: 75 bc 31 c0 c7 45 c8 00 00 00 00 c7 45 b8 00 00 00 00 66 90 8b 55 c8 31 db 8b 74 95 cc eb 0c 90 8d 74 26 00 83 c3 01 39 fb 7d 41 <80> 3c 1e 00 74 f3 8b 45 c4 8b 50 70 89 d0 89 55 b4 e8 7b 29 e1 
[  557.055153] EIP: [] splat_kmem_test4+0x7f/0x1d0 [splat] SS:ESP 0068:edcf9e30
[  557.055157] CR2: 0000000000000000
[  557.055160] ---[ end trace bb1ca29a979e93fb ]---
@sehe
Copy link
Author

sehe commented Jul 7, 2010

I've found that I can consistently crash (oops) the module by SIG-INTing the splat userland process during the atomicity test. I presume it has to do with writing to a defunct fd (ioctl response) or with destructing the testcase state early (_fini function called early?)

This looks, however, different from the above case, which is why I attach the other (SIGINT triggered) kernel trace as well - it is quite lengthy because it appears to dump for all running threads

Jul  7 14:49:49 localhost kernel: [  189.675967] SPL: Loaded Solaris Porting Layer v0.5.0
Jul  7 14:49:49 localhost kernel: [  189.683315] SPLAT: Loaded Solaris Porting LAyer Tests v0.5.0
Jul  7 14:50:00 localhost kernel: [  200.674771] BUG: unable to handle kernel NULL pointer dereference at 00000002
Jul  7 14:50:00 localhost kernel: [  200.677501] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:50:00 localhost kernel: [  200.677501] *pdpt = 0000000036e79001 *pde = 0000000000000000 
Jul  7 14:50:00 localhost kernel: [  200.677501] Oops: 0002 [#1] SMP 
Jul  7 14:50:00 localhost kernel: [  200.677501] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:50:00 localhost kernel: [  200.677501] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:50:00 localhost kernel: [  200.677501] 
Jul  7 14:50:00 localhost kernel: [  200.677501] Pid: 24768, comm: splat_atomic_wo Not tainted 2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:50:00 localhost kernel: [  200.677501] EIP: 0060:[] EFLAGS: 00010246 CPU: 3
Jul  7 14:50:00 localhost kernel: [  200.677501] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:50:00 localhost kernel: [  200.677501] EAX: 00000002 EBX: 000aae60 ECX: ef90be34 EDX: 00000100
Jul  7 14:50:00 localhost kernel: [  200.677501] ESI: ef90be4c EDI: 00000000 EBP: efce3f40 ESP: efce3f40
Jul  7 14:50:00 localhost kernel: [  200.677501]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:50:00 localhost kernel: [  200.677501] Process splat_atomic_wo (pid: 24768, ti=efce2000 task=ef981960 task.ti=efce2000)
Jul  7 14:50:00 localhost kernel: [  200.677501] Stack:
Jul  7 14:50:00 localhost kernel: [  200.677501]  efce3f48 c05c443d efce3fa4 f8758b99 f8769868 f8761bd8 00000014 f87630d7
Jul  7 14:50:00 localhost kernel: [  200.677501] <0> 00000000 009de3e7 00000000 00000002 f6eff5a0 ef90be40 00000000 00000000
Jul  7 14:50:00 localhost kernel: [  200.677501] <0> 009de3e7 00000000 ef981960 00000020 ef90be34 00000000 f6eff5a0 f87589c0
Jul  7 14:50:00 localhost kernel: [  200.677501] Call Trace:
Jul  7 14:50:00 localhost kernel: [  200.677501]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:50:00 localhost kernel: [  200.677501]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:50:00 localhost kernel: [  200.677501]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:50:00 localhost kernel: [  200.677501]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:50:00 localhost kernel: [  200.677501]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:50:00 localhost kernel: [  200.677501]  [] ? kthread+0x74/0x80
Jul  7 14:50:00 localhost kernel: [  200.677501]  [] ? kthread+0x0/0x80
Jul  7 14:50:00 localhost kernel: [  200.677501]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:50:00 localhost kernel: [  200.677501] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:50:00 localhost kernel: [  200.677501] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:efce3f40
Jul  7 14:50:00 localhost kernel: [  200.677501] CR2: 0000000000000002
Jul  7 14:50:00 localhost kernel: [  200.677001] BUG: unable to handle kernel 
Jul  7 14:50:00 localhost kernel: [  200.897607] ---[ end trace 8f4bccb8b278b2b9 ]---
Jul  7 14:50:01 localhost kernel: [  200.677001] NULL pointer dereference at 00000002
Jul  7 14:50:01 localhost kernel: [  200.677001] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:50:01 localhost kernel: [  200.677001] *pdpt = 00000000008bc001 *pde = 0000000000000000 
Jul  7 14:50:01 localhost kernel: [  200.677001] Oops: 0002 [#2] SMP 
Jul  7 14:50:01 localhost kernel: [  200.677001] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:50:01 localhost kernel: [  200.677001] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:50:01 localhost kernel: [  200.677001] 
Jul  7 14:50:01 localhost kernel: [  200.677001] Pid: 24769, comm: splat_atomic_wo Tainted: G      D     2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:50:01 localhost kernel: [  200.677001] EIP: 0060:[] EFLAGS: 00010246 CPU: 2
Jul  7 14:50:01 localhost kernel: [  200.677001] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:50:01 localhost kernel: [  200.677001] EAX: 00000002 EBX: 000aae60 ECX: ef90be34 EDX: 00000100
Jul  7 14:50:01 localhost kernel: [  200.677001] ESI: ef90be4c EDI: 00000001 EBP: f2f99f40 ESP: f2f99f40
Jul  7 14:50:01 localhost kernel: [  200.677001]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:50:01 localhost kernel: [  200.677001] Process splat_atomic_wo (pid: 24769, ti=f2f98000 task=ef984c20 task.ti=f2f98000)
Jul  7 14:50:01 localhost kernel: [  200.677001] Stack:
Jul  7 14:50:01 localhost kernel: [  200.677001]  f2f99f48 c05c443d f2f99fa4 f8758b99 f8769834 f8761bd8 00000014 f87630d7
Jul  7 14:50:01 localhost kernel: [  200.677001] <0> 00000001 009df391 00000000 00000002 f2f99f88 ef90be40 00000000 00000000
Jul  7 14:50:01 localhost kernel: [  200.677001] <0> 009df391 00000000 ef984c20 00000020 ef90be34 00000000 f6eff7e0 f87589c0
Jul  7 14:50:01 localhost kernel: [  200.677001] Call Trace:
Jul  7 14:50:01 localhost kernel: [  200.677001]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:50:01 localhost kernel: [  200.677001]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:50:01 localhost kernel: [  200.677001]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:50:01 localhost kernel: [  200.677001]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:50:01 localhost kernel: [  200.677001]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:50:01 localhost kernel: [  200.677001]  [] ? kthread+0x74/0x80
Jul  7 14:50:01 localhost kernel: [  200.677001]  [] ? kthread+0x0/0x80
Jul  7 14:50:01 localhost kernel: [  200.677001]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:50:01 localhost kernel: [  200.677001] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:50:01 localhost kernel: [  200.677001] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:f2f99f40
Jul  7 14:50:01 localhost kernel: [  200.677001] CR2: 0000000000000002
Jul  7 14:50:01 localhost kernel: [  200.900491] BUG: unable to handle kernel 
Jul  7 14:50:01 localhost kernel: [  201.196012] ---[ end trace 8f4bccb8b278b2ba ]---
Jul  7 14:50:01 localhost kernel: [  200.900491] NULL pointer dereference at 00000002
Jul  7 14:50:01 localhost kernel: [  200.900491] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:50:01 localhost kernel: [  200.900491] *pdpt = 0000000031057001 *pde = 0000000000000000 
Jul  7 14:50:01 localhost kernel: [  200.900491] Oops: 0002 [#3] SMP 
Jul  7 14:50:01 localhost kernel: [  200.900491] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:50:01 localhost kernel: [  200.900491] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:50:01 localhost kernel: [  200.900491] 
Jul  7 14:50:01 localhost kernel: [  200.900491] Pid: 24773, comm: splat_atomic_wo Tainted: G      D     2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:50:01 localhost kernel: [  200.900491] EIP: 0060:[] EFLAGS: 00010246 CPU: 1
Jul  7 14:50:01 localhost kernel: [  200.900491] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:50:01 localhost kernel: [  200.900491] EAX: 00000002 EBX: 0007a120 ECX: ef90be34 EDX: 00000100
Jul  7 14:50:01 localhost kernel: [  200.900491] ESI: ef90be4c EDI: 00000005 EBP: f1c0df40 ESP: f1c0df40
Jul  7 14:50:01 localhost kernel: [  200.900491]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:50:01 localhost kernel: [  200.900491] Process splat_atomic_wo (pid: 24773, ti=f1c0c000 task=ef982610 task.ti=f1c0c000)
Jul  7 14:50:01 localhost kernel: [  200.900491] Stack:
Jul  7 14:50:01 localhost kernel: [  200.900491]  f1c0df48 c05c443d f1c0dfa4 f8758b99 f87698d0 f8761bd8 00000014 f87630d7
Jul  7 14:50:01 localhost kernel: [  200.900491] <0> 00000005 009ca2c9 00000000 00000002 efcf4b40 ef90be40 00000000 00000000
Jul  7 14:50:01 localhost kernel: [  200.900491] <0> 009ca2c9 00000000 ef982610 00000020 ef90be34 00000000 efcf4b40 f87589c0
Jul  7 14:50:01 localhost kernel: [  200.900491] Call Trace:
Jul  7 14:50:01 localhost kernel: [  200.900491]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:50:01 localhost kernel: [  200.900491]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:50:01 localhost kernel: [  200.900491]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:50:01 localhost kernel: [  200.900491]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:50:01 localhost kernel: [  200.900491]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:50:01 localhost kernel: [  200.900491]  [] ? kthread+0x74/0x80
Jul  7 14:50:01 localhost kernel: [  200.900491]  [] ? kthread+0x0/0x80
Jul  7 14:50:01 localhost kernel: [  200.900491]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:50:01 localhost kernel: [  200.900491] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:50:01 localhost kernel: [  200.900491] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:f1c0df40
Jul  7 14:50:01 localhost kernel: [  200.900491] CR2: 0000000000000002
Jul  7 14:50:01 localhost kernel: [  201.197315] BUG: unable to handle kernel 
Jul  7 14:50:01 localhost kernel: [  201.496471] ---[ end trace 8f4bccb8b278b2bb ]---
Jul  7 14:50:01 localhost kernel: [  201.197315] NULL pointer dereference at 00000002
Jul  7 14:50:01 localhost kernel: [  201.197315] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:50:01 localhost kernel: [  201.197315] *pdpt = 0000000037515001 *pde = 0000000000000000 
Jul  7 14:50:01 localhost kernel: [  201.197315] Oops: 0002 [#4] SMP 
Jul  7 14:50:01 localhost kernel: [  201.197315] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:50:01 localhost kernel: [  201.197315] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:50:01 localhost kernel: [  201.197315] 
Jul  7 14:50:01 localhost kernel: [  201.197315] Pid: 24772, comm: splat_atomic_wo Tainted: G      D     2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:50:01 localhost kernel: [  201.197315] EIP: 0060:[] EFLAGS: 00010246 CPU: 0
Jul  7 14:50:01 localhost kernel: [  201.197315] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:50:01 localhost kernel: [  201.197315] EAX: 00000002 EBX: 000aae60 ECX: ef90be34 EDX: 00000100
Jul  7 14:50:01 localhost kernel: [  201.197315] ESI: ef90be4c EDI: 00000004 EBP: efb37f40 ESP: efb37f40
Jul  7 14:50:01 localhost kernel: [  201.197315]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:50:01 localhost kernel: [  201.197315] Process splat_atomic_wo (pid: 24772, ti=efb36000 task=ef980000 task.ti=efb36000)
Jul  7 14:50:01 localhost kernel: [  201.197315] Stack:
Jul  7 14:50:01 localhost kernel: [  201.197315]  efb37f48 c05c443d efb37fa4 f8758b99 f876989c f8761bd8 00000014 f87630d7
Jul  7 14:50:01 localhost kernel: [  201.197315] <0> 00000004 009d13bf 00000000 00000002 efcf4060 ef90be40 00000000 00000000
Jul  7 14:50:01 localhost kernel: [  201.197315] <0> 009d13bf 00000000 ef980000 00000020 ef90be34 00000000 efcf4060 f87589c0
Jul  7 14:50:01 localhost kernel: [  201.197315] Call Trace:
Jul  7 14:50:01 localhost kernel: [  201.197315]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:50:01 localhost kernel: [  201.197315]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:50:01 localhost kernel: [  201.197315]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:50:01 localhost kernel: [  201.197315]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:50:01 localhost kernel: [  201.197315]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:50:01 localhost kernel: [  201.197315]  [] ? kthread+0x74/0x80
Jul  7 14:50:01 localhost kernel: [  201.197315]  [] ? kthread+0x0/0x80
Jul  7 14:50:01 localhost kernel: [  201.197315]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:50:01 localhost kernel: [  201.197315] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:50:01 localhost kernel: [  201.197315] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:efb37f40
Jul  7 14:50:01 localhost kernel: [  201.197315] CR2: 0000000000000002
Jul  7 14:50:01 localhost kernel: [  200.905501] BUG: unable to handle kernel 
Jul  7 14:50:01 localhost kernel: [  201.794790] ---[ end trace 8f4bccb8b278b2bc ]---
Jul  7 14:50:01 localhost kernel: [  201.804007] NULL pointer dereference at 00000002
Jul  7 14:50:01 localhost kernel: [  201.804007] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:50:01 localhost kernel: [  201.804007] *pdpt = 0000000036e79001 *pde = 0000000000000000 
Jul  7 14:50:01 localhost kernel: [  201.804007] Oops: 0002 [#5] SMP 
Jul  7 14:50:01 localhost kernel: [  201.804007] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:50:01 localhost kernel: [  201.804007] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:50:01 localhost kernel: [  201.804007] 
Jul  7 14:50:01 localhost kernel: [  201.804007] Pid: 24770, comm: splat_atomic_wo Tainted: G      D     2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:50:01 localhost kernel: [  201.804007] EIP: 0060:[] EFLAGS: 00010246 CPU: 3
Jul  7 14:50:01 localhost kernel: [  201.804007] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:50:01 localhost kernel: [  201.804007] EAX: 00000002 EBX: 000aae60 ECX: ef90be34 EDX: 00000100
Jul  7 14:50:01 localhost kernel: [  201.804007] ESI: ef90be4c EDI: 00000002 EBP: efd25f40 ESP: efd25f40
Jul  7 14:50:01 localhost kernel: [  201.804007]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:50:01 localhost kernel: [  201.804007] Process splat_atomic_wo (pid: 24770, ti=efd24000 task=ef9858d0 task.ti=efd24000)
Jul  7 14:50:01 localhost kernel: [  201.804007] Stack:
Jul  7 14:50:01 localhost kernel: [  201.804007]  efd25f48 c05c443d efd25fa4 f8758b99 f8769938 f8761bd8 00000014 f87630d7
Jul  7 14:50:01 localhost kernel: [  201.804007] <0> 00000002 009d2fa0 00000000 00000002 efcf4a80 ef90be40 00000000 00000000
Jul  7 14:50:01 localhost kernel: [  201.804007] <0> 009d2fa0 00000000 ef9858d0 00000020 ef90be34 00000000 efcf4a80 f87589c0
Jul  7 14:50:01 localhost kernel: [  201.804007] Call Trace:
Jul  7 14:50:01 localhost kernel: [  201.804007]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:50:01 localhost kernel: [  201.804007]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:50:01 localhost kernel: [  201.804007]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:50:01 localhost kernel: [  201.804007]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:50:01 localhost kernel: [  201.804007]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:50:01 localhost kernel: [  201.804007]  [] ? kthread+0x74/0x80
Jul  7 14:50:01 localhost kernel: [  201.804007]  [] ? kthread+0x0/0x80
Jul  7 14:50:01 localhost kernel: [  201.804007]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:50:01 localhost kernel: [  201.804007] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:50:01 localhost kernel: [  201.804007] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:efd25f40
Jul  7 14:50:01 localhost kernel: [  201.804007] CR2: 0000000000000002
Jul  7 14:50:01 localhost kernel: [  201.201001] BUG: unable to handle kernel NULL pointer dereference at 00000002
Jul  7 14:50:01 localhost kernel: [  201.201001] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:50:01 localhost kernel: [  201.201001] *pdpt = 0000000036e1d001 *pde = 0000000000000000 
Jul  7 14:50:01 localhost kernel: [  201.201001] Oops: 0002 [#6] SMP 
Jul  7 14:50:01 localhost kernel: [  201.201001] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:50:01 localhost kernel: [  201.807262] ---[ end trace 8f4bccb8b278b2bd ]---
Jul  7 14:50:01 localhost kernel: [  201.201001] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:50:01 localhost kernel: [  201.201001] 
Jul  7 14:50:01 localhost kernel: [  201.201001] Pid: 24771, comm: splat_atomic_wo Tainted: G      D     2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:50:01 localhost kernel: [  201.201001] EIP: 0060:[] EFLAGS: 00010246 CPU: 2
Jul  7 14:50:01 localhost kernel: [  201.201001] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:50:01 localhost kernel: [  201.201001] EAX: 00000002 EBX: 000927c0 ECX: ef90be34 EDX: 00000100
Jul  7 14:50:01 localhost kernel: [  201.201001] ESI: ef90be4c EDI: 00000003 EBP: f1615f40 ESP: f1615f40
Jul  7 14:50:01 localhost kernel: [  201.201001]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:50:01 localhost kernel: [  201.201001] Process splat_atomic_wo (pid: 24771, ti=f1614000 task=ef983f70 task.ti=f1614000)
Jul  7 14:50:01 localhost kernel: [  201.201001] Stack:
Jul  7 14:50:01 localhost kernel: [  201.201001]  f1615f48 c05c443d f1615fa4 f8758b99 f8769904 f8761bd8 00000014 f87630d7
Jul  7 14:50:01 localhost kernel: [  201.201001] <0> 00000003 009d432c 00000000 00000002 efcf4a00 ef90be40 00000000 00000000
Jul  7 14:50:01 localhost kernel: [  201.201001] <0> 009d432c 00000000 ef983f70 00000020 ef90be34 00000000 efcf4a00 f87589c0
Jul  7 14:50:01 localhost kernel: [  201.201001] Call Trace:
Jul  7 14:50:01 localhost kernel: [  201.201001]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:50:01 localhost kernel: [  201.201001]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:50:01 localhost kernel: [  201.201001]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:50:01 localhost kernel: [  201.201001]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:50:01 localhost kernel: [  201.201001]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:50:01 localhost kernel: [  201.201001]  [] ? kthread+0x74/0x80
Jul  7 14:50:01 localhost kernel: [  201.201001]  [] ? kthread+0x0/0x80
Jul  7 14:50:01 localhost kernel: [  201.201001]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:50:01 localhost kernel: [  201.201001] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:50:01 localhost kernel: [  201.201001] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:f1615f40
Jul  7 14:50:01 localhost kernel: [  201.201001] CR2: 0000000000000002
Jul  7 14:50:01 localhost kernel: [  201.807605] ---[ end trace 8f4bccb8b278b2be ]---
Jul  7 14:52:48 localhost kernel: [  366.650586] BUG: unable to handle kernel NULL pointer dereference at 00000002
Jul  7 14:52:48 localhost kernel: [  366.650588] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.650594] *pdpt = 00000000008bc001 *pde = 0000000000000000 
Jul  7 14:52:48 localhost kernel: [  366.650597] Oops: 0002 [#7] SMP 
Jul  7 14:52:48 localhost kernel: [  366.650599] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:52:48 localhost kernel: [  366.650601] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:52:48 localhost kernel: [  366.650622] 
Jul  7 14:52:48 localhost kernel: [  366.650624] Pid: 24813, comm: splat_atomic_wo Tainted: G      D     2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:52:48 localhost kernel: [  366.650627] EIP: 0060:[] EFLAGS: 00010246 CPU: 1
Jul  7 14:52:48 localhost kernel: [  366.650629] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.650630] EAX: 00000002 EBX: 0007a120 ECX: f2f3de34 EDX: 00000100
Jul  7 14:52:48 localhost kernel: [  366.650632] ESI: f2f3de4c EDI: 00000004 EBP: f1061f40 ESP: f1061f40
Jul  7 14:52:48 localhost kernel: [  366.650633]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:52:48 localhost kernel: [  366.650635] Process splat_atomic_wo (pid: 24813, ti=f1060000 task=ef9832c0 task.ti=f1060000)
Jul  7 14:52:48 localhost kernel: [  366.650636] Stack:
Jul  7 14:52:48 localhost kernel: [  366.650637]  f1061f48 c05c443d f1061fa4 f8758b99 fa46c660 f8761bd8 00000014 f87630d7
Jul  7 14:52:48 localhost kernel: [  366.650640] <0> 00000004 009b8bf0 00000000 00000002 f2e5ace0 f2f3de40 00000000 00000000
Jul  7 14:52:48 localhost kernel: [  366.650643] <0> 009b8bf0 00000000 ef9832c0 00000020 f2f3de34 00000000 f2e5ace0 f87589c0
Jul  7 14:52:48 localhost kernel: [  366.650646] Call Trace:
Jul  7 14:52:48 localhost kernel: [  366.650650]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:52:48 localhost kernel: [  366.650655]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.650658]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.650665]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.650670]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.650672]  [] ? kthread+0x74/0x80
Jul  7 14:52:48 localhost kernel: [  366.650674]  [] ? kthread+0x0/0x80
Jul  7 14:52:48 localhost kernel: [  366.650677]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:52:48 localhost kernel: [  366.650678] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:52:48 localhost kernel: [  366.650691] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:f1061f40
Jul  7 14:52:48 localhost kernel: [  366.650694] CR2: 0000000000000002
Jul  7 14:52:48 localhost kernel: [  366.650696] ---[ end trace 8f4bccb8b278b2bf ]---
Jul  7 14:52:48 localhost kernel: [  366.654621] BUG: unable to handle kernel NULL pointer dereference at 00000002
Jul  7 14:52:48 localhost kernel: [  366.654624] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.654632] *pdpt = 00000000008bc001 *pde = 0000000000000000 
Jul  7 14:52:48 localhost kernel: [  366.654634] Oops: 0002 [#8] SMP 
Jul  7 14:52:48 localhost kernel: [  366.654636] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:52:48 localhost kernel: [  366.654638] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:52:48 localhost kernel: [  366.654660] 
Jul  7 14:52:48 localhost kernel: [  366.654662] Pid: 24809, comm: splat_atomic_wo Tainted: G      D     2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:52:48 localhost kernel: [  366.654665] EIP: 0060:[] EFLAGS: 00010246 CPU: 2
Jul  7 14:52:48 localhost kernel: [  366.654667] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.654668] EAX: 00000002 EBX: 0007a120 ECX: f2f3de34 EDX: 00000100
Jul  7 14:52:48 localhost kernel: [  366.654670] ESI: f2f3de4c EDI: 00000000 EBP: f6dd1f40 ESP: f6dd1f40
Jul  7 14:52:48 localhost kernel: [  366.654671]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:52:48 localhost kernel: [  366.654673] Process splat_atomic_wo (pid: 24809, ti=f6dd0000 task=ef9858d0 task.ti=f6dd0000)
Jul  7 14:52:48 localhost kernel: [  366.654674] Stack:
Jul  7 14:52:48 localhost kernel: [  366.654676]  f6dd1f48 c05c443d f6dd1fa4 f8758b99 fa46c62c f8761bd8 00000014 f87630d7
Jul  7 14:52:48 localhost kernel: [  366.654678] <0> 00000000 009bc33a 00000000 00000002 f2e5a1e0 f2f3de40 00000000 00000000
Jul  7 14:52:48 localhost kernel: [  366.654681] <0> 009bc33a 00000000 ef9858d0 00000020 f2f3de34 00000000 f2e5a1e0 f87589c0
Jul  7 14:52:48 localhost kernel: [  366.654684] Call Trace:
Jul  7 14:52:48 localhost kernel: [  366.654689]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:52:48 localhost kernel: [  366.654694]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.654698]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.654706]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.654711]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.654714]  [] ? kthread+0x74/0x80
Jul  7 14:52:48 localhost kernel: [  366.654716]  [] ? kthread+0x0/0x80
Jul  7 14:52:48 localhost kernel: [  366.654719]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:52:48 localhost kernel: [  366.654720] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:52:48 localhost kernel: [  366.654734] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:f6dd1f40
Jul  7 14:52:48 localhost kernel: [  366.654737] CR2: 0000000000000002
Jul  7 14:52:48 localhost kernel: [  366.654739] ---[ end trace 8f4bccb8b278b2c0 ]---
Jul  7 14:52:48 localhost kernel: [  366.658846] BUG: unable to handle kernel NULL pointer dereference at 00000002
Jul  7 14:52:48 localhost kernel: [  366.658848] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.658852] *pdpt = 0000000036e79001 *pde = 0000000000000000 
Jul  7 14:52:48 localhost kernel: [  366.658854] Oops: 0002 [#9] SMP 
Jul  7 14:52:48 localhost kernel: [  366.658855] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:52:48 localhost kernel: [  366.658857] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:52:48 localhost kernel: [  366.658875] 
Jul  7 14:52:48 localhost kernel: [  366.658877] Pid: 24810, comm: splat_atomic_wo Tainted: G      D     2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:52:48 localhost kernel: [  366.658880] EIP: 0060:[] EFLAGS: 00010246 CPU: 3
Jul  7 14:52:48 localhost kernel: [  366.658881] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.658883] EAX: 00000002 EBX: 00061a80 ECX: f2f3de34 EDX: 00000100
Jul  7 14:52:48 localhost kernel: [  366.658885] ESI: f2f3de4c EDI: 00000001 EBP: f0bf7f40 ESP: f0bf7f40
Jul  7 14:52:48 localhost kernel: [  366.658886]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:52:48 localhost kernel: [  366.658888] Process splat_atomic_wo (pid: 24810, ti=f0bf6000 task=ef981960 task.ti=f0bf6000)
Jul  7 14:52:48 localhost kernel: [  366.658889] Stack:
Jul  7 14:52:48 localhost kernel: [  366.658890]  f0bf7f48 c05c443d f0bf7fa4 f8758b99 fa46c5f8 f8761bd8 00000014 f87630d7
Jul  7 14:52:48 localhost kernel: [  366.658893] <0> 00000001 009aea7f 00000000 00000002 f2e5a320 f2f3de40 00000000 00000000
Jul  7 14:52:48 localhost kernel: [  366.658896] <0> 009aea7f 00000000 ef981960 00000020 f2f3de34 00000000 f2e5a320 f87589c0
Jul  7 14:52:48 localhost kernel: [  366.658899] Call Trace:
Jul  7 14:52:48 localhost kernel: [  366.658901]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:52:48 localhost kernel: [  366.658905]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.658909]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.658914]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.658919]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.658921]  [] ? kthread+0x74/0x80
Jul  7 14:52:48 localhost kernel: [  366.658923]  [] ? kthread+0x0/0x80
Jul  7 14:52:48 localhost kernel: [  366.658925]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:52:48 localhost kernel: [  366.658926] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:52:48 localhost kernel: [  366.658940] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:f0bf7f40
Jul  7 14:52:48 localhost kernel: [  366.658942] CR2: 0000000000000002
Jul  7 14:52:48 localhost kernel: [  366.658944] ---[ end trace 8f4bccb8b278b2c1 ]---
Jul  7 14:52:48 localhost kernel: [  366.676978] BUG: unable to handle kernel NULL pointer dereference at 00000002
Jul  7 14:52:48 localhost kernel: [  366.676980] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.676982] *pdpt = 00000000008bc001 *pde = 0000000000000000 
Jul  7 14:52:48 localhost kernel: [  366.676985] Oops: 0002 [#10] SMP 
Jul  7 14:52:48 localhost kernel: [  366.676986] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:52:48 localhost kernel: [  366.676987] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:52:48 localhost kernel: [  366.677000] 
Jul  7 14:52:48 localhost kernel: [  366.677000] Pid: 24811, comm: splat_atomic_wo Tainted: G      D     2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:52:48 localhost kernel: [  366.677000] EIP: 0060:[] EFLAGS: 00010246 CPU: 2
Jul  7 14:52:48 localhost kernel: [  366.677000] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.677000] EAX: 00000002 EBX: 00061a80 ECX: f2f3de34 EDX: 00000100
Jul  7 14:52:48 localhost kernel: [  366.677000] ESI: f2f3de4c EDI: 00000002 EBP: f2de9f40 ESP: f2de9f40
Jul  7 14:52:48 localhost kernel: [  366.677000]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:52:48 localhost kernel: [  366.677000] Process splat_atomic_wo (pid: 24811, ti=f2de8000 task=ef980000 task.ti=f2de8000)
Jul  7 14:52:48 localhost kernel: [  366.677000] Stack:
Jul  7 14:52:48 localhost kernel: [  366.677000]  f2de9f48 c05c443d f2de9fa4 f8758b99 fa46c694 f8761bd8 00000014 f87630d7
Jul  7 14:52:48 localhost kernel: [  366.677000] <0> 00000002 009b702f 00000000 00000002 f2e5a8a0 f2f3de40 00000000 00000000
Jul  7 14:52:48 localhost kernel: [  366.677000] <0> 009b702f 00000000 ef980000 00000020 f2f3de34 00000000 f2e5a8a0 f87589c0
Jul  7 14:52:48 localhost kernel: [  366.677000] Call Trace:
Jul  7 14:52:48 localhost kernel: [  366.677000]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:52:48 localhost kernel: [  366.677000]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.677000]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.677000]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.677000]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.677000]  [] ? kthread+0x74/0x80
Jul  7 14:52:48 localhost kernel: [  366.677000]  [] ? kthread+0x0/0x80
Jul  7 14:52:48 localhost kernel: [  366.677000]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:52:48 localhost kernel: [  366.677000] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:52:48 localhost kernel: [  366.677000] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:f2de9f40
Jul  7 14:52:48 localhost kernel: [  366.677000] CR2: 0000000000000002
Jul  7 14:52:48 localhost kernel: [  366.677078] ---[ end trace 8f4bccb8b278b2c2 ]---
Jul  7 14:52:48 localhost kernel: [  366.681427] BUG: unable to handle kernel NULL pointer dereference at 00000002
Jul  7 14:52:48 localhost kernel: [  366.681429] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.681432] *pdpt = 00000000008bc001 *pde = 0000000000000000 
Jul  7 14:52:48 localhost kernel: [  366.681434] Oops: 0002 [#11] SMP 
Jul  7 14:52:48 localhost kernel: [  366.681435] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:52:48 localhost kernel: [  366.681437] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:52:48 localhost kernel: [  366.681455] 
Jul  7 14:52:48 localhost kernel: [  366.681457] Pid: 24812, comm: splat_atomic_wo Tainted: G      D     2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:52:48 localhost kernel: [  366.681459] EIP: 0060:[] EFLAGS: 00010246 CPU: 1
Jul  7 14:52:48 localhost kernel: [  366.681461] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.681462] EAX: 00000002 EBX: 0007a120 ECX: f2f3de34 EDX: 00000100
Jul  7 14:52:48 localhost kernel: [  366.681464] ESI: f2f3de4c EDI: 00000003 EBP: efcd3f40 ESP: efcd3f40
Jul  7 14:52:48 localhost kernel: [  366.681466]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:52:48 localhost kernel: [  366.681467] Process splat_atomic_wo (pid: 24812, ti=efcd2000 task=ef984c20 task.ti=efcd2000)
Jul  7 14:52:48 localhost kernel: [  366.681468] Stack:
Jul  7 14:52:48 localhost kernel: [  366.681469]  efcd3f48 c05c443d efcd3fa4 f8758b99 fa46c6c8 f8761bd8 00000014 f87630d7
Jul  7 14:52:48 localhost kernel: [  366.681472] <0> 00000003 009af5c9 00000000 00000002 f2e5aa40 f2f3de40 00000000 00000000
Jul  7 14:52:48 localhost kernel: [  366.681475] <0> 009af5c9 00000000 ef984c20 00000020 f2f3de34 00000000 f2e5aa40 f87589c0
Jul  7 14:52:48 localhost kernel: [  366.681478] Call Trace:
Jul  7 14:52:48 localhost kernel: [  366.681480]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:52:48 localhost kernel: [  366.681484]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.681487]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.681492]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.681497]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.681499]  [] ? kthread+0x74/0x80
Jul  7 14:52:48 localhost kernel: [  366.681501]  [] ? kthread+0x0/0x80
Jul  7 14:52:48 localhost kernel: [  366.681503]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:52:48 localhost kernel: [  366.681504] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:52:48 localhost kernel: [  366.681517] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:efcd3f40
Jul  7 14:52:48 localhost kernel: [  366.681520] CR2: 0000000000000002
Jul  7 14:52:48 localhost kernel: [  366.681522] ---[ end trace 8f4bccb8b278b2c3 ]---
Jul  7 14:52:48 localhost kernel: [  366.681636] BUG: unable to handle kernel NULL pointer dereference at 00000002
Jul  7 14:52:48 localhost kernel: [  366.681639] IP: [] __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.681642] *pdpt = 000000003698e001 *pde = 0000000000000000 
Jul  7 14:52:48 localhost kernel: [  366.681644] Oops: 0002 [#12] SMP 
Jul  7 14:52:48 localhost kernel: [  366.681645] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/net/eth1/brport/hold_timer
Jul  7 14:52:48 localhost kernel: [  366.681647] Modules linked in: splat spl autofs4 ib_iser rdma_cm ib_cm iw_cm snd_hda_codec_analog ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usbhid usblp snd fbcon tileblit font bitblit softcursor ohci1394 lp parport asus_atk0110 intel_agp agpgart soundcore vga16fb vgastate snd_page_alloc hid ieee1394 skge ahci libahci sky2 pata_jmicron
Jul  7 14:52:48 localhost kernel: [  366.681666] 
Jul  7 14:52:48 localhost kernel: [  366.681668] Pid: 24814, comm: splat_atomic_wo Tainted: G      D     2.6.35-rc3splice+ #1 P5B-Deluxe/System Product Name
Jul  7 14:52:48 localhost kernel: [  366.681670] EIP: 0060:[] EFLAGS: 00010246 CPU: 0
Jul  7 14:52:48 localhost kernel: [  366.681672] EIP is at __ticket_spin_lock+0x8/0x20
Jul  7 14:52:48 localhost kernel: [  366.681674] EAX: 00000002 EBX: 0007a120 ECX: f2f3de34 EDX: 00000100
Jul  7 14:52:48 localhost kernel: [  366.681675] ESI: f2f3de4c EDI: 00000005 EBP: f0d1ff40 ESP: f0d1ff40
Jul  7 14:52:48 localhost kernel: [  366.681677]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul  7 14:52:48 localhost kernel: [  366.681679] Process splat_atomic_wo (pid: 24814, ti=f0d1e000 task=ef982610 task.ti=f0d1e000)
Jul  7 14:52:48 localhost kernel: [  366.681680] Stack:
Jul  7 14:52:48 localhost kernel: [  366.681681]  f0d1ff48 c05c443d f0d1ffa4 f8758b99 fa46c6fc f8761bd8 00000014 f87630d7
Jul  7 14:52:48 localhost kernel: [  366.681684] <0> 00000005 009aab99 00000000 00000002 f2e5a380 f2f3de40 00000000 00000000
Jul  7 14:52:48 localhost kernel: [  366.681686] <0> 009aab99 00000000 ef982610 00000020 f2f3de34 00000000 f2e5a380 f87589c0
Jul  7 14:52:48 localhost kernel: [  366.681689] Call Trace:
Jul  7 14:52:48 localhost kernel: [  366.681692]  [] ? _raw_spin_lock+0xd/0x10
Jul  7 14:52:48 localhost kernel: [  366.681695]  [] ? splat_atomic_work+0x1d9/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.681699]  [] ? splat_atomic_work+0x0/0x340 [splat]
Jul  7 14:52:48 localhost kernel: [  366.681704]  [] ? thread_generic_wrapper+0x5e/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.681709]  [] ? thread_generic_wrapper+0x0/0x70 [spl]
Jul  7 14:52:48 localhost kernel: [  366.681711]  [] ? kthread+0x74/0x80
Jul  7 14:52:48 localhost kernel: [  366.681713]  [] ? kthread+0x0/0x80
Jul  7 14:52:48 localhost kernel: [  366.681715]  [] ? kernel_thread_helper+0x6/0x10
Jul  7 14:52:48 localhost kernel: [  366.681716] Code: b8 3d 28 13 c0 e9 59 ff ff ff 90 b9 40 28 13 c0 b8 43 28 13 c0 e9 49 ff ff ff 90 90 90 90 90 90 90 90 90 55 ba 00 01 00 00 89 e5  66 0f c1 10 38 f2 74 06 f3 90 8a 10 eb f6 5d c3 8d b4 26 00 
Jul  7 14:52:48 localhost kernel: [  366.681730] EIP: [] __ticket_spin_lock+0x8/0x20 SS:ESP 0068:f0d1ff40
Jul  7 14:52:48 localhost kernel: [  366.681733] CR2: 0000000000000002
Jul  7 14:52:48 localhost kernel: [  366.681734] ---[ end trace 8f4bccb8b278b2c4 ]---
Jul  7 14:53:20 localhost kernel: [  400.846134] SPLAT: Unloaded Solaris Porting LAyer Tests v0.5.0

@behlendorf
Copy link
Contributor

Thanks for the bug report. We'll take a look.

@nedbass
Copy link
Contributor

nedbass commented Jul 17, 2010

Hi sehe,
How were DEBUG_KMEM and DEBUG_KMEM_TRACKING set in spl_config.h when you hit the NULL pointer derefernce?

@sehe
Copy link
Author

sehe commented Jul 23, 2010 

sehe@lucid:~/custom/spl$ grep KMEM *.h
#define DEBUG_KMEM 1
/* #undef DEBUG_KMEM_TRACKING */

@behlendorf
Copy link
Contributor

This memory allocation issue should be fixed by commit 1012968. Basically the issue was there were several cases when vmem_alloc() could fail which the Solaris semantics don't allow for. The behavior has been changes to retry the allocation until it succeeds although this might re-manifest itself as a hang and linux watchdog. Unfortunately, there's not much that can be done about that in the short term.

The second SIGINT issue was fixed by commit 8f813bb and was simply a bug in the test case.

@sehe
Copy link
Author

sehe commented Jul 27, 2010

Double-ACK!

@inevity inevity mentioned this issue Apr 30, 2013
Closed
@sehe sehe mentioned this issue Apr 30, 2013
Closed
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@behlendorf @nedbass @sehe