Skip to content
This repository has been archived by the owner on Feb 26, 2020. It is now read-only.

SPL breaks System if KVM Network is up and running hard LOCKUP #66

Closed
BerndHelber opened this issue Dec 7, 2011 · 2 comments
Closed

SPL breaks System if KVM Network is up and running hard LOCKUP #66

BerndHelber opened this issue Dec 7, 2011 · 2 comments
Labels
Bug

Comments

@BerndHelber
Copy link

Hi Guys,

i tried a Setup with Pacemaker, KVM, DRBD

DRBD as Master/Slave Ressource inside Pacemaker works well, see Config attached.

If you get the KVM Network to work, i always ran in this kind of Issue which kills the Kernel completely.
The KVM Guest runs inside a ZVOL replicated with DRBD to the second Node.

Distribution: OpenSuSE 11.4
Kernel: 2.6.37.6-0.7-default
Alternate Kernel 2.6.37.6-0.9-default

But lets have a look

--------------------------------------------------------------------------------
 4928.440000]  [] default_do_nmi+0xb3/0x240
[ 4928.440000]  [] do_nmi+0x68/0x70
[ 4928.440000]  [] nmi+0x1a/0x20

For me this looks like anon maskable Interrupt.
Iron seems to be fine

If i dig deeper

> > [ 4918.868024] Call Trace:
> > [ 4918.868024]  [] __vunmap+0x7a/0xf0
> > [ 4918.868024]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
> > [ 4918.868024]  [] spl_cache_age+0x15/0x50 [spl]
> > [ 4918.868024]  [] process_one_work+0x105/0x410
> > [ 4918.868024]  [] worker_thread+0x167/0x330
> > [ 4918.868024]  [] kthread+0x96/0xa0
> > [ 4918.868024]  [] kernel_thread_helper+0x4/0x10

seems that
spl_slab_reclaim

ist triggering something weird.

If i look at the serial console messages i alsways get this back

[ 4919.500236] last sysfs file:

/sys/devices/virtual/net/virbr0/bridge/topology_change_detected

What makes me wonder is the fact, the VM is running stable if i don't use network with KVM.
If the Network is up and running, within five to ten minutes the box got killed completely.

I was able to reproduce it several times.
Hopefully it will help you guys.

691369.363848] ------------[ cut here ]------------
[691369.363848] WARNING: at
/usr/src/packages/BUILD/kernel-default-2.6.37.6/linux-2.6.37/kernel/watchdog.c:224
watchdog_overflow_callback+0xd4/0x100()
[691369.363848] Hardware name: MS-96B3
[691369.363848] Watchdog detected hard LOCKUP on cpu 0
[691369.363848] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet microcode cpufreq_conservative
cpufreq_userspace cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic e1000e pata_atiixp
edac_core sg edac_mce_amd shpchp i2c_piix4 k10temp pcspkr processor
thermal_sys pci_hotplug button serio_raw kvm_amd kvm raid456
async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx
raid10 raid1 raid0 ohci_hcd radeon ttm drm_kms_helper ssb ehci_hcd mmc_core
usbcore pcmcia pcmcia_core drm i2c_algo_bit
[691369.363848] Pid: 0, comm: swapper Tainted: P      D W  
2.6.37.6-0.7-default #1
[691369.363848] Call Trace:
[691369.363848]  [] dump_trace+0x69/0x2e0
[691369.363848]  [] dump_stack+0x69/0x6f
[691369.363848]  [] warn_slowpath_common+0x7b/0xc0
[691369.363848]  [] warn_slowpath_fmt+0x45/0x50
[691369.363848]  []
watchdog_overflow_callback+0xd4/0x100
[691369.363848]  [] __perf_event_overflow+0x99/0x220
[691369.363848]  [] x86_pmu_handle_irq+0xc6/0x110
[691369.363848]  [] perf_event_nmi_handler+0x5a/0x110
[691369.363848]  [] notifier_call_chain+0x45/0x60
[691369.363848]  [] notify_die+0x2d/0x40
[691369.363848]  [] default_do_nmi+0xb3/0x240
[691369.363848]  [] do_nmi+0x68/0x70
[691369.363848]  [] nmi+0x1a/0x20
[691369.363848]  [] _raw_spin_lock+0x18/0x20
[691369.363848]  [] task_rq_lock+0x48/0x90
[691369.363848]  [] try_to_wake_up+0x2c/0x3c0
[691369.363848]  [] pollwake+0x50/0x60
[691369.363848]  [] __wake_up_common+0x59/0x90
[691369.363848]  [] __wake_up_sync_key+0x4e/0x90
[691369.363848]  [] sock_def_readable+0x31/0x60
[691369.363848]  [] sock_queue_rcv_skb+0x131/0x1b0
[691369.363848]  [] __udp_queue_rcv_skb+0x39/0xe0
[691369.363848]  [] udp_queue_rcv_skb+0x26b/0x370
[691369.363848]  [] __udp4_lib_rcv+0x1ec/0x6e0
[691369.363848]  [] ip_local_deliver_finish+0xd4/0x290
[691369.363848]  [] __netif_receive_skb+0x2ba/0x620
[691369.363848]  [] netif_receive_skb+0x58/0x80
[691369.363848]  [] napi_skb_finish+0x38/0x50
[691369.363848]  [] e1000_clean_rx_irq+0x27b/0x420
[e1000e]
[691369.363848]  [] e1000_clean+0x7b/0x130 [e1000e]
[691369.363848]  [] net_rx_action+0x101/0x220
[691369.363848]  [] __do_softirq+0xc5/0x220
[691369.363848]  [] call_softirq+0x1c/0x30
[691369.363848]  [] do_softirq+0x4d/0x80
[691369.363848]  [] irq_exit+0x8d/0xa0
[691369.363848]  [] do_IRQ+0x67/0xe0
[691369.363848]  [] ret_from_intr+0x0/0xa
[691369.363848]  [] acpi_safe_halt+0x1e/0x30 [processor]
[691369.363848]  [] acpi_idle_enter_c1+0x55/0x9f
[processor]
[691369.363848]  [] cpuidle_idle_call+0xb8/0x2b0
[691369.363848]  [] cpu_idle+0x4c/0x90
[691369.363848]  [] start_kernel+0x38b/0x396
[691369.363848]  [] x86_64_start_kernel+0xf9/0xff
[691369.363848] ---[ end trace dfacaac0cf8b7b3d ]---

-----------------------------------------------------------------------------------------------------

1906375 login: [ 2857.936192] general protection fault: 0000 [#1] SMP 
[ 2857.940005] last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[ 2857.940005] CPU 3 
[ 2857.940005] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet cpufreq_conservative cpufreq_userspace
microcode cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic edac_core sg edac_mce_amd
pcspkr shpchp processor thermal_sys pata_atiixp button k10temp i2c_piix4
serio_raw e1000e kvm_amd pci_hotplug kvm raid456 async_raid6_recov async_pq
raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0 ohci_hcd
ehci_hcd radeon ttm usbcore drm_kms_helper ssb mmc_core pcmcia pcmcia_core
drm i2c_algo_bit
[ 2857.940005] 
[ 2857.940005] Pid: 35, comm: kworker/3:1 Tainted: P           
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[ 2857.940005] RIP: 0010:[]  []
__free_pages+0x4/0x40
[ 2857.940005] RSP: 0018:ffff8802133d5d80  EFLAGS: 00010282
[ 2857.940005] RAX: ffff88016db17800 RBX: 0000000000000040 RCX:
ffffea0005220160
[ 2857.940005] RDX: 000000000000003f RSI: 0000000000000000 RDI:
0000ea00066a8878
[ 2857.940005] RBP: ffff8801cd288ec0 R08: 0000000000000000 R09:
ffff88021ffebf10
[ 2857.940005] R10: 000000000000007e R11: 0000000000000001 R12:
0000000000000001
[ 2857.940005] R13: ffff8802133d5d98 R14: 0000000000000000 R15:
ffffc900135c2000
[ 2857.940005] FS:  00007fc56cd8c700(0000) GS:ffff8800ddd80000(0000)
knlGS:0000000000000000
[ 2857.940005] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2857.940005] CR2: 00007f42bb2138e0 CR3: 000000020636a000 CR4:
00000000000006e0
[ 2857.940005] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 2857.940005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 2857.940005] Process kworker/3:1 (pid: 35, threadinfo ffff8802133d4000,
task ffff8802133d2100)
[ 2857.940005] Stack:
[ 2857.940005]  ffffc900135c2000 ffffffff8112437a ffff88020c3e2000
ffff8802133d5dc0
[ 2857.940005]  ffff8802133d5db0 ffffffffa0314c74 ffffc900135e2018
ffffc900135c2018
[ 2857.940005]  ffffc900135bc010 ffffc900135bc010 ffff88020dbcf088
ffff88020c3e3068
[ 2857.940005] Call Trace:
[ 2857.940005]  [] __vunmap+0x7a/0xf0
[ 2857.940005]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[ 2857.940005]  [] spl_cache_age+0x15/0x50 [spl]
[ 2857.940005]  [] process_one_work+0x105/0x410
[ 2857.940005]  [] worker_thread+0x167/0x330
[ 2857.940005]  [] kthread+0x96/0xa0
[ 2857.940005]  [] kernel_thread_helper+0x4/0x10
[ 2857.940005] Code: 0f 1f 80 00 00 00 00 48 83 ec 08 48 8b 07 31 f6 f6 c4
80 74 03 8b 77 68 48 83 c4 08 e9 96 fa ff ff 66 0f 1f 44 00 00 48 83 ec 08
<8b> 47 08 f0 ff 4f 08 0f 94 c0 84 c0 74 1e 85 f6 74 0a 48 83 c4 
[ 2857.940005] RIP  [] __free_pages+0x4/0x40
[ 2857.940005]  RSP 
[ 2858.560851] ---[ end trace 52838d6a314ff1fe ]---
[ 2858.570204] BUG: unable to handle kernel paging request at
fffffffffffffff8
[ 2858.577143] IP: [] kthread_data+0x7/0x10
[ 2858.577143] PGD 1a05067 PUD 1a06067 PMD 0 
[ 2858.577143] Oops: 0000 [#2] SMP 
[ 2858.577143] last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[ 2858.577143] CPU 3 
[ 2858.577143] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet cpufreq_conservative cpufreq_userspace
microcode cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic edac_core sg edac_mce_amd
pcspkr shpchp processor thermal_sys pata_atiixp button k10temp i2c_piix4
serio_raw e1000e kvm_amd pci_hotplug kvm raid456 async_raid6_recov async_pq
raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0 ohci_hcd
ehci_hcd radeon ttm usbcore drm_kms_helper ssb mmc_core pcmcia pcmcia_core
drm i2c_algo_bit
[ 2858.577143] 
[ 2858.577143] Pid: 35, comm: kworker/3:1 Tainted: P      D    
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[ 2858.577143] RIP: 0010:[]  []
kthread_data+0x7/0x10
[ 2858.577143] RSP: 0018:ffff8802133d5ba0  EFLAGS: 00010002
[ 2858.577143] RAX: 0000000000000000 RBX: 0000000000000003 RCX:
ffff8802133d2100
[ 2858.577143] RDX: 0000000000000aa4 RSI: 0000000000000003 RDI:
ffff8802133d2100
[ 2858.577143] RBP: ffff8802133d5c48 R08: ffff880216f7c548 R09:
dead000000200200
[ 2858.577143] R10: dead000000200200 R11: 0000000000000000 R12:
0000000000000003
[ 2858.577143] R13: ffff8802133d24d0 R14: 0000000000000000 R15:
00000000ffffffff
[ 2858.577143] FS:  00007fc56cd8c700(0000) GS:ffff8800ddd80000(0000)
knlGS:0000000000000000
[ 2858.577143] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2858.577143] CR2: fffffffffffffff8 CR3: 000000020636a000 CR4:
00000000000006e0
[ 2858.577143] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 2858.577143] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 2858.577143] Process kworker/3:1 (pid: 35, threadinfo ffff8802133d4000,
task ffff8802133d2100)
[ 2858.577143] Stack:
[ 2858.577143]  ffffffff81072eb8 ffff8800ddd92500 ffffffff814bb10a
0000000000012500
[ 2858.577143]  ffff8802133d5fd8 0000000000012500 ffff8802133d4000
0000000000012500
[ 2858.577143]  ffff8802133d24d0 0000000000012500 0000000000012500
ffff8802133d5fd8
[ 2858.577143] Call Trace:
[ 2858.577143]  [] wq_worker_sleeping+0x8/0x90
[ 2858.577143]  [] thread_return+0x232/0x398
[ 2858.577143]  [] do_exit+0x235/0x400
[ 2858.577143]  [] oops_end+0x9c/0xe0
[ 2858.577143]  [] general_protection+0x1f/0x30
[ 2858.577143]  [] __free_pages+0x4/0x40
[ 2858.577143]  [] __vunmap+0x7a/0xf0
[ 2858.577143]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[ 2858.577143]  [] spl_cache_age+0x15/0x50 [spl]
[ 2858.577143]  [] process_one_work+0x105/0x410
[ 2858.577143]  [] worker_thread+0x167/0x330
[ 2858.577143]  [] kthread+0x96/0xa0
[ 2858.577143]  [] kernel_thread_helper+0x4/0x10
[ 2858.577143] Code: 48 c7 c6 20 cb 60 81 e8 88 cb fc ff 48 8b 44 24 10 48
81 c4 b8 00 00 00 5b 5d c3 66 0f 1f 84 00 00 00 00 00 48 8b 87 78 03 00 00
<48> 8b 40 f8 c3 0f 1f 40 00 41 55 48 c7 c6 ff 4f 79 81 41 54 65 
[ 2858.577143] RIP  [] kthread_data+0x7/0x10
[ 2858.577143]  RSP 
[ 2858.577143] CR2: fffffffffffffff8
[ 2858.577143] ---[ end trace 52838d6a314ff1ff ]---
[ 2858.577143] Fixing recursive fault but reboot is needed!




Master Resource Control: Running /etc/init.d/after.local
..doneMaster Resource Control: runlevel 3 has been reached
Failed services in runlevel 3:                                           
network

Welcome to openSUSE 11.4 "Celadon" - Kernel 2.6.37.6-0.7-default (ttyS0).


node0 login: [ 2857.936192] general protection fault: 0000 [#1] SMP 
[ 2857.940005] last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[ 2857.940005] CPU 3 
[ 2857.940005] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet cpufreq_conservative cpufreq_userspace
microcode cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic edac_core sg edac_mce_amd
pcspkr shpchp processor thermal_sys pata_atiixp button k10temp i2c_piix4
serio_raw e1000e kvm_amd pci_hotplug kvm raid456 async_raid6_recov async_pq
raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0 ohci_hcd
ehci_hcd radeon ttm usbcore drm_kms_helper ssb mmc_core pcmcia pcmcia_core
drm i2c_algo_bit
[ 2857.940005] 
[ 2857.940005] Pid: 35, comm: kworker/3:1 Tainted: P           
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[ 2857.940005] RIP: 0010:[]  []
__free_pages+0x4/0x40
[ 2857.940005] RSP: 0018:ffff8802133d5d80  EFLAGS: 00010282
[ 2857.940005] RAX: ffff88016db17800 RBX: 0000000000000040 RCX:
ffffea0005220160
[ 2857.940005] RDX: 000000000000003f RSI: 0000000000000000 RDI:
0000ea00066a8878
[ 2857.940005] RBP: ffff8801cd288ec0 R08: 0000000000000000 R09:
ffff88021ffebf10
[ 2857.940005] R10: 000000000000007e R11: 0000000000000001 R12:
0000000000000001
[ 2857.940005] R13: ffff8802133d5d98 R14: 0000000000000000 R15:
ffffc900135c2000
[ 2857.940005] FS:  00007fc56cd8c700(0000) GS:ffff8800ddd80000(0000)
knlGS:0000000000000000
[ 2857.940005] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2857.940005] CR2: 00007f42bb2138e0 CR3: 000000020636a000 CR4:
00000000000006e0
[ 2857.940005] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 2857.940005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 2857.940005] Process kworker/3:1 (pid: 35, threadinfo ffff8802133d4000,
task ffff8802133d2100)
[ 2857.940005] Stack:
[ 2857.940005]  ffffc900135c2000 ffffffff8112437a ffff88020c3e2000
ffff8802133d5dc0
[ 2857.940005]  ffff8802133d5db0 ffffffffa0314c74 ffffc900135e2018
ffffc900135c2018
[ 2857.940005]  ffffc900135bc010 ffffc900135bc010 ffff88020dbcf088
ffff88020c3e3068
[ 2857.940005] Call Trace:
[ 2857.940005]  [] __vunmap+0x7a/0xf0
[ 2857.940005]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[ 2857.940005]  [] spl_cache_age+0x15/0x50 [spl]
[ 2857.940005]  [] process_one_work+0x105/0x410
[ 2857.940005]  [] worker_thread+0x167/0x330
[ 2857.940005]  [] kthread+0x96/0xa0
[ 2857.940005]  [] kernel_thread_helper+0x4/0x10
[ 2857.940005] Code: 0f 1f 80 00 00 00 00 48 83 ec 08 48 8b 07 31 f6 f6 c4
80 74 03 8b 77 68 48 83 c4 08 e9 96 fa ff ff 66 0f 1f 44 00 00 48 83 ec 08
<8b> 47 08 f0 ff 4f 08 0f 94 c0 84 c0 74 1e 85 f6 74 0a 48 83 c4 
[ 2857.940005] RIP  [] __free_pages+0x4/0x40
[ 2857.940005]  RSP 
[ 2858.560851] ---[ end trace 52838d6a314ff1fe ]---
[ 2858.570204] BUG: unable to handle kernel paging request at
fffffffffffffff8
[ 2858.577143] IP: [] kthread_data+0x7/0x10
[ 2858.577143] PGD 1a05067 PUD 1a06067 PMD 0 
[ 2858.577143] Oops: 0000 [#2] SMP 
[ 2858.577143] last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[ 2858.577143] CPU 3 
[ 2858.577143] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet cpufreq_conservative cpufreq_userspace
microcode cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic edac_core sg edac_mce_amd
pcspkr shpchp processor thermal_sys pata_atiixp button k10temp i2c_piix4
serio_raw e1000e kvm_amd pci_hotplug kvm raid456 async_raid6_recov async_pq
raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0 ohci_hcd
ehci_hcd radeon ttm usbcore drm_kms_helper ssb mmc_core pcmcia pcmcia_core
drm i2c_algo_bit
[ 2858.577143] 
[ 2858.577143] Pid: 35, comm: kworker/3:1 Tainted: P      D    
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[ 2858.577143] RIP: 0010:[]  []
kthread_data+0x7/0x10
[ 2858.577143] RSP: 0018:ffff8802133d5ba0  EFLAGS: 00010002
[ 2858.577143] RAX: 0000000000000000 RBX: 0000000000000003 RCX:
ffff8802133d2100
[ 2858.577143] RDX: 0000000000000aa4 RSI: 0000000000000003 RDI:
ffff8802133d2100
[ 2858.577143] RBP: ffff8802133d5c48 R08: ffff880216f7c548 R09:
dead000000200200
[ 2858.577143] R10: dead000000200200 R11: 0000000000000000 R12:
0000000000000003
[ 2858.577143] R13: ffff8802133d24d0 R14: 0000000000000000 R15:
00000000ffffffff
[ 2858.577143] FS:  00007fc56cd8c700(0000) GS:ffff8800ddd80000(0000)
knlGS:0000000000000000
[ 2858.577143] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2858.577143] CR2: fffffffffffffff8 CR3: 000000020636a000 CR4:
00000000000006e0
[ 2858.577143] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 2858.577143] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 2858.577143] Process kworker/3:1 (pid: 35, threadinfo ffff8802133d4000,
task ffff8802133d2100)
[ 2858.577143] Stack:
[ 2858.577143]  ffffffff81072eb8 ffff8800ddd92500 ffffffff814bb10a
0000000000012500
[ 2858.577143]  ffff8802133d5fd8 0000000000012500 ffff8802133d4000
0000000000012500
[ 2858.577143]  ffff8802133d24d0 0000000000012500 0000000000012500
ffff8802133d5fd8
[ 2858.577143] Call Trace:
[ 2858.577143]  [] wq_worker_sleeping+0x8/0x90
[ 2858.577143]  [] thread_return+0x232/0x398
[ 2858.577143]  [] do_exit+0x235/0x400
[ 2858.577143]  [] oops_end+0x9c/0xe0
[ 2858.577143]  [] general_protection+0x1f/0x30
[ 2858.577143]  [] __free_pages+0x4/0x40
[ 2858.577143]  [] __vunmap+0x7a/0xf0
[ 2858.577143]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[ 2858.577143]  [] spl_cache_age+0x15/0x50 [spl]
[ 2858.577143]  [] process_one_work+0x105/0x410
[ 2858.577143]  [] worker_thread+0x167/0x330
[ 2858.577143]  [] kthread+0x96/0xa0
[ 2858.577143]  [] kernel_thread_helper+0x4/0x10
[ 2858.577143] Code: 48 c7 c6 20 cb 60 81 e8 88 cb fc ff 48 8b 44 24 10 48
81 c4 b8 00 00 00 5b 5d c3 66 0f 1f 84 00 00 00 00 00 48 8b 87 78 03 00 00
<48> 8b 40 f8 c3 0f 1f 40 00 41 55 48 c7 c6 ff 4f 79 81 41 54 65 
[ 2858.577143] RIP  [] kthread_data+0x7/0x10
[ 2858.577143]  RSP 
[ 2858.577143] CR2: fffffffffffffff8
[ 2858.577143] ---[ end trace 52838d6a314ff1ff ]---
[ 2858.577143] Fixing recursive fault but reboot is needed!
[ 2859.313993] ------------[ cut here ]------------
[ 2859.313993] WARNING: at
/usr/src/packages/BUILD/kernel-default-2.6.37.6/linux-2.6.37/kernel/watchdog.c:224
watchdog_overflow_callback+0xd4/0x100()
[ 2859.313993] Hardware name: MS-96B3
[ 2859.313993] Watchdog detected hard LOCKUP on cpu 0
[ 2859.313993] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet cpufreq_conservative cpufreq_userspace
microcode cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic edac_core sg edac_mce_amd
pcspkr shpchp processor thermal_sys pata_atiixp button k10temp i2c_piix4
serio_raw e1000e kvm_amd pci_hotplug kvm raid456 async_raid6_recov async_pq
raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0 ohci_hcd
ehci_hcd radeon ttm usbcore drm_kms_helper ssb mmc_core pcmcia pcmcia_core
drm i2c_algo_bit
[ 2859.313993] Pid: 6457, comm: lrmd Tainted: P      D    
2.6.37.6-0.7-default #1
[ 2859.313993] Call Trace:
[ 2859.313993]  [] dump_trace+0x69/0x2e0
[ 2859.313993]  [] dump_stack+0x69/0x6f
[ 2859.313993]  [] warn_slowpath_common+0x7b/0xc0
[ 2859.313993]  [] warn_slowpath_fmt+0x45/0x50
[ 2859.313993]  [] watchdog_overflow_callback+0xd4/0x100
[ 2859.313993]  [] __perf_event_overflow+0x99/0x220
[ 2859.313993]  [] x86_pmu_handle_irq+0xc6/0x110
[ 2859.313993]  [] perf_event_nmi_handler+0x5a/0x110
[ 2859.313993]  [] notifier_call_chain+0x45/0x60
[ 2859.313993]  [] notify_die+0x2d/0x40
[ 2859.313993]  [] default_do_nmi+0xb3/0x240
[ 2859.313993]  [] do_nmi+0x68/0x70
[ 2859.313993]  [] nmi+0x1a/0x20
[ 2859.313993]  [] _raw_spin_lock+0x11/0x20
[ 2859.313993]  [] double_lock_balance+0x49/0x70
[ 2859.313993]  [] find_lock_lowest_rq+0x5e/0xf0
[ 2859.313993]  [] push_rt_task+0x90/0x180
[ 2859.313993]  [] push_rt_tasks+0x18/0x30
[ 2859.313993]  [] thread_return+0x19c/0x398
[ 2859.313993]  []
schedule_hrtimeout_range_clock+0xb7/0x150
[ 2859.313993]  [] poll_schedule_timeout+0x44/0x60
[ 2859.313993]  [] do_poll+0x1dc/0x2a0
[ 2859.313993]  [] do_sys_poll+0x178/0x1f0
[ 2859.313993]  [] sys_poll+0x71/0x110
[ 2859.313993]  [] system_call_fastpath+0x16/0x1b
[ 2859.313993]  [<00007f28a42264b8>] 0x7f28a42264b8
[ 2859.313993] ---[ end trace 52838d6a314ff200 

--------------------------------------------------------------------------------------------

node0 login: [ 4918.865037] general protection fault: 0000 [#1] SMP 
[ 4918.868024] last sysfs file:
/sys/devices/virtual/net/virbr0/bridge/topology_change_detected
[ 4918.868024] CPU 3 
[ 4918.868024] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet microcode cpufreq_conservative
cpufreq_userspace cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic i2c_piix4 edac_core
edac_mce_amd pata_atiixp button sg k10temp processor thermal_sys e1000e
serio_raw shpchp pci_hotplug pcspkr kvm_amd kvm raid456 async_raid6_recov
async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0
ohci_hcd ehci_hcd radeon ttm usbcore ssb drm_kms_helper mmc_core pcmcia
pcmcia_core drm i2c_algo_bit
[ 4918.868024] 
[ 4918.868024] Pid: 58, comm: kworker/3:1 Tainted: P           
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[ 4918.868024] RIP: 0010:[]  []
__free_pages+0x4/0x40
[ 4918.868024] RSP: 0018:ffff88021025bd80  EFLAGS: 00010282
[ 4918.868024] RAX: ffff880203326600 RBX: 0000000000000040 RCX:
ffffea0006dc4fd8
[ 4918.868024] RDX: 000000000000003f RSI: 0000000000000000 RDI:
0000ea0005cc9370
[ 4918.868024] RBP: ffff8802081ec680 R08: 0000000000000000 R09:
ffff88021ffebf70
[ 4918.868024] R10: 000000000000008a R11: 0000000000000001 R12:
0000000000000001
[ 4918.868024] R13: ffff88021025bd98 R14: 0000000000000000 R15:
ffffc9001f364000
[ 4918.868024] FS:  00007f7c4a7b77c0(0000) GS:ffff8800ddd80000(0000)
knlGS:0000000000000000
[ 4918.868024] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4918.868024] CR2: 00007fbde449a8e0 CR3: 00000001f2022000 CR4:
00000000000006e0
[ 4918.868024] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 4918.868024] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 4918.868024] Process kworker/3:1 (pid: 58, threadinfo ffff88021025a000,
task ffff8802102586c0)
[ 4918.868024] Stack:
[ 4918.868024]  ffffc9001f364000 ffffffff8112437a ffff88021418c000
ffff88021025bdc0
[ 4918.868024]  ffffc9002064d000 ffffffffa035cc74 ffffc9001803a018
ffffc9001f364018
[ 4918.868024]  ffffc9001802e010 ffffc9001f361010 ffff8802141bf088
ffff88021418d068
[ 4918.868024] Call Trace:
[ 4918.868024]  [] __vunmap+0x7a/0xf0
[ 4918.868024]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[ 4918.868024]  [] spl_cache_age+0x15/0x50 [spl]
[ 4918.868024]  [] process_one_work+0x105/0x410
[ 4918.868024]  [] worker_thread+0x167/0x330
[ 4918.868024]  [] kthread+0x96/0xa0
[ 4918.868024]  [] kernel_thread_helper+0x4/0x10
[ 4918.868024] Code: 0f 1f 80 00 00 00 00 48 83 ec 08 48 8b 07 31 f6 f6 c4
80 74 03 8b 77 68 48 83 c4 08 e9 96 fa ff ff 66 0f 1f 44 00 00 48 83 ec 08
<8b> 47 08 f0 ff 4f 08 0f 94 c0 84 c0 74 1e 85 f6 74 0a 48 83 c4 
[ 4918.868024] RIP  [] __free_pages+0x4/0x40
[ 4918.868024]  RSP 
[ 4919.490875] ---[ end trace 21ecce834cb5e1af ]---
[ 4919.500210] BUG: unable to handle kernel paging request at
fffffffffffffff8
[ 4919.500216] IP: [] kthread_data+0x7/0x10
[ 4919.500224] PGD 1a05067 PUD 1a06067 PMD 0 
[ 4919.500230] Oops: 0000 [#2] SMP 
[ 4919.500236] last sysfs file:
/sys/devices/virtual/net/virbr0/bridge/topology_change_detected
[ 4919.500240] CPU 3 
[ 4919.500242] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet microcode cpufreq_conservative
cpufreq_userspace cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic i2c_piix4 edac_core
edac_mce_amd pata_atiixp button sg k10temp processor thermal_sys e1000e
serio_raw shpchp pci_hotplug pcspkr kvm_amd kvm raid456 async_raid6_recov
async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0
ohci_hcd ehci_hcd radeon ttm usbcore ssb drm_kms_helper mmc_core pcmcia
pcmcia_core drm i2c_algo_bit
[ 4919.500335] 
[ 4919.500340] Pid: 58, comm: kworker/3:1 Tainted: P      D    
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[ 4919.500349] RIP: 0010:[]  []
kthread_data+0x7/0x10
[ 4919.500357] RSP: 0018:ffff88021025bba0  EFLAGS: 00010002
[ 4919.500361] RAX: 0000000000000000 RBX: 0000000000000003 RCX:
ffff8802102586c0
[ 4919.500366] RDX: 000000000000812e RSI: 0000000000000003 RDI:
ffff8802102586c0
[ 4919.500371] RBP: ffff88021025bc48 R08: ffff88021f9f7948 R09:
dead000000200200
[ 4919.500375] R10: dead000000200200 R11: 0000000000000000 R12:
0000000000000003
[ 4919.500379] R13: ffff880210258a90 R14: 0000000000000000 R15:
00000000ffffffff
[ 4919.500385] FS:  00007f7c4a7b77c0(0000) GS:ffff8800ddd80000(0000)
knlGS:0000000000000000
[ 4919.500389] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4919.500394] CR2: fffffffffffffff8 CR3: 00000001f2022000 CR4:
00000000000006e0
[ 4919.500398] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 4919.500403] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 4919.500408] Process kworker/3:1 (pid: 58, threadinfo ffff88021025a000,
task ffff8802102586c0)
[ 4919.500411] Stack:
[ 4919.500413]  ffffffff81072eb8 ffff8800ddd92500 ffffffff814bb10a
0000000000012500
[ 4919.500421]  ffff88021025bfd8 0000000000012500 ffff88021025a000
0000000000012500
[ 4919.500428]  ffff880210258a90 0000000000012500 0000000000012500
ffff88021025bfd8
[ 4919.500434] Call Trace:
[ 4919.500444]  [] wq_worker_sleeping+0x8/0x90
[ 4919.500456]  [] thread_return+0x232/0x398
[ 4919.500471]  [] do_exit+0x235/0x400
[ 4919.500484]  [] oops_end+0x9c/0xe0
[ 4919.500495]  [] general_protection+0x1f/0x30
[ 4919.500508]  [] __free_pages+0x4/0x40
[ 4919.500517]  [] __vunmap+0x7a/0xf0
[ 4919.500539]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[ 4919.500577]  [] spl_cache_age+0x15/0x50 [spl]
[ 4919.500600]  [] process_one_work+0x105/0x410
[ 4919.500610]  [] worker_thread+0x167/0x330
[ 4919.500620]  [] kthread+0x96/0xa0
[ 4919.500629]  [] kernel_thread_helper+0x4/0x10
[ 4919.500636] Code: 48 c7 c6 20 cb 60 81 e8 88 cb fc ff 48 8b 44 24 10 48
81 c4 b8 00 00 00 5b 5d c3 66 0f 1f 84 00 00 00 00 00 48 8b 87 78 03 00 00
<48> 8b 40 f8 c3 0f 1f 40 00 41 55 48 c7 c6 ff 4f 79 81 41 54 65 
[ 4919.500676] RIP  [] kthread_data+0x7/0x10
[ 4919.500683]  RSP 
[ 4919.500686] CR2: fffffffffffffff8
[ 4919.500689] ---[ end trace 21ecce834cb5e1b0 ]---
[ 4919.500692] Fixing recursive fault but reboot is needed!

----------------------------------------------------------------------------------------
" - Kernel 2.6.37.6-0.7-default (ttyS0).


node0 login: [ 4918.865037] general protection fault: 0000 [#1] SMP 
[ 4918.868024] last sysfs file:
/sys/devices/virtual/net/virbr0/bridge/topology_change_detected
[ 4918.868024] CPU 3 
[ 4918.868024] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet microcode cpufreq_conservative
cpufreq_userspace cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic i2c_piix4 edac_core
edac_mce_amd pata_atiixp button sg k10temp processor thermal_sys e1000e
serio_raw shpchp pci_hotplug pcspkr kvm_amd kvm raid456 async_raid6_recov
async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0
ohci_hcd ehci_hcd radeon ttm usbcore ssb drm_kms_helper mmc_core pcmcia
pcmcia_core drm i2c_algo_bit
[ 4918.868024] 
[ 4918.868024] Pid: 58, comm: kworker/3:1 Tainted: P           
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[ 4918.868024] RIP: 0010:[]  []
__free_pages+0x4/0x40
[ 4918.868024] RSP: 0018:ffff88021025bd80  EFLAGS: 00010282
[ 4918.868024] RAX: ffff880203326600 RBX: 0000000000000040 RCX:
ffffea0006dc4fd8
[ 4918.868024] RDX: 000000000000003f RSI: 0000000000000000 RDI:
0000ea0005cc9370
[ 4918.868024] RBP: ffff8802081ec680 R08: 0000000000000000 R09:
ffff88021ffebf70
[ 4918.868024] R10: 000000000000008a R11: 0000000000000001 R12:
0000000000000001
[ 4918.868024] R13: ffff88021025bd98 R14: 0000000000000000 R15:
ffffc9001f364000
[ 4918.868024] FS:  00007f7c4a7b77c0(0000) GS:ffff8800ddd80000(0000)
knlGS:0000000000000000
[ 4918.868024] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4918.868024] CR2: 00007fbde449a8e0 CR3: 00000001f2022000 CR4:
00000000000006e0
[ 4918.868024] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 4918.868024] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 4918.868024] Process kworker/3:1 (pid: 58, threadinfo ffff88021025a000,
task ffff8802102586c0)
[ 4918.868024] Stack:
[ 4918.868024]  ffffc9001f364000 ffffffff8112437a ffff88021418c000
ffff88021025bdc0
[ 4918.868024]  ffffc9002064d000 ffffffffa035cc74 ffffc9001803a018
ffffc9001f364018
[ 4918.868024]  ffffc9001802e010 ffffc9001f361010 ffff8802141bf088
ffff88021418d068
[ 4918.868024] Call Trace:
[ 4918.868024]  [] __vunmap+0x7a/0xf0
[ 4918.868024]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[ 4918.868024]  [] spl_cache_age+0x15/0x50 [spl]
[ 4918.868024]  [] process_one_work+0x105/0x410
[ 4918.868024]  [] worker_thread+0x167/0x330
[ 4918.868024]  [] kthread+0x96/0xa0
[ 4918.868024]  [] kernel_thread_helper+0x4/0x10
[ 4918.868024] Code: 0f 1f 80 00 00 00 00 48 83 ec 08 48 8b 07 31 f6 f6 c4
80 74 03 8b 77 68 48 83 c4 08 e9 96 fa ff ff 66 0f 1f 44 00 00 48 83 ec 08
<8b> 47 08 f0 ff 4f 08 0f 94 c0 84 c0 74 1e 85 f6 74 0a 48 83 c4 
[ 4918.868024] RIP  [] __free_pages+0x4/0x40
[ 4918.868024]  RSP 
[ 4919.490875] ---[ end trace 21ecce834cb5e1af ]---
[ 4919.500210] BUG: unable to handle kernel paging request at
fffffffffffffff8
[ 4919.500216] IP: [] kthread_data+0x7/0x10
[ 4919.500224] PGD 1a05067 PUD 1a06067 PMD 0 
[ 4919.500230] Oops: 0000 [#2] SMP 
[ 4919.500236] last sysfs file:
/sys/devices/virtual/net/virbr0/bridge/topology_change_detected
[ 4919.500240] CPU 3 
[ 4919.500242] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet microcode cpufreq_conservative
cpufreq_userspace cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic i2c_piix4 edac_core
edac_mce_amd pata_atiixp button sg k10temp processor thermal_sys e1000e
serio_raw shpchp pci_hotplug pcspkr kvm_amd kvm raid456 async_raid6_recov
async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0
ohci_hcd ehci_hcd radeon ttm usbcore ssb drm_kms_helper mmc_core pcmcia
pcmcia_core drm i2c_algo_bit
[ 4919.500335] 
[ 4919.500340] Pid: 58, comm: kworker/3:1 Tainted: P      D    
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[ 4919.500349] RIP: 0010:[]  []
kthread_data+0x7/0x10
[ 4919.500357] RSP: 0018:ffff88021025bba0  EFLAGS: 00010002
[ 4919.500361] RAX: 0000000000000000 RBX: 0000000000000003 RCX:
ffff8802102586c0
[ 4919.500366] RDX: 000000000000812e RSI: 0000000000000003 RDI:
ffff8802102586c0
[ 4919.500371] RBP: ffff88021025bc48 R08: ffff88021f9f7948 R09:
dead000000200200
[ 4919.500375] R10: dead000000200200 R11: 0000000000000000 R12:
0000000000000003
[ 4919.500379] R13: ffff880210258a90 R14: 0000000000000000 R15:
00000000ffffffff
[ 4919.500385] FS:  00007f7c4a7b77c0(0000) GS:ffff8800ddd80000(0000)
knlGS:0000000000000000
[ 4919.500389] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4919.500394] CR2: fffffffffffffff8 CR3: 00000001f2022000 CR4:
00000000000006e0
[ 4919.500398] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 4919.500403] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 4919.500408] Process kworker/3:1 (pid: 58, threadinfo ffff88021025a000,
task ffff8802102586c0)
[ 4919.500411] Stack:
[ 4919.500413]  ffffffff81072eb8 ffff8800ddd92500 ffffffff814bb10a
0000000000012500
[ 4919.500421]  ffff88021025bfd8 0000000000012500 ffff88021025a000
0000000000012500
[ 4919.500428]  ffff880210258a90 0000000000012500 0000000000012500
ffff88021025bfd8
[ 4919.500434] Call Trace:
[ 4919.500444]  [] wq_worker_sleeping+0x8/0x90
[ 4919.500456]  [] thread_return+0x232/0x398
[ 4919.500471]  [] do_exit+0x235/0x400
[ 4919.500484]  [] oops_end+0x9c/0xe0
[ 4919.500495]  [] general_protection+0x1f/0x30
[ 4919.500508]  [] __free_pages+0x4/0x40
[ 4919.500517]  [] __vunmap+0x7a/0xf0
[ 4919.500539]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[ 4919.500577]  [] spl_cache_age+0x15/0x50 [spl]
[ 4919.500600]  [] process_one_work+0x105/0x410
[ 4919.500610]  [] worker_thread+0x167/0x330
[ 4919.500620]  [] kthread+0x96/0xa0
[ 4919.500629]  [] kernel_thread_helper+0x4/0x10
[ 4919.500636] Code: 48 c7 c6 20 cb 60 81 e8 88 cb fc ff 48 8b 44 24 10 48
81 c4 b8 00 00 00 5b 5d c3 66 0f 1f 84 00 00 00 00 00 48 8b 87 78 03 00 00
<48> 8b 40 f8 c3 0f 1f 40 00 41 55 48 c7 c6 ff 4f 79 81 41 54 65 
[ 4919.500676] RIP  [] kthread_data+0x7/0x10
[ 4919.500683]  RSP 
[ 4919.500686] CR2: fffffffffffffff8
[ 4919.500689] ---[ end trace 21ecce834cb5e1b0 ]---
[ 4919.500692] Fixing recursive fault but reboot is needed!
[ 4928.440000] ------------[ cut here ]------------
[ 4928.440000] WARNING: at
/usr/src/packages/BUILD/kernel-default-2.6.37.6/linux-2.6.37/kernel/watchdog.c:224
watchdog_overflow_callback+0xd4/0x100()
[ 4928.440000] Hardware name: MS-96B3
[ 4928.440000] Watchdog detected hard LOCKUP on cpu 1
[ 4928.440000] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet microcode cpufreq_conservative
cpufreq_userspace cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic i2c_piix4 edac_core
edac_mce_amd pata_atiixp button sg k10temp processor thermal_sys e1000e
serio_raw shpchp pci_hotplug pcspkr kvm_amd kvm raid456 async_raid6_recov
async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0
ohci_hcd ehci_hcd radeon ttm usbcore ssb drm_kms_helper mmc_core pcmcia
pcmcia_core drm i2c_algo_bit
[ 4928.440000] Pid: 748, comm: qemu-kvm Tainted: P      D    
2.6.37.6-0.7-default #1
[ 4928.440000] Call Trace:
[ 4928.440000]  [] dump_trace+0x69/0x2e0
[ 4928.440000]  [] dump_stack+0x69/0x6f
[ 4928.440000]  [] warn_slowpath_common+0x7b/0xc0
[ 4928.440000]  [] warn_slowpath_fmt+0x45/0x50
[ 4928.440000]  [] watchdog_overflow_callback+0xd4/0x100
[ 4928.440000]  [] __perf_event_overflow+0x99/0x220
[ 4928.440000]  [] x86_pmu_handle_irq+0xc6/0x110
[ 4928.440000]  [] perf_event_nmi_handler+0x5a/0x110
[ 4928.440000]  [] notifier_call_chain+0x45/0x60
[ 4928.440000]  [] notify_die+0x2d/0x40
[ 4928.440000]  [] default_do_nmi+0xb3/0x240
[ 4928.440000]  [] do_nmi+0x68/0x70
[ 4928.440000]  [] nmi+0x1a/0x20
[ 4928.440000]  [<00007fff1c9ff670>] 0x7fff1c9ff670
[ 4928.440000] ---[ end trace 21ecce834cb5e1b1 ]---

h1906375 login: [  541.520750] general protection fault: 0000 [#1] SMP 
[  541.524006] last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[  541.524006] CPU 0 
[  541.524006] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd ib_iser rdma_cm ib_cm iw_cm
ib_sa lru_cache ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet cpufreq_conservative microcode
cpufreq_userspace cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic pata_atiixp sg i2c_piix4
edac_core edac_mce_amd k10temp serio_raw e1000e shpchp pci_hotplug pcspkr
kvm_amd kvm processor thermal_sys button raid456 async_raid6_recov async_pq
raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0 ohci_hcd
ehci_hcd radeon ttm drm_kms_helper usbcore ssb mmc_core pcmcia pcmcia_core
drm i2c_algo_bit
[  541.524006] 
[  541.524006] Pid: 11, comm: kworker/0:1 Tainted: P           
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[  541.524006] RIP: 0010:[]  []
__free_pages+0x4/0x40
[  541.524006] RSP: 0018:ffff88021694bd80  EFLAGS: 00010282
[  541.524006] RAX: ffff88017ba38200 RBX: 0000000000000040 RCX:
ffffea00071f7f48
[  541.524006] RDX: 000000000000003f RSI: 0000000000000000 RDI:
0000ea00071f7f90
[  541.524006] RBP: ffff880188c7fec0 R08: 0000000000000000 R09:
ffff88021ffebfb8
[  541.524006] R10: 0000000000000015 R11: 0000000000000002 R12:
0000000000000001
[  541.524006] R13: ffff88021694bd98 R14: 0000000000000000 R15:
ffffc90065540000
[  541.524006] FS:  00007f7939dbd760(0000) GS:ffff8800ddc00000(0000)
knlGS:0000000000000000
[  541.524006] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  541.524006] CR2: 00007f6ee3825a70 CR3: 00000001dfd83000 CR4:
00000000000006f0
[  541.524006] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  541.524006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[  541.524006] Process kworker/0:1 (pid: 11, threadinfo ffff88021694a000,
task ffff8802169482c0)
[  541.524006] Stack:
[  541.524006]  ffffc90065540000 ffffffff8112437a ffff8802151d2000
ffff88021694bdc0
[  541.524006]  ffffc9006553d000 ffffffffa03a6c74 ffffc9006553a018
ffffc90065540018
[  541.524006]  ffffc900654fb010 ffffc9006553d010 ffff8802151d1088
ffff8802151d3068
[  541.524006] Call Trace:
[  541.524006]  [] __vunmap+0x7a/0xf0
[  541.524006]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[  541.524006]  [] spl_cache_age+0x15/0x50 [spl]
[  541.524006]  [] process_one_work+0x105/0x410
[  541.524006]  [] worker_thread+0x167/0x330
[  541.524006]  [] kthread+0x96/0xa0
[  541.524006]  [] kernel_thread_helper+0x4/0x10
[  541.524006] Code: 0f 1f 80 00 00 00 00 48 83 ec 08 48 8b 07 31 f6 f6 c4
80 74 03 8b 77 68 48 83 c4 08 e9 96 fa ff ff 66 0f 1f 44 00 00 48 83 ec 08
<8b> 47 08 f0 ff 4f 08 0f 94 c0 84 c0 74 1e 85 f6 74 0a 48 83 c4 
[  541.524006] RIP  [] __free_pages+0x4/0x40
[  541.524006]  RSP 
[  542.153086] ---[ end trace 838755f44c5fb1e6 ]---
[  542.162620] BUG: unable to handle kernel 



Message from syslogd@h1906375 at Dec  6 14:03:41 ...
 kernel:[ 1027.136206] general protection fault: 0000 [#1] SMP 

Message from syslogd@h1906375 at Dec  6 14:03:41 ...
 kernel:[ 1027.140009] last sysfs file:
/sys/devices/virtual/net/virbr0/bridge/topology_change_detected

node0 login: [  831.936265] general protection fault: 0000 [#1] SMP 
[  831.940006] last sysfs file: /sys/devices/system/node/node0/meminfo
[  831.940006] CPU 3 
[  831.940006] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet microcode cpufreq_conservative
cpufreq_userspace cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic edac_core kvm_amd
edac_mce_amd sg kvm i2c_piix4 k10temp pata_atiixp serio_raw e1000e
processor thermal_sys shpchp pcspkr pci_hotplug button raid456
async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx
raid10 raid1 raid0 ohci_hcd ehci_hcd radeon ttm drm_kms_helper ssb mmc_core
usbcore pcmcia pcmcia_core drm i2c_algo_bit
[  831.940006] 
[  831.940006] Pid: 58, comm: kworker/3:1 Tainted: P           
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[  831.940006] RIP: 0010:[]  []
__free_pages+0x4/0x40
[  831.940006] RSP: 0018:ffff88021026bd80  EFLAGS: 00010282
[  831.940006] RAX: ffff880203019c00 RBX: 0000000000000040 RCX:
ffffea0006f389b0
[  831.940006] RDX: 000000000000003f RSI: 0000000000000000 RDI:
0000ea0004b01ee0
[  831.940006] RBP: ffff8801fc3e4a00 R08: 0000000000000000 R09:
ffff88021ffebfa0
[  831.940006] R10: 000000000000002d R11: 0000000000000001 R12:
0000000000000001
[  831.940006] R13: ffff88021026bd98 R14: 0000000000000000 R15:
ffffc9001c648000
[  831.940006] FS:  00007f5e044a3da0(0000) GS:ffff8800ddd80000(0000)
knlGS:0000000000000000
[  831.940006] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  831.940006] CR2: 00007f49f84758e0 CR3: 0000000207ad8000 CR4:
00000000000006e0
[  831.940006] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  831.940006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[  831.940006] Process kworker/3:1 (pid: 58, threadinfo ffff88021026a000,
task ffff8802102686c0)
[  831.940006] Stack:
[  831.940006]  ffffc9001c648000 ffffffff8112437a ffff8802147b6000
ffff88021026bdc0
[  831.940006]  ffff88021026bdb0 ffffffffa035dc74 ffffc9001c60c018
ffffc9001c648018
[  831.940006]  ffffc9001c609010 ffffc9001c609010 ffff8802147b5088
ffff8802147b7068
[  831.940006] Call Trace:
[  831.940006]  [] __vunmap+0x7a/0xf0
[  831.940006]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[  831.940006]  [] spl_cache_age+0x15/0x50 [spl]
[  831.940006]  [] process_one_work+0x105/0x410
[  831.940006]  [] worker_thread+0x167/0x330
[  831.940006]  [] kthread+0x96/0xa0
[  831.940006]  [] kernel_thread_helper+0x4/0x10
[  831.940006] Code: 0f 1f 80 00 00 00 00 48 83 ec 08 48 8b 07 31 f6 f6 c4
80 74 03 8b 77 68 48 83 c4 08 e9 96 fa ff ff 66 0f 1f 44 00 00 48 83 ec 08
<8b> 47 08 f0 ff 4f 08 0f 94 c0 84 c0 74 1e 85 f6 74 0a 48 83 c4 
[  831.940006] RIP  [] __free_pages+0x4/0x40
[  831.940006]  RSP 
[  832.557762] ---[ end trace 4cfb6827b75c1d13 ]---
[  832.567170] BUG: unable to handle kernel paging request at
fffffffffffffff8
[  832.572028] IP: [] kthread_data+0x7/0x10
[  832.572028] PGD 1a05067 PUD 1a06067 PMD 0 
[  832.572028] Oops: 0000 [#2] SMP 
[  832.572028] last sysfs file: /sys/devices/system/node/node0/meminfo
[  832.572028] CPU 3 
[  832.572028] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet microcode cpufreq_conservative
cpufreq_userspace cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic edac_core kvm_amd
edac_mce_amd sg kvm i2c_piix4 k10temp pata_atiixp serio_raw e1000e
processor thermal_sys shpchp pcspkr pci_hotplug button raid456
async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx
raid10 raid1 raid0 ohci_hcd ehci_hcd radeon ttm drm_kms_helper ssb mmc_core
usbcore pcmcia pcmcia_core drm i2c_algo_bit
[  832.572028] 
[  832.572028] Pid: 58, comm: kworker/3:1 Tainted: P      D    
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[  832.572028] RIP: 0010:[]  []
kthread_data+0x7/0x10
[  832.572028] RSP: 0018:ffff88021026bba0  EFLAGS: 00010002
[  832.572028] RAX: 0000000000000000 RBX: 0000000000000003 RCX:
ffff8802102686c0
[  832.572028] RDX: 000000000000778a RSI: 0000000000000003 RDI:
ffff8802102686c0
[  832.572028] RBP: ffff88021026bc48 R08: ffff88021f9d8548 R09:
dead000000200200
[  832.572028] R10: dead000000200200 R11: 0000000000000000 R12:
0000000000000003
[  832.572028] R13: ffff880210268a90 R14: 0000000000000000 R15:
00000000ffffffff
[  832.572028] FS:  00007f5e044a3da0(0000) GS:ffff8800ddd80000(0000)
knlGS:0000000000000000
[  832.572028] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  832.572028] CR2: fffffffffffffff8 CR3: 0000000207ad8000 CR4:
00000000000006e0
[  832.572028] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  832.572028] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[  832.572028] Process kworker/3:1 (pid: 58, threadinfo ffff88021026a000,
task ffff8802102686c0)
[  832.572028] Stack:
[  832.572028]  ffffffff81072eb8 ffff8800ddd92500 ffffffff814bb10a
0000000000012500
[  832.572028]  ffff88021026bfd8 0000000000012500 ffff88021026a000
0000000000012500
[  832.572028]  ffff880210268a90 0000000000012500 0000000000012500
ffff88021026bfd8
[  832.572028] Call Trace:
[  832.572028]  [] wq_worker_sleeping+0x8/0x90
[  832.572028]  [] thread_return+0x232/0x398
[  832.572028]  [] do_exit+0x235/0x400
[  832.572028]  [] oops_end+0x9c/0xe0
[  832.572028]  [] general_protection+0x1f/0x30
[  832.572028]  [] __free_pages+0x4/0x40
[  832.572028]  [] __vunmap+0x7a/0xf0
[  832.572028]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[  832.572028]  [] spl_cache_age+0x15/0x50 [spl]
[  832.572028]  [] process_one_work+0x105/0x410
[  832.572028]  [] worker_thread+0x167/0x330
[  832.572028]  [] kthread+0x96/0xa0
[  832.572028]  [] kernel_thread_helper+0x4/0x10
[  832.572028] Code: 48 c7 c6 20 cb 60 81 e8 88 cb fc ff 48 8b 44 24 10 48
81 c4 b8 00 00 00 5b 5d c3 66 0f 1f 84 00 00 00 00 00 48 8b 87 78 03 00 00
<48> 8b 40 f8 c3 0f 1f 40 00 41 55 48 c7 c6 ff 4f 79 81 41 54 65 
[  832.572028] RIP  [] kthread_data+0x7/0x10
[  832.572028]  RSP 
[  832.572028] CR2: fffffffffffffff8
[  832.572028] ---[ end trace 4cfb6827b75c1d14 ]---
[  832.572028] Fixing recursive fault but reboot is needed!




Message from syslogd@node0 at Dec  6 15:06:38 ...
 kernel:[  572.901627] ------------[ cut here ]------------

Message from syslogd@node0 at Dec  6 15:06:38 ...
 kernel:[  572.904099] invalid opcode: 0000 [#1] SMP 

Message from syslogd@node0 at Dec  6 15:06:38 ...
 kernel:[  572.904099] last sysfs file: /sys/devices/virtual/block/md0/dev

Message from syslogd@node0 at Dec  6 15:06:38 ...
 kernel:[  572.904099] Stack:

Message from syslogd@node0 at Dec  6 15:06:38 ...
 kernel:[  572.904099] Call Trace:

Message from syslogd@node0 at Dec  6 15:06:38 ...
 kernel:[  572.904099] Code: 90 c7 85 b4 01 00 00 00 00 00 00 f0 80 4d 28
02 f0 80 4d 28 04 48 8b bd 40 01 00 00 bb 01 00 00 00 e8 6b ac ff ff e9 6c
ff ff ff <0f> 0b 0f 1f 40 00 48 83 ec 18 48 c7 c0 ea ff ff ff 48 89 1c 24 


---------------------------------------------------------------------------------------------------

Welcome to openSUSE 11.4 "Celadon" - Kernel 2.6.37.6-0.7-default (ttyS0).


h1906375 login: [  556.176256] general protection fault: 0000 [#1] SMP 
[  556.180019] last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[  556.180019] CPU 0 
[  556.180019] Modules linked in: tun ip6t_LOG xt_pkttype xt_physdev
xt_TCPMSS ipt_LOG xt_limit ipt_MASQUERADE iptable_nat nf_nat xt_state
xt_tcpudp bridge stp llc sha1_generic drbd lru_cache ib_iser rdma_cm ib_cm
iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet microcode cpufreq_conservative
cpufreq_userspace cpufreq_powersave powernow_k8 mperf edd ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT
iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns
nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack
ip6table_filter ip6_tables x_tables dm_mod zfs(P) zcommon(P) znvpair(P)
zavl(P) zunicode(P) spl zlib_deflate ata_generic processor k10temp sg
i2c_piix4 serio_raw pata_atiixp edac_core pcspkr e1000e edac_mce_amd
thermal_sys shpchp kvm_amd pci_hotplug kvm button raid456 async_raid6_recov
async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0
ohci_hcd ehci_hcd radeon ttm usbcore drm_kms_helper ssb mmc_core pcmcia
pcmcia_core drm i2c_algo_bit
[  556.180019] 
[  556.180019] Pid: 11, comm: kworker/0:1 Tainted: P           
2.6.37.6-0.7-default #1 MICRO-STAR INTERNATIONAL CO., LTD MS-96B3/MS-96B3
[  556.180019] RIP: 0010:[]  []
__free_pages+0x4/0x40
[  556.180019] RSP: 0018:ffff88021694bd80  EFLAGS: 00010282
[  556.180019] RAX: ffff88017e2f7200 RBX: 0000000000000040 RCX:
ffffea0006ddc818
[  556.180019] RDX: 000000000000003f RSI: 0000000000000000 RDI:
0000ea0006ddc780
[  556.180019] RBP: ffff880215dc47c0 R08: 0000000000000000 R09:
ffff88021ffebf70
[  556.180019] R10: 0000000000000093 R11: 0000000000000001 R12:
0000000000000001
[  556.180019] R13: ffff88021694bd98 R14: 0000000000000000 R15:
ffffc90015073000
[  556.180019] FS:  00007f68143f4700(0000) GS:ffff8800ddc00000(0000)
knlGS:0000000000000000
[  556.180019] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  556.180019] CR2: 00007f183794b020 CR3: 0000000214f98000 CR4:
00000000000006f0
[  556.180019] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  556.180019] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[  556.180019] Process kworker/0:1 (pid: 11, threadinfo ffff88021694a000,
task ffff8802169482c0)
[  556.180019] Stack:
[  556.180019]  ffffc90015073000 ffffffff8112437a ffff88020f1bc000
ffff88021694bdc0
[  556.180019]  ffff88021694bdb0 ffffffffa0397c74 ffffc9001503d018
ffffc90015073018
[  556.180019]  ffffc90015034010 ffffc90015034010 ffff880213ff1088
ffff88020f1bd068
[  556.180019] Call Trace:
[  556.180019]  [] __vunmap+0x7a/0xf0
[  556.180019]  [] spl_slab_reclaim+0x1f4/0x270 [spl]
[  556.180019]  [] spl_cache_age+0x15/0x50 [spl]
[  556.180019]  [] process_one_work+0x105/0x410
[  556.180019]  [] worker_thread+0x167/0x330
[  556.180019]  [] kthread+0x96/0xa0
[  556.180019]  [] kernel_thread_helper+0x4/0x10
[  556.180019] Code: 0f 1f 80 00 00 00 00 48 83 ec 08 48 8b 07 31 f6 f6 c4
80 74 03 8b 77 68 48 83 c4 08 e9 96 fa ff ff 66 0f 1f 44 00 00 48 83 ec 08
<8b> 47 08 f0 ff 4f 08 0f 94 c0 84 c0 74 1e 85 f6 74 0a 48 83 c4 
[  556.772321] RIP  [] __free_pages+0x4/0x40
[  556.772321]  RSP 
[  556.808732] ---[ end trace 8ff79c1d8e827a54 ]---
[  556.818245] BUG: unable to handle kernel 

--------------------------------------------------------------------------------------------


DRBD Config

---------------------------------------------------------------------------
node1 :~ # cat /etc/drbd.d/zfsdrbd.res 
resource zfsdrbd {
    protocol    C;

    handlers {
        fence-peer  /usr/lib/drbd/crm-fence-peer.sh;
        split-brain "/usr/lib/drbd/notify-split-brain.sh root";
        after-resync-target /usr/lib/drbd/crm-unfence-peer.sh;
    }

    startup {
        become-primary-on   both;
    }

    net {
        allow-two-primaries;
    }

    disk {
        fencing resource-and-stonith;
    }

    on node0  {
        device      /dev/drbd0;
        disk        /dev/zd0;
        flexible-meta-disk  internal;
        address     10.31.148.7:7788;
    }
    on node1  {
        device      /dev/drbd0;
        disk        /dev/zd0;
        flexible-meta-disk  internal;
        address     10.31.148.9:7788;
    }
}
@behlendorf
Copy link
Contributor

I haven't seen this one reported before, thanks for filing it. We'll take a look time permitting.

@behlendorf
Copy link
Contributor

I believe this will be fixed in the latest master source, what will soon be -rc11. If you get a chance can you verify this.

@ghost ghost mentioned this issue Nov 21, 2013
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@behlendorf @BerndHelber