raidz1: device IO failure when zfs filesystem is full

[michaelletzgus]

I'm still stress testing zfs with tools like bonnie, iozone, dbench and some self made beastiness on some small (~100 MB) loopback devices. ;-)

I think I've found a problem regarding the device management in raidz1 mode: If the file system is full "zpool status -v" reports files errors in metadata:

root@heros:~/test# zpool status -v
pool: pool1
state: ONLINE
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see: http://zfsonlinux.org/msg/ZFS-8000-HC
scan: scrub repaired 0 in 0h0m with 0 errors on Sun May 13 13:16:16 2012
config:

    NAME        STATE     READ WRITE CKSUM
    pool1       ONLINE       0    18     0
      raidz1-0  ONLINE       0     0     0
        loop1   ONLINE       0     0     0
        loop2   ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

    <metadata>:<0x0>
    <metadata>:<0x100>
    <metadata>:<0x10e>
    <metadata>:<0x1e>
    <metadata>:<0x44>
    <metadata>:<0x4c>
    <metadata>:<0x56>
    <metadata>:<0x6a>
    <metadata>:<0xb4>
    <metadata>:<0xbc>
    <metadata>:<0xfb>
    pool1:<0xac>

The pool ends up in this irreversible state of blocked IO.

So far, this happens in two- and three-device raidz1-configuration, but not in one-, two- or three-device mirrors or pools.

The error occurs with and without permanent (one started scrub per second) scrubbig.

[michaelletzgus]

Maybe this is a special issue with loopback devices in general.

I tried to reproduce the problem with "real" devices in a vbox-VM (different kernel, 2.6.32), but no problems so far. Only the raidz1 based on loopback devices won't work, it killed the whole VM:

May 13 16:00:55 debian kernel: [ 1016.353645] BUG: unable to handle kernel NULL pointer dereference at (null)
May 13 16:00:55 debian kernel: [ 1016.355117] IP: [] remove_vm_area+0x42/0x77
May 13 16:00:55 debian kernel: [ 1016.356214] PGD 363a5067 PUD 363a0067 PMD 0
May 13 16:00:55 debian kernel: [ 1016.357088] Oops: 0000 [#1] SMP

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.357088] Oops: 0000 [#1] SMP

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.357088] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.357088] Stack:

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.357088] Call Trace:

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.357088] Code: 74 54 48 8b 58 50 f6 43 18 20 75 38 48 c7 c7 50 be 49 81 e8 0a 5a 22 00 48 8b 05 65 18 5d 00 48 c7 c2 50 8c 6a 81 eb 06 48 89 c2 <48> 8b 00 48 39 d8 75 f5 48 8b 03 48 89 02 f0 81 05 40 4a 3c 00

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.357088] CR2: 0000000000000000

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.440834] Oops: 0000 [#2] SMP

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.440834] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.440834] Stack:

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.440834] Call Trace:

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.440834] Code: fa 66 0f 1f 44 00 00 65 8b 04 25 a8 e3 00 00 48 98 49 8b 94 c4 f0 02 00 00 8b 4a 18 89 4c 24 14 48 8b 1a 48 85 db 74 0c 8b 42 14 <48> 8b 04 c3 48 89 02 eb 19 48 8b 4c 24 08 49 89 d0 44 89 ee 83

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.440834] CR2: 0000000000000001
May 13 16:00:55 debian kernel: [ 1016.357088] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler
May 13 16:00:55 debian kernel: [ 1016.357088] CPU 1
May 13 16:00:55 debian kernel: [ 1016.357088] Modules linked in: loop zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm parport_pc snd_timer parport psmouse serio_raw pcspkr i2c_piix4 evdev snd soundcore ac joydev button processor i2c_core snd_page_alloc vboxguest ext3 jbd mbcache usbhid hid sd_mod crc_t10dif ohci_hcd ahci libata thermal thermal_sys e1000 ehci_hcd scsi_mod usbcore nls_base [last unloaded: scsi_wait_scan]
May 13 16:00:55 debian kernel: [ 1016.357088] Pid: 15736, comm: zpool Tainted: P 2.6.32-5-amd64 #1 VirtualBox
May 13 16:00:55 debian kernel: [ 1016.357088] RIP: 0010:[] [] remove_vm_area+0x42/0x77
May 13 16:00:55 debian kernel: [ 1016.357088] RSP: 0018:ffff88003639f9c8 EFLAGS: 00010203
May 13 16:00:55 debian kernel: [ 1016.357088] RAX: 0000000000000000 RBX: ffff880033e3a6c0 RCX: 0000000000400000
May 13 16:00:55 debian kernel: [ 1016.357088] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff8149be50
May 13 16:00:55 debian kernel: [ 1016.357088] RBP: ffff880035697a00 R08: ffffc9001e3f2000 R09: 0000000000011ac0
May 13 16:00:55 debian kernel: [ 1016.357088] R10: 0000000000020000 R11: ffffffffa0368779 R12: 0000000000000001
May 13 16:00:55 debian kernel: [ 1016.357088] R13: 0000000000000000 R14: ffff88003639fa28 R15: 0000000000000000
May 13 16:00:55 debian kernel: [ 1016.357088] FS: 00007f19f37bfbe0(0000) GS:ffff880001900000(0000) knlGS:0000000000000000
May 13 16:00:55 debian kernel: [ 1016.357088] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 13 16:00:55 debian kernel: [ 1016.357088] CR2: 0000000000000000 CR3: 00000000363a2000 CR4: 00000000000006e0
May 13 16:00:55 debian kernel: [ 1016.357088] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 13 16:00:55 debian kernel: [ 1016.357088] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 13 16:00:55 debian kernel: [ 1016.357088] Process zpool (pid: 15736, threadinfo ffff88003639e000, task ffff88003932cdb0)
May 13 16:00:55 debian kernel: [ 1016.357088] Stack:
May 13 16:00:55 debian kernel: [ 1016.357088] 0000000000000000 ffffc9001e3f2000 ffff88003dcbc000 ffffffff810d7507
May 13 16:00:55 debian kernel: [ 1016.357088] <0> ffff88003639fa28 ffff88003dcbc000 ffff88003639fa38 ffffffffa02413ec
May 13 16:00:55 debian kernel: [ 1016.357088] <0> 0000000000000000 0000000000000000 0000000100000000 ffff88003dcbd0f8
May 13 16:00:55 debian kernel: [ 1016.357088] Call Trace:
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? __vunmap+0x31/0xac
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? spl_slab_reclaim+0x2de/0x347 [spl]
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? spl_kmem_cache_reap_now+0x114/0x175 [spl]
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? arc_kmem_reap_now+0x54/0x93 [zfs]
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? arc_shrinker_func+0xd1/0x1c0 [zfs]
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? shrink_slab+0xe0/0x153
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? try_to_free_pages+0x26a/0x38e
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? isolate_pages_global+0x0/0x20f
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? congestion_wait+0x74/0x80
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? autoremove_wake_function+0x0/0x2e
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? __alloc_pages_nodemask+0x3d4/0x5fc
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? generic_file_aio_read+0x413/0x536
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? do_sync_read+0xce/0x113
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? autoremove_wake_function+0x0/0x2e
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? mmap_region+0x3b5/0x4f3
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? vfs_read+0xa6/0xff
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? sys_read+0x45/0x6e
May 13 16:00:55 debian kernel: [ 1016.357088] [] ? system_call_fastpath+0x16/0x1b
May 13 16:00:55 debian kernel: [ 1016.357088] Code: 74 54 48 8b 58 50 f6 43 18 20 75 38 48 c7 c7 50 be 49 81 e8 0a 5a 22 00 48 8b 05 65 18 5d 00 48 c7 c2 50 8c 6a 81 eb 06 48 89 c2 <48> 8b 00 48 39 d8 75 f5 48 8b 03 48 89 02 f0 81 05 40 4a 3c 00
May 13 16:00:55 debian kernel: [ 1016.357088] RIP [] remove_vm_area+0x42/0x77
May 13 16:00:55 debian kernel: [ 1016.357088] RSP
May 13 16:00:55 debian kernel: [ 1016.357088] CR2: 0000000000000000
May 13 16:00:55 debian kernel: [ 1016.356035] BUG: unable to handle kernel
May 13 16:00:55 debian kernel: [ 1016.439532] ---[ end trace 1bf0d7ed3d5d7628 ]---
May 13 16:00:55 debian kernel: [ 1016.440834] NULL pointer dereference at 0000000000000001
May 13 16:00:55 debian kernel: [ 1016.440834] IP: [] __kmalloc+0xd2/0x141
May 13 16:00:55 debian kernel: [ 1016.440834] PGD 2a057067 PUD 21e9f067 PMD 0
May 13 16:00:55 debian kernel: [ 1016.440834] Oops: 0000 [#2] SMP
May 13 16:00:55 debian kernel: [ 1016.440834] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler
May 13 16:00:55 debian kernel: [ 1016.440834] CPU 0
May 13 16:00:55 debian kernel: [ 1016.440834] Modules linked in: loop zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm parport_pc snd_timer parport psmouse serio_raw pcspkr i2c_piix4 evdev snd soundcore ac joydev button processor i2c_core snd_page_alloc vboxguest ext3 jbd mbcache usbhid hid sd_mod crc_t10dif ohci_hcd ahci libata thermal thermal_sys e1000 ehci_hcd scsi_mod usbcore nls_base [last unloaded: scsi_wait_scan]
May 13 16:00:55 debian kernel: [ 1016.440834] Pid: 6504, comm: z_wr_iss_h/0 Tainted: P D 2.6.32-5-amd64 #1 VirtualBox
May 13 16:00:55 debian kernel: [ 1016.440834] RIP: 0010:[] [] __kmalloc+0xd2/0x141
May 13 16:00:55 debian kernel: [ 1016.440834] RSP: 0018:ffff880032e55c90 EFLAGS: 00010002
May 13 16:00:55 debian kernel: [ 1016.440834] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000040
May 13 16:00:55 debian kernel: [ 1016.440834] RDX: ffff880001811d90 RSI: 00000000000080d0 RDI: ffffffffa0242f10
May 13 16:00:55 debian kernel: [ 1016.440834] RBP: 0000000000000246 R08: 0000000000000000 R09: 0000000000000000
May 13 16:00:55 debian kernel: [ 1016.440834] R10: ffffc90007d65000 R11: ffffffffa033c395 R12: ffffffff8147f7e0
May 13 16:00:55 debian kernel: [ 1016.440834] R13: 00000000000080d0 R14: 00000000000080d0 R15: 0000000000000028
May 13 16:00:55 debian kernel: [ 1016.440834] FS: 0000000000000000(0000) GS:ffff880001800000(0000) knlGS:0000000000000000
May 13 16:00:55 debian kernel: [ 1016.440834] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
May 13 16:00:55 debian kernel: [ 1016.440834] CR2: 0000000000000001 CR3: 000000003ae94000 CR4: 00000000000006f0
May 13 16:00:55 debian kernel: [ 1016.440834] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 13 16:00:55 debian kernel: [ 1016.440834] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 13 16:00:55 debian kernel: [ 1016.440834] Process z_wr_iss_h/0 (pid: 6504, threadinfo ffff880032e54000, task ffff88003fa162e0)
May 13 16:00:55 debian kernel: [ 1016.440834] Stack:
May 13 16:00:55 debian kernel: [ 1016.440834] ffffffff814bb420 ffffffffa0242f10 0000004024eaeae0 0000000000000000
May 13 16:00:55 debian kernel: [ 1016.440834] <0> 00000000000080d0 0000000000000028 0000000000000010 00000000000080d0
May 13 16:00:55 debian kernel: [ 1016.440834] <0> 0000000000000052 ffffffffa0242f10 0000000000010000 ffff880024eaeae0
May 13 16:00:55 debian kernel: [ 1016.440834] Call Trace:
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? kmem_alloc_debug+0x123/0x2b6 [spl]
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? kmem_alloc_debug+0x123/0x2b6 [spl]
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? mutex_lock+0xd/0x31
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? vdev_mirror_io_start+0x7b/0x303 [zfs]
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? zio_vdev_io_start+0x3f/0x280 [zfs]
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? zio_execute+0xf7/0x121 [zfs]
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? taskq_thread+0x2b2/0x4f2 [spl]
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? try_to_wake_up+0x289/0x29b
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? default_wake_function+0x0/0x9
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? taskq_thread+0x0/0x4f2 [spl]
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? kthread+0x79/0x81
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? child_rip+0xa/0x20
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? kthread+0x0/0x81
May 13 16:00:55 debian kernel: [ 1016.440834] [] ? child_rip+0x0/0x20
May 13 16:00:55 debian kernel: [ 1016.440834] Code: fa 66 0f 1f 44 00 00 65 8b 04 25 a8 e3 00 00 48 98 49 8b 94 c4 f0 02 00 00 8b 4a 18 89 4c 24 14 48 8b 1a 48 85 db 74 0c 8b 42 14 <48> 8b 04 c3 48 89 02 eb 19 48 8b 4c 24 08 49 89 d0 44 89 ee 83
May 13 16:00:55 debian kernel: [ 1016.440834] RIP [] __kmalloc+0xd2/0x141
May 13 16:00:55 debian kernel: [ 1016.440834] RSP
May 13 16:00:55 debian kernel: [ 1016.440834] CR2: 0000000000000001
May 13 16:00:55 debian kernel: [ 1016.440834] ---[ end trace 1bf0d7ed3d5d7629 ]---
May 13 16:00:55 debian kernel: [ 1016.719834] BUG: unable to handle kernel paging request at ffff88003fff0000
May 13 16:00:55 debian kernel: [ 1016.719841] IP: [] vdev_mirror_io_done+0x2f/0x2bd [zfs]
May 13 16:00:55 debian kernel: [ 1016.719872] PGD 1002063 PUD 1006063 PMD 8067 PTE 0
May 13 16:00:55 debian kernel: [ 1016.719879] Oops: 0000 [#3] SMP

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.719879] Oops: 0000 [#3] SMP
May 13 16:00:55 debian kernel: [ 1016.719884] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.719884] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler
May 13 16:00:55 debian kernel: [ 1016.719890] CPU 0
May 13 16:00:55 debian kernel: [ 1016.719895] Modules linked in: loop zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm parport_pc snd_timer parport psmouse serio_raw pcspkr i2c_piix4 evdev snd soundcore ac joydev button processor i2c_core snd_page_alloc vboxguest ext3 jbd mbcache usbhid hid sd_mod crc_t10dif ohci_hcd ahci libata thermal thermal_sys e1000 ehci_hcd scsi_mod usbcore nls_base [last unloaded: scsi_wait_scan]
May 13 16:00:55 debian kernel: [ 1016.719917] Pid: 6529, comm: z_wr_int_h/4 Tainted: P D 2.6.32-5-amd64 #1 VirtualBox
May 13 16:00:55 debian kernel: [ 1016.719922] RIP: 0010:[] [] vdev_mirror_io_done+0x2f/0x2bd [zfs]
May 13 16:00:55 debian kernel: [ 1016.719941] RSP: 0018:ffff880032dc9bd0 EFLAGS: 00010286
May 13 16:00:55 debian kernel: [ 1016.719946] RAX: ffff88003ffeffe0 RBX: ffff88003e2fb2a0 RCX: 00000000006b8c4c
May 13 16:00:55 debian kernel: [ 1016.719951] RDX: ffff88003ffeffe8 RSI: 000000000000d07f RDI: 0000000035e9db80
May 13 16:00:55 debian kernel: [ 1016.719956] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff81481dc0
May 13 16:00:55 debian kernel: [ 1016.719969] R10: ffff88003a568800 R11: ffff88002ca40d50 R12: ffff88003e2fb2a0
May 13 16:00:55 debian kernel: [ 1016.719974] R13: ffff880035e9d8c0 R14: 0000000000160574 R15: ffff880038e4c178
May 13 16:00:55 debian kernel: [ 1016.719983] FS: 0000000000000000(0000) GS:ffff880001800000(0000) knlGS:0000000000000000
May 13 16:00:55 debian kernel: [ 1016.719989] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
May 13 16:00:55 debian kernel: [ 1016.719994] CR2: ffff88003fff0000 CR3: 0000000038d97000 CR4: 00000000000006f0
May 13 16:00:55 debian kernel: [ 1016.720003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 13 16:00:55 debian kernel: [ 1016.720005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 13 16:00:55 debian kernel: [ 1016.720018] Process z_wr_int_h/4 (pid: 6529, threadinfo ffff880032dc8000, task ffff88003938dbd0)
May 13 16:00:55 debian kernel: [ 1016.720019] Stack:

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.720019] Stack:
May 13 16:00:55 debian kernel: [ 1016.720020] ffffffff81192fb6 ffffffff810e60a1 ffff88002fa86a28 ffffffff812fc3a6
May 13 16:00:55 debian kernel: [ 1016.720023] <0> 0000000000000028 ffffffff812fc3a6 ffff88003e2fb2a0 ffffffffa036859a
May 13 16:00:55 debian kernel: [ 1016.720025] <0> ffff88003e2fb2a0 0000000000000000 0000000000000000 ffffffffa0392590
May 13 16:00:55 debian kernel: [ 1016.720027] Call Trace:

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.720027] Call Trace:
May 13 16:00:55 debian kernel: [ 1016.720031] [] ? rb_erase+0x1b2/0x279
May 13 16:00:55 debian kernel: [ 1016.720035] [] ? virt_to_head_page+0x9/0x2a
May 13 16:00:55 debian kernel: [ 1016.720038] [] ? mutex_lock+0xd/0x31
May 13 16:00:55 debian kernel: [ 1016.720039] [] ? mutex_lock+0xd/0x31
May 13 16:00:55 debian kernel: [ 1016.720051] [] ? zio_wait_for_children+0x4f/0x59 [zfs]
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? zio_vdev_io_done+0xec/0x138 [zfs]
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? zio_done+0xa10/0xa9d [zfs]
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? kmem_free_debug+0xc5/0x10d [spl]
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? zio_done+0xa10/0xa9d [zfs]
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? mutex_lock+0xd/0x31
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? zio_execute+0xf7/0x121 [zfs]
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? taskq_thread+0x2b2/0x4f2 [spl]
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? try_to_wake_up+0x289/0x29b
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? default_wake_function+0x0/0x9
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? taskq_thread+0x0/0x4f2 [spl]
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? kthread+0x79/0x81
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? child_rip+0xa/0x20
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? kthread+0x0/0x81
May 13 16:00:55 debian kernel: [ 1016.720058] [] ? child_rip+0x0/0x20
May 13 16:00:55 debian kernel: [ 1016.720058] Code: 31 f6 45 31 f6 41 55 41 54 49 89 fc 55 53 48 83 ec 40 4c 8b af 00 02 00 00 41 8b 7d 00 eb 29 48 63 c1 48 6b c0 18 49 8d 44 05 00 <83> 78 20 00 48 8d 50 20 74 0a 80 7a 05 01 41 83 d6 00 eb 07 80

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.720058] Code: 31 f6 45 31 f6 41 55 41 54 49 89 fc 55 53 48 83 ec 40 4c 8b af 00 02 00 00 41 8b 7d 00 eb 29 48 63 c1 48 6b c0 18 49 8d 44 05 00 <83> 78 20 00 48 8d 50 20 74 0a 80 7a 05 01 41 83 d6 00 eb 07 80
May 13 16:00:55 debian kernel: [ 1016.720058] RIP [] vdev_mirror_io_done+0x2f/0x2bd [zfs]
May 13 16:00:55 debian kernel: [ 1016.720058] RSP
May 13 16:00:55 debian kernel: [ 1016.720058] CR2: ffff88003fff0000

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.720058] CR2: ffff88003fff0000
May 13 16:00:55 debian kernel: [ 1016.720058] ---[ end trace 1bf0d7ed3d5d762a ]---
May 13 16:00:55 debian kernel: [ 1016.732533] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
May 13 16:00:55 debian kernel: [ 1016.732537] IP: [] kmem_cache_alloc+0x61/0xf0
May 13 16:00:55 debian kernel: [ 1016.732548] PGD 18019067 PUD 36366067 PMD 0
May 13 16:00:55 debian kernel: [ 1016.732550] Oops: 0000 [#4] SMP

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.732550] Oops: 0000 [#4] SMP
May 13 16:00:55 debian kernel: [ 1016.732552] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.732552] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler
May 13 16:00:55 debian kernel: [ 1016.732554] CPU 0
May 13 16:00:55 debian kernel: [ 1016.732555] Modules linked in: loop zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm parport_pc snd_timer parport psmouse serio_raw pcspkr i2c_piix4 evdev snd soundcore ac joydev button processor i2c_core snd_page_alloc vboxguest ext3 jbd mbcache usbhid hid sd_mod crc_t10dif ohci_hcd ahci libata thermal thermal_sys e1000 ehci_hcd scsi_mod usbcore nls_base [last unloaded: scsi_wait_scan]
May 13 16:00:55 debian kernel: [ 1016.732574] Pid: 15738, comm: sh Tainted: P D 2.6.32-5-amd64 #1 VirtualBox
May 13 16:00:55 debian kernel: [ 1016.732575] RIP: 0010:[] [] kmem_cache_alloc+0x61/0xf0
May 13 16:00:55 debian kernel: [ 1016.732579] RSP: 0018:ffff8800363ade48 EFLAGS: 00010002
May 13 16:00:55 debian kernel: [ 1016.732580] RAX: 0000000000000000 RBX: 0000000000000246 RCX: ffffffff810403e2
May 13 16:00:55 debian kernel: [ 1016.732581] RDX: ffff880001811d90 RSI: 00000000000000d0 RDI: ffffffff8147f7e0
May 13 16:00:55 debian kernel: [ 1016.732583] RBP: 0000000000000001 R08: 0000000000000007 R09: 00007ffffa910e74
May 13 16:00:55 debian kernel: [ 1016.732590] R10: 0000000000000000 R11: 0000000000000202 R12: 00000000000000d0
May 13 16:00:55 debian kernel: [ 1016.732592] R13: ffffffff8147f7e0 R14: ffffffff81069cdd R15: 0000000000000040
May 13 16:00:55 debian kernel: [ 1016.732597] FS: 00007f927339c700(0000) GS:ffff880001800000(0000) knlGS:0000000000000000
May 13 16:00:55 debian kernel: [ 1016.732599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 13 16:00:55 debian kernel: [ 1016.732600] CR2: 0000000000000001 CR3: 0000000018001000 CR4: 00000000000006f0
May 13 16:00:55 debian kernel: [ 1016.732606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 13 16:00:55 debian kernel: [ 1016.732608] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 13 16:00:55 debian kernel: [ 1016.732609] Process sh (pid: 15738, threadinfo ffff8800363ac000, task ffff88003faee9f0)
May 13 16:00:55 debian kernel: [ 1016.732610] Stack:

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.732610] Stack:
May 13 16:00:55 debian kernel: [ 1016.732611] 000000d0105d7188 ffff88003faeed80 ffff88002cb51b00 ffff88002cb51b00
May 13 16:00:55 debian kernel: [ 1016.732613] <0> ffff88003e6c1000 000000000061b680 0000000001f97528 ffffffff81069cdd
May 13 16:00:55 debian kernel: [ 1016.732615] <0> ffff88003faeed80 ffff88002cb51b00 ffff88002cb51b00 ffffffff810f37ba
May 13 16:00:55 debian kernel: [ 1016.732618] Call Trace:

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.732618] Call Trace:
May 13 16:00:55 debian kernel: [ 1016.732622] [] ? prepare_exec_creds+0x15/0xe1
May 13 16:00:55 debian kernel: [ 1016.732624] [] ? prepare_bprm_creds+0x2f/0x51
May 13 16:00:55 debian kernel: [ 1016.732626] [] ? do_execve+0x9c/0x2c3
May 13 16:00:55 debian kernel: [ 1016.732630] [] ? sys_execve+0x35/0x4c
May 13 16:00:55 debian kernel: [ 1016.732632] [] ? stub_execve+0x6a/0xc0
May 13 16:00:55 debian kernel: [ 1016.732633] Code: 48 89 c3 fa 66 0f 1f 44 00 00 65 8b 04 25 a8 e3 00 00 48 98 49 8b 94 c5 f0 02 00 00 48 8b 2a 44 8b 7a 18 48 85 ed 74 0d 8b 42 14 <48> 8b 44 c5 00 48 89 02 eb 17 49 89 d0 4c 89 f1 83 ca ff 44 89

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.732633] Code: 48 89 c3 fa 66 0f 1f 44 00 00 65 8b 04 25 a8 e3 00 00 48 98 49 8b 94 c5 f0 02 00 00 48 8b 2a 44 8b 7a 18 48 85 ed 74 0d 8b 42 14 <48> 8b 44 c5 00 48 89 02 eb 17 49 89 d0 4c 89 f1 83 ca ff 44 89
May 13 16:00:55 debian kernel: [ 1016.732646] RIP [] kmem_cache_alloc+0x61/0xf0
May 13 16:00:55 debian kernel: [ 1016.732648] RSP
May 13 16:00:55 debian kernel: [ 1016.732649] CR2: 0000000000000001

Message from syslogd@debian at May 13 16:00:55 ...
kernel:[ 1016.732649] CR2: 0000000000000001
May 13 16:00:55 debian kernel: [ 1016.732651] ---[ end trace 1bf0d7ed3d5d762b ]---
May 13 16:00:56 debian kernel: [ 1017.252004] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
May 13 16:00:56 debian kernel: [ 1017.254947] IP: [] kmem_cache_alloc+0x61/0xf0
May 13 16:00:56 debian kernel: [ 1017.255308] PGD 31cfa067 PUD 2c3dc067 PMD 0
May 13 16:00:56 debian kernel: [ 1017.255308] Oops: 0000 [#5] SMP

Message from syslogd@debian at May 13 16:00:56 ...
kernel:[ 1017.255308] Oops: 0000 [#5] SMP
May 13 16:00:56 debian kernel: [ 1017.255308] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler

Message from syslogd@debian at May 13 16:00:56 ...
kernel:[ 1017.255308] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler
May 13 16:00:56 debian kernel: [ 1017.255308] CPU 0
May 13 16:00:56 debian kernel: [ 1017.255308] Modules linked in: loop zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm parport_pc snd_timer parport psmouse serio_raw pcspkr i2c_piix4 evdev snd soundcore ac joydev button processor i2c_core snd_page_alloc vboxguest ext3 jbd mbcache usbhid hid sd_mod crc_t10dif ohci_hcd ahci libata thermal thermal_sys e1000 ehci_hcd scsi_mod usbcore nls_base [last unloaded: scsi_wait_scan]
May 13 16:00:56 debian kernel: [ 1017.255308] Pid: 12027, comm: watch Tainted: P D 2.6.32-5-amd64 #1 VirtualBox
May 13 16:00:56 debian kernel: [ 1017.255308] RIP: 0010:[] [] kmem_cache_alloc+0x61/0xf0
May 13 16:00:56 debian kernel: [ 1017.255308] RSP: 0018:ffff8800279cbd90 EFLAGS: 00010002
May 13 16:00:56 debian kernel: [ 1017.255308] RAX: 0000000000000000 RBX: 0000000000000246 RCX: ffff88003faef810
May 13 16:00:56 debian kernel: [ 1017.255308] RDX: ffff880001811d90 RSI: 00000000000000d0 RDI: ffffffff8147f7e0
May 13 16:00:56 debian kernel: [ 1017.255308] RBP: 0000000000000001 R08: ffff880039d8ab40 R09: ffff88000000a680
May 13 16:00:56 debian kernel: [ 1017.255308] R10: 000000004698b33d R11: ffffffff811537ff R12: 00000000000000d0
May 13 16:00:56 debian kernel: [ 1017.255308] R13: ffffffff8147f7e0 R14: ffffffff81069bf5 R15: 0000000000000040
May 13 16:00:56 debian kernel: [ 1017.255308] FS: 00007ff79b77a700(0000) GS:ffff880001800000(0000) knlGS:0000000000000000
May 13 16:00:56 debian kernel: [ 1017.255308] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 13 16:00:56 debian kernel: [ 1017.255308] CR2: 0000000000000001 CR3: 000000002ceb4000 CR4: 00000000000006f0
May 13 16:00:56 debian kernel: [ 1017.255308] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 13 16:00:56 debian kernel: [ 1017.255308] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 13 16:00:56 debian kernel: [ 1017.255308] Process watch (pid: 12027, threadinfo ffff8800279ca000, task ffff88003faef810)
May 13 16:00:56 debian kernel: [ 1017.255308] Stack:

Message from syslogd@debian at May 13 16:00:56 ...
kernel:[ 1017.255308] Stack:
May 13 16:00:56 debian kernel: [ 1017.255308] 000000d03932d4c0 00000000fffffff4 0000000001200011 ffff880032e5d000
May 13 16:00:56 debian kernel: [ 1017.255308] <0> ffff88003932d4c0 0000000001200011 0000000000000000 ffffffff81069bf5
May 13 16:00:56 debian kernel: [ 1017.255308] <0> ffff88003638e000 ffff88003faef810 ffff88003932d4c0 0000000001200011
May 13 16:00:56 debian kernel: [ 1017.255308] Call Trace:

Message from syslogd@debian at May 13 16:00:56 ...
kernel:[ 1017.255308] Call Trace:
May 13 16:00:56 debian kernel: [ 1017.255308] [] ? copy_creds+0xe1/0x1b4
May 13 16:00:56 debian kernel: [ 1017.255308] [] ? copy_process+0x24d/0x1165
May 13 16:00:56 debian kernel: [ 1017.255308] [] ? do_fork+0x157/0x31e
May 13 16:00:56 debian kernel: [ 1017.255308] [] ? alloc_fd+0x67/0x10c
May 13 16:00:56 debian kernel: [ 1017.255308] [] ? fd_install+0x2e/0x5a
May 13 16:00:56 debian kernel: [ 1017.255308] [] ? stub_clone+0x13/0x20
May 13 16:00:56 debian kernel: [ 1017.255308] [] ? system_call_fastpath+0x16/0x1b
May 13 16:00:56 debian kernel: [ 1017.255308] Code: 48 89 c3 fa 66 0f 1f 44 00 00 65 8b 04 25 a8 e3 00 00 48 98 49 8b 94 c5 f0 02 00 00 48 8b 2a 44 8b 7a 18 48 85 ed 74 0d 8b 42 14 <48> 8b 44 c5 00 48 89 02 eb 17 49 89 d0 4c 89 f1 83 ca ff 44 89

Message from syslogd@debian at May 13 16:00:56 ...
kernel:[ 1017.255308] Code: 48 89 c3 fa 66 0f 1f 44 00 00 65 8b 04 25 a8 e3 00 00 48 98 49 8b 94 c5 f0 02 00 00 48 8b 2a 44 8b 7a 18 48 85 ed 74 0d 8b 42 14 <48> 8b 44 c5 00 48 89 02 eb 17 49 89 d0 4c 89 f1 83 ca ff 44 89
May 13 16:00:56 debian kernel: [ 1017.255308] RIP [] kmem_cache_alloc+0x61/0xf0
May 13 16:00:56 debian kernel: [ 1017.255308] RSP
May 13 16:00:56 debian kernel: [ 1017.255308] CR2: 0000000000000001

Message from syslogd@debian at May 13 16:00:56 ...
kernel:[ 1017.255308] CR2: 0000000000000001
May 13 16:00:56 debian kernel: [ 1017.255308] ---[ end trace 1bf0d7ed3d5d762c ]---
May 13 16:00:58 debian kernel: [ 1019.726736] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
May 13 16:00:58 debian kernel: [ 1019.729066] IP: [] __kmalloc+0xd2/0x141
May 13 16:00:58 debian kernel: [ 1019.729066] PGD 2f89e067 PUD 39345067 PMD 0
May 13 16:00:58 debian kernel: [ 1019.729066] Oops: 0000 [#6] SMP

Message from syslogd@debian at May 13 16:00:58 ...
kernel:[ 1019.729066] Oops: 0000 [#6] SMP
May 13 16:00:58 debian kernel: [ 1019.729066] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler

Message from syslogd@debian at May 13 16:00:58 ...
kernel:[ 1019.729066] last sysfs file: /sys/devices/pci0000:00/0000:00:0d.0/host3/target3:0:0/3:0:0:0/block/sdd/queue/scheduler
May 13 16:00:58 debian kernel: [ 1019.729066] CPU 0
May 13 16:00:58 debian kernel: [ 1019.729066] Modules linked in: loop zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm parport_pc snd_timer parport psmouse serio_raw pcspkr i2c_piix4 evdev snd soundcore ac joydev button processor i2c_core snd_page_alloc vboxguest ext3 jbd mbcache usbhid hid sd_mod crc_t10dif ohci_hcd ahci libata thermal thermal_sys e1000 ehci_hcd scsi_mod usbcore nls_base [last unloaded: scsi_wait_scan]
May 13 16:00:58 debian kernel: [ 1019.729066] Pid: 1085, comm: VBoxService Tainted: P D 2.6.32-5-amd64 #1 VirtualBox
May 13 16:00:58 debian kernel: [ 1019.729066] RIP: 0010:[] [] __kmalloc+0xd2/0x141
May 13 16:00:58 debian kernel: [ 1019.729066] RSP: 0018:ffff880038db79d8 EFLAGS: 00010002
May 13 16:00:58 debian kernel: [ 1019.729066] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000040
May 13 16:00:58 debian kernel: [ 1019.729066] RDX: ffff880001811d90 RSI: 00000000000000d0 RDI: ffffffffa0186855
May 13 16:00:58 debian kernel: [ 1019.729066] RBP: 0000000000000246 R08: ffffffffa0195e20 R09: 00000000ffffffff
May 13 16:00:58 debian kernel: [ 1019.729066] R10: 0000000000000000 R11: ffffffff811537e0 R12: ffffffff8147f7e0
May 13 16:00:58 debian kernel: [ 1019.729066] R13: 00000000000000d0 R14: 00000000000000d0 R15: 0000000000000030
May 13 16:00:58 debian kernel: [ 1019.729066] FS: 00007f2f9b494700(0000) GS:ffff880001800000(0000) knlGS:0000000000000000
May 13 16:00:58 debian kernel: [ 1019.729066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 13 16:00:58 debian kernel: [ 1019.729066] CR2: 0000000000000001 CR3: 0000000038d56000 CR4: 00000000000006f0
May 13 16:00:58 debian kernel: [ 1019.729066] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 13 16:00:58 debian kernel: [ 1019.729066] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 13 16:00:58 debian kernel: [ 1019.729066] Process VBoxService (pid: 1085, threadinfo ffff880038db6000, task ffff88003ad1b170)
May 13 16:00:58 debian kernel: [ 1019.729066] Stack:

Message from syslogd@debian at May 13 16:00:58 ...
kernel:[ 1019.729066] Stack:
May 13 16:00:58 debian kernel: [ 1019.729066] ffff88003d695800 ffffffffa0186855 000000402452f930 0000000000000003
May 13 16:00:58 debian kernel: [ 1019.729066] <0> ffff880036295970 0000000080000000 0000000000000020 ffff880036295980
May 13 16:00:58 debian kernel: [ 1019.729066] <0> ffff880036295980 ffffffffa0186855 0000000000000000 ffff88002452f930
May 13 16:00:58 debian kernel: [ 1019.729066] Call Trace:

Message from syslogd@debian at May 13 16:00:58 ...
kernel:[ 1019.729066] Call Trace:
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? rtR0MemAlloc+0x88/0xb8 [vboxguest]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? rtR0MemAlloc+0x88/0xb8 [vboxguest]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? RTMemAlloc+0xb/0x1d [vboxguest]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? VbglR0HGCMInternalCall+0x254/0xae9 [vboxguest]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? VBoxGuestHGCMAsyncWaitCallbackInterruptible+0x0/0xc [vboxguest]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? generic_make_request+0x299/0x2f9
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? ext3_get_block+0x0/0xf1 [ext3]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? submit_bio+0xd6/0xf2
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? ext3_get_block+0x0/0xf1 [ext3]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? add_to_page_cache_locked+0x76/0xc1
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? mpage_bio_submit+0x22/0x26
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? mpage_readpages+0xfb/0x112
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? ext3_get_block+0x0/0xf1 [ext3]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? futex_wait_queue_me+0xa4/0xbd
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? VBoxGuestCommonIOCtl_HGCMCall+0x1e6/0x27e [vboxguest]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? current_fs_time+0x1e/0x24
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? VBoxGuestCommonIOCtl+0x4b2/0x188d [vboxguest]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? __do_page_cache_readahead+0x123/0x1b4
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? find_get_page+0x1a/0x77
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? common_interrupt+0xe/0x13
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? __kmalloc+0x12f/0x141
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? rtR0MemAlloc+0x88/0xb8 [vboxguest]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? rtR0MemAlloc+0x88/0xb8 [vboxguest]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? vboxguestLinuxIOCtl+0xc1/0x18f [vboxguest]
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? lock_sock_nested+0xa0/0xab
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? virt_to_head_page+0x9/0x2a
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? vfs_ioctl+0x21/0x6c
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? do_vfs_ioctl+0x48d/0x4cb
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? dput+0x2c/0x15e
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? mntput_no_expire+0x23/0xee
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? sys_ioctl+0x51/0x70
May 13 16:00:58 debian kernel: [ 1019.729066] [] ? system_call_fastpath+0x16/0x1b
May 13 16:00:58 debian kernel: [ 1019.729066] Code: fa 66 0f 1f 44 00 00 65 8b 04 25 a8 e3 00 00 48 98 49 8b 94 c4 f0 02 00 00 8b 4a 18 89 4c 24 14 48 8b 1a 48 85 db 74 0c 8b 42 14 <48> 8b 04 c3 48 89 02 eb 19 48 8b 4c 24 08 49 89 d0 44 89 ee 83

Message from syslogd@debian at May 13 16:00:58 ...
kernel:[ 1019.729066] Code: fa 66 0f 1f 44 00 00 65 8b 04 25 a8 e3 00 00 48 98 49 8b 94 c4 f0 02 00 00 8b 4a 18 89 4c 24 14 48 8b 1a 48 85 db 74 0c 8b 42 14 <48> 8b 04 c3 48 89 02 eb 19 48 8b 4c 24 08 49 89 d0 44 89 ee 83
May 13 16:00:58 debian kernel: [ 1019.729066] RIP [] __kmalloc+0xd2/0x141
May 13 16:00:58 debian kernel: [ 1019.729066] RSP
May 13 16:00:58 debian kernel: [ 1019.729066] CR2: 0000000000000001

Message from syslogd@debian at May 13 16:00:58 ...
kernel:[ 1019.729066] CR2: 0000000000000001
May 13 16:00:58 debian kernel: [ 1019.729066] ---[ end trace 1bf0d7ed3d5d762d ]---

[behlendorf]

Regarding running on loopback devices you might try using the latest master source instead of -rc8. There were a few changes merged which improved the memory management and may avoid certain deadlock scenarios similar to what you've described.

[behlendorf]

This should be resolved in the master source, what will be tagged as -rc11.