Connected to domain zfstest Escape character is ^] [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 4.3.3-hardened-r4 (root@localhost) (gcc version 4.9.3 (Gentoo Hardened 4.9.3 p1.5, pie-0.6.4) ) #5 SMP PREEMPT Sat Feb 6 09:40:39 2016 [ 0.000000] Command line: root=/dev/vdc1 rootfstype=ext4 console=tty0 console=ttyS0,115200 [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers' [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 0x340 bytes, using 'standard' format. [ 0.000000] x86/fpu: Using 'eager' FPU context switches. [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003ffdffff] usable [ 0.000000] BIOS-e820: [mem 0x000000003ffe0000-0x000000003fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x3ffe0 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT [ 0.000000] found SMP MP-table at [mem 0x000f6430-0x000f643f] mapped at [ffff8800000f6430] [ 0.000000] Scanning 1 areas for low memory corruption [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] [ 0.000000] init_memory_mapping: [mem 0x3fc00000-0x3fdfffff] [ 0.000000] init_memory_mapping: [mem 0x20000000-0x3fbfffff] [ 0.000000] init_memory_mapping: [mem 0x00100000-0x1fffffff] [ 0.000000] init_memory_mapping: [mem 0x3fe00000-0x3ffdffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F61C0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x000000003FFE16D2 000030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x000000003FFE0BDA 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x000000003FFE0040 000B9A (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x000000003FFE0000 000040 [ 0.000000] ACPI: SSDT 0x000000003FFE0C4E 0009FC (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 0x000000003FFE164A 000088 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:1d0f0001, primary cpu clock [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x000000003ffdffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x000000003ffdffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000003ffdffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 3 CPUs, 0 hotplug CPUs [ 0.000000] e820: [mem 0x40000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:4 nr_cpumask_bits:4 nr_cpu_ids:3 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 29 pages/cpu @ffff88003fc00000 s80040 r8192 d30552 u524288 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 3fc0b800 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 257897 [ 0.000000] Kernel command line: root=/dev/vdc1 rootfstype=ext4 console=tty0 console=ttyS0,115200 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.000000] Memory: 1010832K/1048056K available (8351K kernel code, 829K rwdata, 3812K rodata, 2976K init, 1132K bss, 37224K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=3, Nodes=1 [ 0.000000] Preemptible hierarchical RCU implementation. [ 0.000000] Build-time adjustment of leaf fanout to 64. [ 0.000000] RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=3. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=3 [ 0.000000] NR_IRQS:4352 nr_irqs:448 16 [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-2. [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] tsc: Detected 2693.764 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5387.52 BogoMIPS (lpj=2693764) [ 0.004003] pid_max: default: 32768 minimum: 501 [ 0.005021] ACPI: Core revision 20150818 [ 0.007475] ACPI: 2 ACPI AML tables successfully acquired and loaded [ 0.010096] Security Framework initialized [ 0.011007] SELinux: Initializing. [ 0.012052] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes) [ 0.013011] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes) [ 0.014247] Initializing cgroup subsys io [ 0.015008] Initializing cgroup subsys memory [ 0.016011] Initializing cgroup subsys devices [ 0.017005] Initializing cgroup subsys freezer [ 0.018005] Initializing cgroup subsys pids [ 0.019071] PAX: PCID detected [ 0.020004] PAX: INVPCID detected [ 0.021008] PAX: strong UDEREF enabled [ 0.023519] mce: CPU supports 10 MCE banks [ 0.024097] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.025006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.074778] Freeing SMP alternatives memory: 32K (ffffffff9d0e8000 - ffffffff9d0f0000) [ 0.104876] ftrace: allocating 24790 entries in 97 pages [ 0.114768] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.115000] smpboot: CPU0: Intel Core Processor (Haswell, no TSX) (family: 0x6, model: 0x3c, stepping: 0x1) [ 0.117038] Performance Events: unsupported p6 CPU model 60 no PMU driver, software events only. [ 0.128291] x86: Booting SMP configuration: [ 0.129012] .... node #0, CPUs: #1 [ 0.125114] kvm-clock: cpu 1, msr 0:1d0f0041, secondary cpu clock [ 0.125114] PAX: PCID detected [ 0.125114] PAX: INVPCID detected [ 0.125114] PAX: strong UDEREF enabled [ 0.147048] KVM setup async PF for cpu 1 [ 0.148000] kvm-stealtime: cpu 1, msr 3fc8b800 [ 0.147065] #2 [ 0.147065] kvm-clock: cpu 2, msr 0:1d0f0081, secondary cpu clock [ 0.147065] PAX: PCID detected [ 0.147065] PAX: INVPCID detected [ 0.147065] PAX: strong UDEREF enabled [ 0.169055] x86: Booted up 1 node, 3 CPUs [ 0.169035] KVM setup async PF for cpu 2 [ 0.169035] kvm-stealtime: cpu 2, msr 3fd0b800 [ 0.172005] smpboot: Total of 3 processors activated (16162.58 BogoMIPS) [ 0.174393] devtmpfs: initialized [ 0.177158] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.178316] NET: Registered protocol family 16 [ 0.181060] cpuidle: using governor ladder [ 0.184051] cpuidle: using governor menu [ 0.185298] ACPI: bus type PCI registered [ 0.186076] PCI: Using configuration type 1 for base access [ 0.193257] ACPI: Added _OSI(Module Device) [ 0.194006] ACPI: Added _OSI(Processor Device) [ 0.195006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196004] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198227] ACPI: Interpreter enabled [ 0.199009] ACPI: (supports S0 S5) [ 0.200012] ACPI: Using IOAPIC for interrupt routing [ 0.201032] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.206216] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.207014] acpi PNP0A03:00: _OSC: OS supports [Segments] [ 0.208010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.209055] PCI host bridge to bus 0000:00 [ 0.210005] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.211005] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.212004] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.213005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.214005] pci_bus 0000:00: root bus resource [mem 0x40000000-0xfebfffff window] [ 0.226040] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.227005] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.228005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.229006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.230622] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.231015] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.415745] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.417118] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.419121] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.421109] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.423071] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.425529] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.426201] vgaarb: setting as boot device: PCI:0000:00:02.0 [ 0.427000] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.427020] vgaarb: loaded [ 0.428005] vgaarb: bridge control possible 0000:00:02.0 [ 0.429193] SCSI subsystem initialized [ 0.430207] ACPI: bus type USB registered [ 0.431051] usbcore: registered new interface driver usbfs [ 0.432032] usbcore: registered new interface driver hub [ 0.433083] usbcore: registered new device driver usb [ 0.434065] pps_core: LinuxPPS API ver. 1 registered [ 0.435009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.436128] PCI: Using ACPI for IRQ routing [ 0.439990] clocksource: Switched to clocksource kvm-clock [ 0.446945] pnp: PnP ACPI init [ 0.448869] pnp: PnP ACPI: found 5 devices [ 0.459268] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.465097] NET: Registered protocol family 2 [ 0.468554] TCP established hash table entries: 8192 (order: 4, 65536 bytes) [ 0.472399] TCP bind hash table entries: 8192 (order: 6, 262144 bytes) [ 0.475003] TCP: Hash tables configured (established 8192 bind 8192) [ 0.477117] UDP hash table entries: 512 (order: 3, 49152 bytes) [ 0.479193] UDP-Lite hash table entries: 512 (order: 3, 49152 bytes) [ 0.481394] NET: Registered protocol family 1 [ 0.483053] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.485038] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.486978] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.497291] Scanning for low memory corruption every 60 seconds [ 0.501134] cryptomgr_test (43) used greatest stack depth: 15024 bytes left [ 0.509973] sha1_ssse3: Using AVX2 optimized SHA-1 implementation [ 0.512844] sha256_ssse3: Using AVX2 optimized SHA-256 implementation [ 0.515319] sha512_ssse3: Using AVX2 optimized SHA-512 implementation [ 0.526241] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 0.528810] audit: initializing netlink subsys (disabled) [ 0.530880] audit: type=2000 audit(1455353465.555:1): initialized [ 0.533448] Initialise system trusted keyring [ 0.535494] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.542426] fuse init (API version 7.23) [ 0.544497] Key type big_key registered [ 0.570407] NET: Registered protocol family 38 [ 0.572264] Key type asymmetric registered [ 0.574192] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.577392] io scheduler noop registered (default) [ 0.579368] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.584234] ACPI: Power Button [PWRF] [ 0.626697] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 0.629358] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver [ 0.665587] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.667638] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 0.702314] virtio-pci 0000:00:07.0: virtio_pci: leaving for legacy driver [ 0.730033] virtio-pci 0000:00:09.0: virtio_pci: leaving for legacy driver [ 0.763677] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 0.765689] virtio-pci 0000:00:0a.0: virtio_pci: leaving for legacy driver [ 0.793579] virtio-pci 0000:00:0b.0: virtio_pci: leaving for legacy driver [ 0.830734] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.834171] virtio-pci 0000:00:0c.0: virtio_pci: leaving for legacy driver [ 0.837796] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2005.398424] audit: type=1400 audit(1455355470.561:25): avc: denied { getattr } for pid=11254 comm="setfiles" name="/" dev="debugfs" ino=1 scontext=root:sysadm_r:setfiles_t tcontext=system_u:object_r:debugfs_t tclass=filesystem permissive=1 [ 2581.325456] SPL: Loaded module v0.6.5-43_g8f3b403 (DEBUG mode) [ 2581.846528] ZFS: Loaded module v0.6.5-1 (DEBUG mode), ZFS pool version 5000, ZFS filesystem version 5 [ 2597.600830] audit: type=1400 audit(1455356062.763:26): avc: denied { create } for pid=25517 comm="zpool" name="blkid" scontext=root:sysadm_r:mount_t tcontext=root:object_r:var_run_t tclass=dir permissive=1 [ 2597.607794] audit: type=1400 audit(1455356062.770:27): avc: denied { create } for pid=25517 comm="zpool" name="blkid.tab" scontext=root:sysadm_r:mount_t tcontext=root:object_r:var_run_t tclass=file permissive=1 [ 2597.616310] audit: type=1400 audit(1455356062.779:28): avc: denied { write open } for pid=25517 comm="zpool" path="/run/blkid/blkid.tab" dev="tmpfs" ino=310066 scontext=root:sysadm_r:mount_t tcontext=root:object_r:var_run_t tclass=file permissive=1 [ 2597.726436] SPL: using hostid 0x00000000 [ 2598.540043] audit: type=1400 audit(1455356063.703:29): avc: denied { write } for pid=26505 comm="spa_async" name="zpool.cache" dev="vdc1" ino=139646 scontext=system_u:system_r:kernel_t tcontext=root:object_r:etc_runtime_t tclass=file permissive=1 [ 2598.546496] audit: type=1400 audit(1455356063.709:30): avc: denied { open } for pid=26505 comm="spa_async" path="/etc/zfs/zpool.cache" dev="vdc1" ino=139646 scontext=system_u:system_r:kernel_t tcontext=root:object_r:etc_runtime_t tclass=file permissive=1 [ 2598.555321] audit: type=1400 audit(1455356063.718:31): avc: denied { getattr } for pid=26505 comm="spa_async" path="/etc/zfs/zpool.cache" dev="vdc1" ino=139646 scontext=system_u:system_r:kernel_t tcontext=root:object_r:etc_runtime_t tclass=file permissive=1 [ 2598.614808] grsec: mount of tank to /tank by /sbin/mount.zfs[mount.zfs:26507] uid/euid:0/0 gid/egid:0/0, parent /bin/mount[mount:26506] uid/euid:0/0 gid/egid:0/0 [ 2643.698619] grsec: mount of tank/test1 to /tank/test1 by /sbin/mount.zfs[mount.zfs:26511] uid/euid:0/0 gid/egid:0/0, parent /bin/mount[mount:26510] uid/euid:0/0 gid/egid:0/0 [ 3044.002089] sysrq: SysRq : Show Blocked State [ 3044.002997] task PC stack pid father [ 3044.002997] init D ffff88003e1085b0 8 1 0 0x00000000 [ 3044.002997] ffffc9000000b3c8 0000000000000086 ffffffff9c100e10 ffffc9000000b408 [ 3044.002997] ffffc9000324b3f8 ffff880037795600 ffff88003e108000 ffffc9000000b3a8 [ 3044.002997] ffffffffc000ca27 725cb22d39b5ad1c ffff88003e108a78 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] ? zfs_znode_hold_exit+0x15b/0x200 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] dispose_list+0x38/0x70 [ 3044.002997] [] prune_icache_sb+0x61/0x90 [ 3044.002997] [] super_cache_scan+0x278/0x450 [ 3044.002997] [] ? __list_lru_count_one.isra.2+0x44/0x80 [ 3044.002997] [] shrink_slab.part.40+0x3f9/0x610 [ 3044.002997] [] ? shrink_lruvec+0x625/0x720 [ 3044.002997] [] shrink_zone+0x28e/0x2c0 [ 3044.002997] [] do_try_to_free_pages+0x16b/0x490 [ 3044.002997] [] ? pfmemalloc_watermark_ok+0xbc/0xf0 [ 3044.002997] [] try_to_free_pages+0xe1/0x1c0 [ 3044.002997] [] __alloc_pages_nodemask+0x55b/0x990 [ 3044.002997] [] __do_page_cache_readahead+0x130/0x290 [ 3044.002997] [] ? find_get_entry+0x6c/0xa0 [ 3044.002997] [] ? pagecache_get_page+0x2d/0x1c0 [ 3044.002997] [] filemap_fault+0x3a3/0x460 [ 3044.002997] [] __do_fault+0x7f/0x130 [ 3044.002997] [] handle_mm_fault+0xb75/0x1750 [ 3044.002997] [] __do_page_fault+0x220/0x6d0 [ 3044.002997] [] trace_do_page_fault+0x49/0x150 [ 3044.002997] [] do_async_page_fault+0x2c/0xa0 [ 3044.002997] [] async_page_fault+0x28/0x30 [ 3044.002997] khugepaged D ffffffff9c107ef0 10016 37 2 0x00000000 [ 3044.002997] ffffc9000013b558 0000000000000046 ffffffff9c100e10 ffffc9000013b598 [ 3044.002997] ffffc9000041b848 ffff88003e1e2b00 ffff88003e2bb5c0 ffffc9000013b538 [ 3044.002997] ffffffffc000ca27 d361c9d2683a23af ffff88003e2bc038 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] ? zfs_znode_hold_exit+0x15b/0x200 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] dispose_list+0x38/0x70 [ 3044.002997] [] prune_icache_sb+0x61/0x90 [ 3044.002997] [] super_cache_scan+0x278/0x450 [ 3044.002997] [] ? __list_lru_count_one.isra.2+0x44/0x80 [ 3044.002997] [] shrink_slab.part.40+0x3f9/0x610 [ 3044.002997] [] ? shrink_lruvec+0x625/0x720 [ 3044.002997] [] shrink_zone+0x28e/0x2c0 [ 3044.002997] [] do_try_to_free_pages+0x16b/0x490 [ 3044.002997] [] try_to_free_pages+0xe1/0x1c0 [ 3044.002997] [] __alloc_pages_nodemask+0x55b/0x990 [ 3044.002997] [] khugepaged+0x475/0x16a0 [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] ? maybe_pmd_mkwrite+0x40/0x40 [ 3044.002997] [] ? maybe_pmd_mkwrite+0x40/0x40 [ 3044.002997] [] kthread+0x10a/0x120 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] [] ret_from_fork+0x3e/0x70 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] kswapd0 D ffff88003d92dbb0 10704 129 2 0x00000000 [ 3044.002997] ffffc9000041b808 0000000000000046 ffffffff9c100e10 ffffc9000041b848 [ 3044.002997] ffffc900001ab338 ffff8800349fab00 ffff88003d92d600 ffffc9000041b7e8 [ 3044.002997] ffffffffc000ca27 28ccbfc59f154f1c ffff88003d92e078 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] ? zfs_znode_hold_exit+0x15b/0x200 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] dispose_list+0x38/0x70 [ 3044.002997] [] prune_icache_sb+0x61/0x90 [ 3044.002997] [] super_cache_scan+0x278/0x450 [ 3044.002997] [] ? __list_lru_count_one.isra.2+0x44/0x80 [ 3044.002997] [] shrink_slab.part.40+0x3f9/0x610 [ 3044.002997] [] ? shrink_lruvec+0x625/0x720 [ 3044.002997] [] shrink_zone+0x28e/0x2c0 [ 3044.002997] [] kswapd+0x53f/0x9a0 [ 3044.002997] [] ? mem_cgroup_shrink_node_zone+0x200/0x200 [ 3044.002997] [] kthread+0x10a/0x120 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] [] ret_from_fork+0x3e/0x70 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] z_wr_iss D ffffffff9c107ef0 12512 26270 2 0x00000000 [ 3044.002997] ffffc900025b3ac8 0000000000000046 ffffffff9c100e10 0000000000000000 [ 3044.002997] ffff88003fc12140 ffffffff9cc04980 ffff880034ae35c0 ffffc900025b3a98 [ 3044.002997] ffffffff9c822406 938c0da7dd4e8eec ffff880034ae4038 000000010029dec3 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? schedule_timeout+0x176/0x310 [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] schedule_timeout+0x176/0x310 [ 3044.002997] [] ? trace_event_raw_event_tick_stop+0xf0/0xf0 [ 3044.002997] [] schedule_timeout_uninterruptible+0x30/0x40 [ 3044.002997] [] __alloc_pages_nodemask+0x8b5/0x990 [ 3044.002997] [] _abd_alloc_scatter+0x1d5/0x3f0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock+0x1f/0x40 [ 3044.002997] [] zio_write_bp_init+0x86e/0x14d0 [zfs] [ 3044.002997] [] zio_execute+0xed/0x2f0 [zfs] [ 3044.002997] [] taskq_thread+0x2a2/0x5c0 [spl] [ 3044.002997] [] ? wake_up_q+0x80/0x80 [ 3044.002997] [] ? taskq_thread_should_stop+0x90/0x90 [spl] [ 3044.002997] [] kthread+0x10a/0x120 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] [] ret_from_fork+0x3e/0x70 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] z_wr_iss D ffff880034ae7130 12128 26271 2 0x00000000 [ 3044.002997] ffffc900025bb758 0000000000000046 ffffffff9c100e10 0000000000000000 [ 3044.002997] ffff88003fc12140 ffff880034ae35c0 ffff880034ae6b80 ffffc900025bb728 [ 3044.002997] ffffffff9c822406 86e83dda264b8829 ffff880034ae75f8 000000010029dec2 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? schedule_timeout+0x176/0x310 [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] schedule_timeout+0x176/0x310 [ 3044.002997] [] ? trace_event_raw_event_tick_stop+0xf0/0xf0 [ 3044.002997] [] schedule_timeout_uninterruptible+0x30/0x40 [ 3044.002997] [] __alloc_pages_nodemask+0x8b5/0x990 [ 3044.002997] [] new_slab+0x31d/0x430 [ 3044.002997] [] __slab_alloc.constprop.70+0x34c/0x460 [ 3044.002997] [] ? spl_kmem_cache_alloc+0xa9/0xc80 [spl] [ 3044.002997] [] kmem_cache_alloc+0x1e7/0x240 [ 3044.002997] [] ? spl_kmem_cache_alloc+0xa9/0xc80 [spl] [ 3044.002997] [] spl_kmem_cache_alloc+0xa9/0xc80 [spl] [ 3044.002997] [] ? try_to_wake_up+0x46/0x400 [ 3044.002997] [] ? spa_config_held+0x8e/0xb0 [zfs] [ 3044.002997] [] zio_create+0x8a/0x660 [zfs] [ 3044.002997] [] zio_vdev_child_io+0xf2/0x190 [zfs] [ 3044.002997] [] ? vdev_config_sync+0x1e0/0x1e0 [zfs] [ 3044.002997] [] vdev_mirror_io_start+0xac/0x1e0 [zfs] [ 3044.002997] [] ? vdev_config_sync+0x1e0/0x1e0 [zfs] [ 3044.002997] [] zio_vdev_io_start+0x211/0x510 [zfs] [ 3044.002997] [] zio_nowait+0xd6/0x2b0 [zfs] [ 3044.002997] [] vdev_mirror_io_start+0xb4/0x1e0 [zfs] [ 3044.002997] [] ? vdev_config_sync+0x1e0/0x1e0 [zfs] [ 3044.002997] [] zio_vdev_io_start+0x324/0x510 [zfs] [ 3044.002997] [] zio_execute+0xed/0x2f0 [zfs] [ 3044.002997] [] taskq_thread+0x2a2/0x5c0 [spl] [ 3044.002997] [] ? wake_up_q+0x80/0x80 [ 3044.002997] [] ? taskq_thread_should_stop+0x90/0x90 [spl] [ 3044.002997] [] kthread+0x10a/0x120 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] [] ret_from_fork+0x3e/0x70 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] z_iput D ffff88003b5630b0 11392 26483 2 0x00000000 [ 3044.002997] ffffc90002d4bab0 0000000000000046 ffffffff9c100e10 ffffc90002d4baf0 [ 3044.002997] ffffc90002d53af0 ffff88003d92d600 ffff88003b562b00 ffffc90002d4ba90 [ 3044.002997] ffffffffc000ca27 a694d3e8a27bd0ab ffff88003b563578 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] iput+0x1cc/0x230 [ 3044.002997] [] taskq_thread+0x2a2/0x5c0 [spl] [ 3044.002997] [] ? wake_up_q+0x80/0x80 [ 3044.002997] [] ? taskq_thread_should_stop+0x90/0x90 [spl] [ 3044.002997] [] kthread+0x10a/0x120 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] [] ret_from_fork+0x3e/0x70 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] z_iput D ffff88003b564630 11648 26484 2 0x00000000 [ 3044.002997] ffffc90002d53ab0 0000000000000046 ffffffff9c100e10 ffffc90002d53af0 [ 3044.002997] ffff88003ac51480 ffff88003b562b00 ffff88003b564080 ffffc90002d53a90 [ 3044.002997] ffffffffc000ca27 1f2ed056cd04983c ffff88003b564af8 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] iput+0x1cc/0x230 [ 3044.002997] [] taskq_thread+0x2a2/0x5c0 [spl] [ 3044.002997] [] ? wake_up_q+0x80/0x80 [ 3044.002997] [] ? taskq_thread_should_stop+0x90/0x90 [spl] [ 3044.002997] [] kthread+0x10a/0x120 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] [] ret_from_fork+0x3e/0x70 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] z_iput D ffff88003b563b70 11336 26485 2 0x00000000 [ 3044.002997] ffffc90002d5bab0 0000000000000046 ffffffff9c100e10 ffffc90002d5baf0 [ 3044.002997] ffffc900001c3428 ffff880037795600 ffff88003b5635c0 ffffc90002d5ba90 [ 3044.002997] ffffffffc000ca27 ea755f09078865cb ffff88003b564038 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] iput+0x1cc/0x230 [ 3044.002997] [] taskq_thread+0x2a2/0x5c0 [spl] [ 3044.002997] [] ? wake_up_q+0x80/0x80 [ 3044.002997] [] ? taskq_thread_should_stop+0x90/0x90 [spl] [ 3044.002997] [] kthread+0x10a/0x120 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] [] ret_from_fork+0x3e/0x70 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] txg_sync D ffff880037796670 11328 26504 2 0x00000000 [ 3044.002997] ffffc90003073a68 0000000000000046 0000000000000000 ffff880037baf420 [ 3044.002997] ffffc90003073a70 ffff880034ae35c0 ffff8800377960c0 ffffffffc0430af0 [ 3044.002997] ffffffff9c8224ea 0cb6e9c5c6dc2647 ffff880037796b38 7fffffffffffffff [ 3044.002997] Call Trace: [ 3044.002997] [] ? zio_taskq_member.isra.6.constprop.13+0x70/0x70 [zfs] [ 3044.002997] [] ? schedule_timeout+0x25a/0x310 [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] schedule_timeout+0x25a/0x310 [ 3044.002997] [] ? preempt_count_add+0x54/0xa0 [ 3044.002997] [] ? kvm_clock_read+0x23/0x50 [ 3044.002997] [] ? kvm_clock_get_cycles+0x9/0x20 [ 3044.002997] [] ? ktime_get+0x4d/0xd0 [ 3044.002997] [] io_schedule_timeout+0xb1/0x130 [ 3044.002997] [] cv_wait_common+0x117/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait_io+0x27/0x40 [spl] [ 3044.002997] [] zio_wait+0x181/0x360 [zfs] [ 3044.002997] [] dsl_pool_sync+0x118/0x5f0 [zfs] [ 3044.002997] [] spa_sync+0x36d/0xe10 [zfs] [ 3044.002997] [] ? kvm_clock_read+0x23/0x50 [ 3044.002997] [] txg_sync_thread+0x3c5/0x6a0 [zfs] [ 3044.002997] [] ? txg_fini+0x2e0/0x2e0 [zfs] [ 3044.002997] [] thread_generic_wrapper+0x95/0xe0 [spl] [ 3044.002997] [] ? __thread_exit+0x20/0x20 [spl] [ 3044.002997] [] kthread+0x10a/0x120 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] [] ret_from_fork+0x3e/0x70 [ 3044.002997] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3044.002997] htop D ffffffff9c107ef0 8 26516 1300 0x00000000 [ 3044.002997] ffffc900031db388 0000000000000086 ffffffff9c100e10 ffffc900031db3c8 [ 3044.002997] ffffc9000013b598 ffff88003e1e2b00 ffff88003ae7cb40 ffffc900031db368 [ 3044.002997] ffffffffc000ca27 229c73a74d59d769 ffff88003ae7d5b8 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] ? zfs_znode_hold_exit+0x15b/0x200 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] dispose_list+0x38/0x70 [ 3044.002997] [] prune_icache_sb+0x61/0x90 [ 3044.002997] [] super_cache_scan+0x278/0x450 [ 3044.002997] [] ? __list_lru_count_one.isra.2+0x44/0x80 [ 3044.002997] [] shrink_slab.part.40+0x3f9/0x610 [ 3044.002997] [] ? shrink_lruvec+0x625/0x720 [ 3044.002997] [] ? alloc_indirect.isra.3+0x2d/0x70 [ 3044.002997] [] shrink_zone+0x28e/0x2c0 [ 3044.002997] [] do_try_to_free_pages+0x16b/0x490 [ 3044.002997] [] ? pfmemalloc_watermark_ok+0xbc/0xf0 [ 3044.002997] [] try_to_free_pages+0xe1/0x1c0 [ 3044.002997] [] __alloc_pages_nodemask+0x55b/0x990 [ 3044.002997] [] __do_page_cache_readahead+0x130/0x290 [ 3044.002997] [] ? find_get_entry+0x6c/0xa0 [ 3044.002997] [] ? pagecache_get_page+0x2d/0x1c0 [ 3044.002997] [] filemap_fault+0x3a3/0x460 [ 3044.002997] [] __do_fault+0x7f/0x130 [ 3044.002997] [] handle_mm_fault+0xb75/0x1750 [ 3044.002997] [] __do_page_fault+0x220/0x6d0 [ 3044.002997] [] ? debug_smp_processor_id+0x17/0x20 [ 3044.002997] [] trace_do_page_fault+0x49/0x150 [ 3044.002997] [] do_async_page_fault+0x2c/0xa0 [ 3044.002997] [] async_page_fault+0x28/0x30 [ 3044.002997] find D ffff88003ae7a5f0 8 26530 1277 0x00000000 [ 3044.002997] ffffc9000324b3b8 0000000000000086 ffffffff9c100e10 ffffc9000324b3f8 [ 3044.002997] ffffc90000293b70 ffff8800349fab00 ffff88003ae7a040 ffffc9000324b398 [ 3044.002997] ffffffffc000ca27 413d722e74f23d6e ffff88003ae7aab8 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] ? zfs_znode_hold_exit+0x15b/0x200 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] dispose_list+0x38/0x70 [ 3044.002997] [] prune_icache_sb+0x61/0x90 [ 3044.002997] [] super_cache_scan+0x278/0x450 [ 3044.002997] [] ? __list_lru_count_one.isra.2+0x44/0x80 [ 3044.002997] [] shrink_slab.part.40+0x3f9/0x610 [ 3044.002997] [] ? shrink_lruvec+0x625/0x720 [ 3044.002997] [] shrink_zone+0x28e/0x2c0 [ 3044.002997] [] do_try_to_free_pages+0x16b/0x490 [ 3044.002997] [] ? pfmemalloc_watermark_ok+0xbc/0xf0 [ 3044.002997] [] try_to_free_pages+0xe1/0x1c0 [ 3044.002997] [] __alloc_pages_nodemask+0x55b/0x990 [ 3044.002997] [] ? set_next_entity+0xaa/0x820 [ 3044.002997] [] __do_page_cache_readahead+0x130/0x290 [ 3044.002997] [] ? find_get_entry+0x6c/0xa0 [ 3044.002997] [] ? pagecache_get_page+0x2d/0x1c0 [ 3044.002997] [] filemap_fault+0x3a3/0x460 [ 3044.002997] [] __do_fault+0x7f/0x130 [ 3044.002997] [] handle_mm_fault+0xb75/0x1750 [ 3044.002997] [] __do_page_fault+0x220/0x6d0 [ 3044.002997] [] trace_do_page_fault+0x49/0x150 [ 3044.002997] [] do_async_page_fault+0x2c/0xa0 [ 3044.002997] [] async_page_fault+0x28/0x30 [ 3044.002997] python3.4 D ffff88003ae785b0 8 26561 26521 0x00000000 [ 3044.002997] ffffc90004dbb3b8 0000000000000086 ffffffff9c100e10 ffffc90004dbb3f8 [ 3044.002997] ffffc9000000b408 ffff8800349fab00 ffff88003ae78000 ffffc90004dbb398 [ 3044.002997] ffffffffc000ca27 1431ffc5673d486d ffff88003ae78a78 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] ? zfs_znode_hold_exit+0x15b/0x200 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] dispose_list+0x38/0x70 [ 3044.002997] [] prune_icache_sb+0x61/0x90 [ 3044.002997] [] super_cache_scan+0x278/0x450 [ 3044.002997] [] ? __list_lru_count_one.isra.2+0x44/0x80 [ 3044.002997] [] shrink_slab.part.40+0x3f9/0x610 [ 3044.002997] [] ? shrink_lruvec+0x625/0x720 [ 3044.002997] [] shrink_zone+0x28e/0x2c0 [ 3044.002997] [] do_try_to_free_pages+0x16b/0x490 [ 3044.002997] [] ? pfmemalloc_watermark_ok+0xbc/0xf0 [ 3044.002997] [] try_to_free_pages+0xe1/0x1c0 [ 3044.002997] [] __alloc_pages_nodemask+0x55b/0x990 [ 3044.002997] [] __do_page_cache_readahead+0x130/0x290 [ 3044.002997] [] ? find_get_entry+0x6c/0xa0 [ 3044.002997] [] ? pagecache_get_page+0x2d/0x1c0 [ 3044.002997] [] filemap_fault+0x3a3/0x460 [ 3044.002997] [] __do_fault+0x7f/0x130 [ 3044.002997] [] handle_mm_fault+0xb75/0x1750 [ 3044.002997] [] __do_page_fault+0x220/0x6d0 [ 3044.002997] [] trace_do_page_fault+0x49/0x150 [ 3044.002997] [] do_async_page_fault+0x2c/0xa0 [ 3044.002997] [] async_page_fault+0x28/0x30 [ 3044.002997] rsync D ffff88003ae7f130 8 26592 1277 0x00000000 [ 3044.002997] ffffc900001ab2f8 0000000000000086 ffffffff9c100e10 ffffc900001ab338 [ 3044.002997] ffffc90004dbb3f8 ffff880037795600 ffff88003ae7eb80 ffffc900001ab2d8 [ 3044.002997] ffffffffc000ca27 ceb886141ba4ec57 ffff88003ae7f5f8 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] ? zfs_znode_hold_exit+0x15b/0x200 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] dispose_list+0x38/0x70 [ 3044.002997] [] prune_icache_sb+0x61/0x90 [ 3044.002997] [] super_cache_scan+0x278/0x450 [ 3044.002997] [] ? __list_lru_count_one.isra.2+0x44/0x80 [ 3044.002997] [] shrink_slab.part.40+0x3f9/0x610 [ 3044.002997] [] ? shrink_lruvec+0x625/0x720 [ 3044.002997] [] ? __bt_get.isra.10+0xe2/0x1d0 [ 3044.002997] [] shrink_zone+0x28e/0x2c0 [ 3044.002997] [] do_try_to_free_pages+0x16b/0x490 [ 3044.002997] [] ? pfmemalloc_watermark_ok+0xbc/0xf0 [ 3044.002997] [] try_to_free_pages+0xe1/0x1c0 [ 3044.002997] [] __alloc_pages_nodemask+0x55b/0x990 [ 3044.002997] [] __do_page_cache_readahead+0x130/0x290 [ 3044.002997] [] ? find_get_entry+0x6c/0xa0 [ 3044.002997] [] ? pagecache_get_page+0x2d/0x1c0 [ 3044.002997] [] filemap_fault+0x3a3/0x460 [ 3044.002997] [] __do_fault+0x7f/0x130 [ 3044.002997] [] handle_mm_fault+0xb75/0x1750 [ 3044.002997] [] ? unlock_page+0x5a/0x70 [ 3044.002997] [] __do_page_fault+0x220/0x6d0 [ 3044.002997] [] trace_do_page_fault+0x49/0x150 [ 3044.002997] [] do_async_page_fault+0x2c/0xa0 [ 3044.002997] [] async_page_fault+0x28/0x30 [ 3044.002997] [] ? retint_user+0x15/0x20 [ 3044.002997] rsync D ffffffff9c107ef0 8 26593 26592 0x00000000 [ 3044.002997] ffffc900001c33e8 0000000000000086 ffff88003ac512d0 ffffc900001c3398 [ 3044.002997] ffffffff9c81dffb ffffffff9cc04980 ffff88003ae78ac0 0000000000000001 [ 3044.002997] ffffffffc000ca27 d9b543b19c7b5518 ffff88003ae79538 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? preempt_schedule+0x2b/0x30 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] ? zfs_znode_hold_exit+0x15b/0x200 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] dispose_list+0x38/0x70 [ 3044.002997] [] prune_icache_sb+0x61/0x90 [ 3044.002997] [] super_cache_scan+0x278/0x450 [ 3044.002997] [] ? __list_lru_count_one.isra.2+0x44/0x80 [ 3044.002997] [] shrink_slab.part.40+0x3f9/0x610 [ 3044.002997] [] ? shrink_lruvec+0x625/0x720 [ 3044.002997] [] shrink_zone+0x28e/0x2c0 [ 3044.002997] [] do_try_to_free_pages+0x16b/0x490 [ 3044.002997] [] ? pfmemalloc_watermark_ok+0xbc/0xf0 [ 3044.002997] [] try_to_free_pages+0xe1/0x1c0 [ 3044.002997] [] __alloc_pages_nodemask+0x55b/0x990 [ 3044.002997] [] __do_page_cache_readahead+0x130/0x290 [ 3044.002997] [] ? find_get_entry+0x6c/0xa0 [ 3044.002997] [] ? pagecache_get_page+0x2d/0x1c0 [ 3044.002997] [] filemap_fault+0x3a3/0x460 [ 3044.002997] [] __do_fault+0x7f/0x130 [ 3044.002997] [] handle_mm_fault+0xb75/0x1750 [ 3044.002997] [] __do_page_fault+0x220/0x6d0 [ 3044.002997] [] trace_do_page_fault+0x49/0x150 [ 3044.002997] [] do_async_page_fault+0x2c/0xa0 [ 3044.002997] [] async_page_fault+0x28/0x30 [ 3044.002997] rsync D ffffffff9c107ef0 8 26594 26593 0x00000000 [ 3044.002997] ffffc9000027b388 0000000000000086 ffffffff9c100e10 ffffc9000027b3c8 [ 3044.002997] ffffc900031db3c8 ffff88003e1e2b00 ffff880037796b80 ffffc9000027b368 [ 3044.002997] ffffffffc000ca27 0aba47d994e78643 ffff8800377975f8 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3044.002997] [] zfs_rmnode+0x136/0x440 [zfs] [ 3044.002997] [] ? zfs_znode_hold_exit+0x15b/0x200 [zfs] [ 3044.002997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3044.002997] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3044.002997] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3044.002997] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3044.002997] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3044.002997] [] evict+0xbf/0x1a0 [ 3044.002997] [] dispose_list+0x38/0x70 [ 3044.002997] [] prune_icache_sb+0x61/0x90 [ 3044.002997] [] super_cache_scan+0x278/0x450 [ 3044.002997] [] ? __list_lru_count_one.isra.2+0x44/0x80 [ 3044.002997] [] shrink_slab.part.40+0x3f9/0x610 [ 3044.002997] [] ? shrink_lruvec+0x625/0x720 [ 3044.002997] [] shrink_zone+0x28e/0x2c0 [ 3044.002997] [] do_try_to_free_pages+0x16b/0x490 [ 3044.002997] [] ? pfmemalloc_watermark_ok+0xbc/0xf0 [ 3044.002997] [] try_to_free_pages+0xe1/0x1c0 [ 3044.002997] [] __alloc_pages_nodemask+0x55b/0x990 [ 3044.002997] [] __do_page_cache_readahead+0x130/0x290 [ 3044.002997] [] ? find_get_entry+0x6c/0xa0 [ 3044.002997] [] ? pagecache_get_page+0x2d/0x1c0 [ 3044.002997] [] filemap_fault+0x3a3/0x460 [ 3044.002997] [] __do_fault+0x7f/0x130 [ 3044.002997] [] handle_mm_fault+0xb75/0x1750 [ 3044.002997] [] __do_page_fault+0x220/0x6d0 [ 3044.002997] [] trace_do_page_fault+0x49/0x150 [ 3044.002997] [] do_async_page_fault+0x2c/0xa0 [ 3044.002997] [] async_page_fault+0x28/0x30 [ 3044.002997] rm D ffff88003fc92140 8 26597 1277 0x00000000 [ 3044.002997] ffffc90000293b30 0000000000000086 ffffffff9c100e10 0000000000000000 [ 3044.002997] 0000000000000001 ffff88003ae7a040 ffff88003ae7e0c0 0000000000000001 [ 3044.002997] ffffffffc000ca27 08c783baa08bf728 ffff88003ae7eb38 ffff88003ac51290 [ 3044.002997] Call Trace: [ 3044.002997] [] ? get_parent_ip+0x10/0x50 [ 3044.002997] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] schedule+0x3d/0x90 [ 3044.002997] [] cv_wait_common+0x197/0x2f0 [spl] [ 3044.002997] [] ? wait_woken+0xa0/0xa0 [ 3044.002997] [] __cv_wait+0x24/0x30 [spl] [ 3044.002997] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3044.002997] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3044.002997] [] zfs_remove+0x189/0xa00 [zfs] [ 3044.002997] [] ? get_parent_ip+0xa/0x50 [ 3044.002997] [] zpl_unlink+0x68/0x170 [zfs] [ 3044.002997] [] vfs_unlink+0x111/0x1a0 [ 3044.002997] [] do_unlinkat+0x281/0x350 [ 3044.002997] [] SyS_unlinkat+0x2f/0x50 [ 3044.002997] [] entry_SYSCALL_64_fastpath+0x12/0x8a [ 3044.002997] [] ? debug_smp_processor_id+0x17/0x20 [ 3044.002997] [] ? pax_randomize_kstack+0x4e/0x70 [ 3120.190032] INFO: task init:1 blocked for more than 120 seconds. [ 3120.196193] Tainted: P O 4.3.3-hardened-r4 #5 [ 3120.202762] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 3120.214654] init D ffff88003e1085b0 8 1 0 0x00000000 [ 3120.225910] ffffc9000000b3c8 0000000000000086 ffffffff9c100e10 ffffc9000000b408 [ 3120.229978] ffffc9000324b3f8 ffff880037795600 ffff88003e108000 ffffc9000000b3a8 [ 3120.235124] ffffffffc000ca27 725cb22d39b5ad1c ffff88003e108a78 ffff88003ac51290 [ 3120.240593] Call Trace: [ 3120.243197] [] ? get_parent_ip+0x10/0x50 [ 3120.246685] [] ? cv_wait_common+0x197/0x2f0 [spl] [ 3120.252121] [] schedule+0x3d/0x90 [ 3120.255406] [] cv_wait_common+0x197/0x2f0 [spl] [ 3120.259343] [] ? wait_woken+0xa0/0xa0 [ 3120.262937] [] __cv_wait+0x24/0x30 [spl] [ 3120.266350] [] txg_wait_open+0xf0/0x1c0 [zfs] [ 3120.270511] [] dmu_tx_wait+0x503/0x510 [zfs] [ 3120.274564] [] dmu_tx_assign+0xce/0x750 [zfs] [ 3120.278382] [] zfs_rmnode+0x136/0x440 [zfs] [ 3120.282352] [] ? zfs_znode_hold_exit+0x15b/0x200 [zfs] [ 3120.286997] [] zfs_zinactive+0xf2/0x150 [zfs] [ 3120.291143] [] zfs_inactive+0x7b/0x2b0 [zfs] [ 3120.295855] [] ? _raw_spin_unlock_irq+0x26/0x50 [ 3120.300036] [] zpl_evict_inode+0x46/0xa0 [zfs] [ 3120.304194] [] ? __FUNCTION__.49956+0x4e/0x4e [zfs] [ 3120.308556] [] evict+0xbf/0x1a0 [ 3120.310791] [] dispose_list+0x38/0x70 [ 3120.313318] [] prune_icache_sb+0x61/0x90 [ 3120.317149] [] super_cache_scan+0x278/0x450 [ 3120.321077] [] ? __list_lru_count_one.isra.2+0x44/0x80 [ 3120.325557] [] shrink_slab.part.40+0x3f9/0x610 [ 3120.329609] [] ? shrink_lruvec+0x625/0x720 [ 3120.333265] [] shrink_zone+0x28e/0x2c0 [ 3120.336767] [] do_try_to_free_pages+0x16b/0x490 [ 3120.339511] [] ? pfmemalloc_watermark_ok+0xbc/0xf0 [ 3120.342420] [] try_to_free_pages+0xe1/0x1c0 [ 3120.345040] [] __alloc_pages_nodemask+0x55b/0x990 [ 3120.348003] [] __do_page_cache_readahead+0x130/0x290 [ 3120.350869] [] ? find_get_entry+0x6c/0xa0 [ 3120.353651] [] ? pagecache_get_page+0x2d/0x1c0 [ 3120.356389] [] filemap_fault+0x3a3/0x460 [ 3120.358993] [] __do_fault+0x7f/0x130 [ 3120.361414] [] handle_mm_fault+0xb75/0x1750 [ 3120.364235] [] __do_page_fault+0x220/0x6d0 [ 3120.366902] [] trace_do_page_fault+0x49/0x150 [ 3120.369627] [] do_async_page_fault+0x2c/0xa0 [ 3120.372395] [] async_page_fault+0x28/0x30 [ 3120.379301] Sending NMI to all CPUs: [ 3120.382699] NMI backtrace for cpu 0 [ 3120.385057] CPU: 0 PID: 26300 Comm: z_wr_int_1 Tainted: P O 4.3.3-hardened-r4 #5 [ 3120.391254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 [ 3120.399779] task: ffff880037352040 ti: ffff880037352ab8 task.ti: ffff880037352ab8 [ 3120.405148] RIP: 0010:[] [] get_page_from_freelist+0x19c/0x6d0 [ 3120.411919] RSP: 0000:ffffc900026a3a10 EFLAGS: 00000293 [ 3120.415533] RAX: 0000000000000000 RBX: 0000000000000042 RCX: 0000000000000598 [ 3120.420570] RDX: 0000000000000598 RSI: 0000000000000000 RDI: ffffffff9ccacf80 [ 3120.425863] RBP: ffffc900026a3ac0 R08: 0000000000000042 R09: 00000000000003b7 [ 3120.427597] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000040 [ 3120.430052] R13: ffffffff9ccacf80 R14: ffffffff9ccacf80 R15: 0000000000000000 [ 3120.433261] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000 [ 3120.436877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3120.439576] CR2: 0000000603995178 CR3: 000000001c834000 CR4: 00000000001606f0 [ 3120.442775] Stack: [ 3120.444018] 0000000000000000 ffffffff9ccae548 0000000000000010 0000000000000000 [ 3120.447879] 0000000000000001 0000000000000040 0000000000000000 ffffffff9ccae548 [ 3120.453304] ffffffff9c1e1591 0000000000000042 0000000000000010 0002001000000000 [ 3120.458424] Call Trace: [ 3120.459884] [] ? try_to_free_pages+0xe1/0x1c0 [ 3120.464115] [] __alloc_pages_nodemask+0x70d/0x990 [ 3120.468594] [] _abd_alloc_scatter+0x1d5/0x3f0 [zfs] [ 3120.473059] [] ? spa_feature_is_enabled+0x83/0x100 [zfs] [ 3120.477784] [] vdev_queue_io_to_issue+0x7af/0xcb0 [zfs] [ 3120.481339] [] vdev_queue_io_done+0x1fc/0x3d0 [zfs] [ 3120.486428] [] ? __param_str_zfs_vdev_scheduler+0x40/0x40 [zfs] [ 3120.492511] [] zio_vdev_io_done+0x90/0x1c0 [zfs] [ 3120.497898] [] zio_execute+0xed/0x2f0 [zfs] [ 3120.502992] [] taskq_thread+0x2a2/0x5c0 [spl] [ 3120.508027] [] ? wake_up_q+0x80/0x80 [ 3120.512590] [] ? taskq_thread_should_stop+0x90/0x90 [spl] [ 3120.518364] [] kthread+0x10a/0x120 [ 3120.522795] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3120.528304] [] ret_from_fork+0x3e/0x70 [ 3120.532886] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3120.538161] Code: 4d 00 00 84 c0 0f 84 56 ff ff ff 48 8b 45 b8 4c 89 f7 44 8b 45 98 8b 75 d0 8b 48 18 44 89 c0 83 e0 03 49 8b 14 c6 e8 14 d4 ff ff <84> c0 75 0a f6 45 98 04 0f 84 28 ff ff ff 8b 45 ac c1 e8 08 83 [ 3120.545957] NMI backtrace for cpu 1 [ 3120.547754] CPU: 1 PID: 34 Comm: khungtaskd Tainted: P O 4.3.3-hardened-r4 #5 [ 3120.551204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 [ 3120.555745] task: ffff88003e2b9580 ti: ffff88003e2b9ff8 task.ti: ffff88003e2b9ff8 [ 3120.558978] RIP: 0010:[] [] flat_send_IPI_mask+0x79/0xc0 [ 3120.563692] RSP: 0000:ffffc90000123da0 EFLAGS: 00000046 [ 3120.566565] RAX: 0000000000000000 RBX: 0000000000000c00 RCX: 0000000000000000 [ 3120.569754] RDX: 0000000000000c00 RSI: 0000000000000002 RDI: 0000000000000286 [ 3120.572891] RBP: ffffc90000123dc0 R08: 0000000000000000 R09: 0000000000000000 [ 3120.576391] R10: ffffc90000008000 R11: ffff88003c182f00 R12: 0000000000000007 [ 3120.579478] R13: 0000000000000002 R14: 00000000000003ff R15: 00000000003fffff [ 3120.582502] FS: 0000000000000000(0000) GS:ffff88003fc80000(0000) knlGS:0000000000000000 [ 3120.585976] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3120.588621] CR2: 0000001152fbfab4 CR3: 000000001c836000 CR4: 00000000001606f0 [ 3120.591791] Stack: [ 3120.593090] 0000000000000286 ffffffff9ccc8a20 0000000000000001 ffffffff9c03fe80 [ 3120.596417] ffffc90000123dd8 ffffffff9c03feaf 000000000000f100 ffffc90000123e18 [ 3120.599665] ffffffff9c45ccb4 0000000000000000 0000000000000001 ffff88003e108000 [ 3120.602854] Call Trace: [ 3120.604331] [] ? irq_force_complete_move+0x40/0x40 [ 3120.607264] [] nmi_raise_cpu_backtrace+0x2f/0x40 [ 3120.610186] [] nmi_trigger_all_cpu_backtrace+0x254/0x260 [ 3120.612971] [] arch_trigger_all_cpu_backtrace+0x1b/0x30 [ 3120.615956] [] watchdog+0x320/0x3a0 [ 3120.618380] [] ? reset_hung_task_detector+0x20/0x20 [ 3120.621249] [] kthread+0x10a/0x120 [ 3120.623597] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3120.626569] [] ret_from_fork+0x3e/0x70 [ 3120.629064] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3120.632011] Code: c3 5f ff 80 e6 10 75 f2 44 89 e2 c1 e2 18 89 14 25 10 c3 5f ff 44 89 ea 09 da 80 cf 04 41 83 fd 02 0f 44 d3 89 14 25 00 c3 5f ff <57> 9d 0f 1f 44 00 00 48 83 c4 08 5b 41 5c 41 5d 5d 48 0f ba 2c [ 3120.639770] NMI backtrace for cpu 2 [ 3120.641541] CPU: 2 PID: 26271 Comm: z_wr_iss Tainted: P O 4.3.3-hardened-r4 #5 [ 3120.645215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 [ 3120.649849] task: ffff880034ae6b80 ti: ffff880034ae75f8 task.ti: ffff880034ae75f8 [ 3120.652577] RIP: 0010:[] [] __mutex_unlock_slowpath+0x31/0x160 [ 3120.656440] RSP: 0000:ffffc900025bb518 EFLAGS: 00000246 [ 3120.658952] RAX: 0000000000000246 RBX: ffff88003da6a800 RCX: ffff88003da6a840 [ 3120.662110] RDX: ffff88003da6a840 RSI: 0000000000000000 RDI: ffff88003da6a800 [ 3120.665309] RBP: ffffc900025bb538 R08: 28f5c28f5c28f5c3 R09: 28f5c28f5c28f5c3 [ 3120.668404] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88003da6a860 [ 3120.671548] R13: 0000000000000031 R14: 0000000000000000 R15: 0000000000000000 [ 3120.674672] FS: 0000000000000000(0000) GS:ffff88003fd00000(0000) knlGS:0000000000000000 [ 3120.678255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3120.680593] CR2: 0000000603995178 CR3: 000000001c838000 CR4: 00000000001606f0 [ 3120.683750] Stack: [ 3120.685043] ffff88003da6a800 ffff88003da6a860 0000000000000031 0000000000000000 [ 3120.688252] ffffc900025bb550 ffffffff9c820e05 0000000000000000 ffffc900025bb568 [ 3120.691478] ffffffff9c5a2537 0000000000000000 ffffc900025bb680 ffffffff9c1de21e [ 3120.695103] Call Trace: [ 3120.696593] [] mutex_unlock+0x15/0x20 [ 3120.699051] [] ttm_dma_pool_shrink_count+0x47/0x60 [ 3120.701938] [] shrink_slab.part.40+0x16e/0x610 [ 3120.704684] [] ? shrink_lruvec+0x625/0x720 [ 3120.707292] [] ? css_next_descendant_pre+0x24/0x60 [ 3120.710219] [] shrink_zone+0x28e/0x2c0 [ 3120.712684] [] do_try_to_free_pages+0x16b/0x490 [ 3120.715511] [] try_to_free_pages+0xe1/0x1c0 [ 3120.718145] [] __alloc_pages_nodemask+0x55b/0x990 [ 3120.720913] [] new_slab+0x31d/0x430 [ 3120.723339] [] __slab_alloc.constprop.70+0x34c/0x460 [ 3120.726295] [] ? spl_kmem_cache_alloc+0xa9/0xc80 [spl] [ 3120.729341] [] kmem_cache_alloc+0x1e7/0x240 [ 3120.731379] [] ? spl_kmem_cache_alloc+0xa9/0xc80 [spl] [ 3120.734486] [] spl_kmem_cache_alloc+0xa9/0xc80 [spl] [ 3120.737490] [] ? try_to_wake_up+0x46/0x400 [ 3120.740210] [] ? spa_config_held+0x8e/0xb0 [zfs] [ 3120.743155] [] zio_create+0x8a/0x660 [zfs] [ 3120.745763] [] zio_vdev_child_io+0xf2/0x190 [zfs] [ 3120.748625] [] ? vdev_config_sync+0x1e0/0x1e0 [zfs] [ 3120.751672] [] vdev_mirror_io_start+0xac/0x1e0 [zfs] [ 3120.754837] [] ? vdev_config_sync+0x1e0/0x1e0 [zfs] [ 3120.757836] [] zio_vdev_io_start+0x211/0x510 [zfs] [ 3120.760803] [] zio_nowait+0xd6/0x2b0 [zfs] [ 3120.763507] [] vdev_mirror_io_start+0xb4/0x1e0 [zfs] [ 3120.766596] [] ? vdev_config_sync+0x1e0/0x1e0 [zfs] [ 3120.769662] [] zio_vdev_io_start+0x324/0x510 [zfs] [ 3120.772593] [] zio_execute+0xed/0x2f0 [zfs] [ 3120.775579] [] taskq_thread+0x2a2/0x5c0 [spl] [ 3120.778795] [] ? wake_up_q+0x80/0x80 [ 3120.781240] [] ? taskq_thread_should_stop+0x90/0x90 [spl] [ 3120.784475] [] kthread+0x10a/0x120 [ 3120.786873] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3120.789857] [] ret_from_fork+0x3e/0x70 [ 3120.792685] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3120.795740] Code: 48 89 e5 41 56 41 55 41 54 53 48 89 fb 8b 35 af 1a 90 00 85 f6 75 12 65 8b 05 a4 8f 7e 63 a9 00 ff 1f 00 0f 85 cd 00 00 00 9c 58 <0f> 1f 44 00 00 49 89 c5 fa 66 0f 1f 44 00 00 4c 8d 63 08 31 c0 [ 3120.804281] Kernel panic - not syncing: hung_task: blocked tasks [ 3120.805036] CPU: 1 PID: 34 Comm: khungtaskd Tainted: P O 4.3.3-hardened-r4 #5 [ 3120.805036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 [ 3120.805036] ffff880000000002 cffb8bf6abc45dcf 0000000000000000 ffffc90000123e40 [ 3120.805036] ffffc90000123da0 ffffffff9c457d93 ffffffff9ca728a9 ffffc90000123e30 [ 3120.805036] ffffffff9c1c7497 0000000000000008 ffffc90000123e40 ffffc90000123dd0 [ 3120.805036] Call Trace: [ 3120.805036] [] dump_stack+0x4e/0x8b [ 3120.805036] [] panic+0xd7/0x221 [ 3120.805036] [] watchdog+0x32e/0x3a0 [ 3120.805036] [] ? reset_hung_task_detector+0x20/0x20 [ 3120.805036] [] kthread+0x10a/0x120 [ 3120.805036] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3120.805036] [] ret_from_fork+0x3e/0x70 [ 3120.805036] [] ? kthread_create_on_node+0x1c0/0x1c0 [ 3120.805036] Kernel Offset: 0x1b000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 3120.805036] Rebooting in 60 seconds..