Benchmark tool "fio" creates an unimportable pool. Kernel panic: Out of memory

[mabod]

System information

Type	Version/Name
Distribution Name	Manjaro
Distribution Version	rolling
Linux Kernel	4.16 ; 4.15
Architecture	amd64
ZFS Version	0.7.8-1
SPL Version	0.7.8-1
fio	3.6

Describe the problem you're observing

When I run fio (version 3.6) with two times RAM size on my RAID10 pool, the kernel panics during next boot saying: Out of memory.

Describe how to reproduce the problem

A few days ago I create issue #7466: "zfs freezes kernel during boot". In the meantime I did more tests and can reproduce the freeze. The issue is with the benchmark tool fio.

I have a separate zfs volume to do benchmarks: zstore/BENCHMARK. I use the following fio config for the test:

# cat fio-rw-generic.fio

[global]
ioengine=libaio
invalidate=1
iodepth=1
refill_buffers
randrepeat=0
fallocate=none
size=${SIZE}

[read]
bs=1m
stonewall
rw=read

[write]
bs=1m
stonewall
rw=write

I run this test two times in a row:

cd /mnt/zstore/BENCHMARK

export SIZE=64G
fio fio-rw-generic.fio
sleep 3

export SIZE=64G
fio fio-rw-generic.fio

The PC has 32 GB RAM. The test runs fine, but when I shutdown the PC and restart afterwards it hangs during boot. The kernel panics saying: Out of Memory. A photo of the kernel panic is the only log I can provide:

I can only recover by booting a kernel without zfs modules, install the zfs modules manually, load them and do a forced import.

The pool is a RAID10 with 4 identical drives:

# zpool status zstore
  pool: zstore
 state: ONLINE
  scan: scrub repaired 0B in 5h43m with 0 errors on Sat Apr 21 00:08:23 2018
config:

	NAME                 STATE     READ WRITE CKSUM
	zstore               ONLINE       0     0     0
	  mirror-0           ONLINE       0     0     0
	    WD-WCC4E5HF3P4S  ONLINE       0     0     0
	    WD-WCC4E1SSP28F  ONLINE       0     0     0
	  mirror-1           ONLINE       0     0     0
	    WD-WCC4E1SSP6NC  ONLINE       0     0     0
	    WD-WCC7K7EK9VC4  ONLINE       0     0     0

errors: No known data errors

It has still plenty of space available:

# zfs list zstore/BENCHMARK 
NAME               USED  AVAIL  REFER  MOUNTPOINT
zstore/BENCHMARK   176G  3,58T   176G  /mnt/zstore/BENCHMARK

The zfs parameters for zstore/BENCHMARK are:

# zfs get all zstore/BENCHMARK
NAME              PROPERTY              VALUE                  SOURCE
zstore/BENCHMARK  type                  filesystem             -
zstore/BENCHMARK  creation              Di Jan 23 14:40 2018   -
zstore/BENCHMARK  used                  176G                   -
zstore/BENCHMARK  available             3,58T                  -
zstore/BENCHMARK  referenced            176G                   -
zstore/BENCHMARK  compressratio         1.00x                  -
zstore/BENCHMARK  mounted               yes                    -
zstore/BENCHMARK  quota                 none                   default
zstore/BENCHMARK  reservation           none                   default
zstore/BENCHMARK  recordsize            1M                     local
zstore/BENCHMARK  mountpoint            /mnt/zstore/BENCHMARK  inherited from zstore
zstore/BENCHMARK  sharenfs              off                    default
zstore/BENCHMARK  checksum              on                     default
zstore/BENCHMARK  compression           off                    local
zstore/BENCHMARK  atime                 on                     inherited from zstore
zstore/BENCHMARK  devices               on                     default
zstore/BENCHMARK  exec                  on                     default
zstore/BENCHMARK  setuid                on                     default
zstore/BENCHMARK  readonly              off                    default
zstore/BENCHMARK  zoned                 off                    default
zstore/BENCHMARK  snapdir               hidden                 default
zstore/BENCHMARK  aclinherit            restricted             default
zstore/BENCHMARK  createtxg             17                     -
zstore/BENCHMARK  canmount              on                     default
zstore/BENCHMARK  xattr                 sa                     inherited from zstore
zstore/BENCHMARK  copies                1                      default
zstore/BENCHMARK  version               5                      -
zstore/BENCHMARK  utf8only              off                    -
zstore/BENCHMARK  normalization         none                   -
zstore/BENCHMARK  casesensitivity       sensitive              -
zstore/BENCHMARK  vscan                 off                    default
zstore/BENCHMARK  nbmand                off                    default
zstore/BENCHMARK  sharesmb              off                    default
zstore/BENCHMARK  refquota              none                   default
zstore/BENCHMARK  refreservation        none                   default
zstore/BENCHMARK  guid                  7947132819733491852    -
zstore/BENCHMARK  primarycache          all                    default
zstore/BENCHMARK  secondarycache        all                    default
zstore/BENCHMARK  usedbysnapshots       0B                     -
zstore/BENCHMARK  usedbydataset         176G                   -
zstore/BENCHMARK  usedbychildren        0B                     -
zstore/BENCHMARK  usedbyrefreservation  0B                     -
zstore/BENCHMARK  logbias               latency                default
zstore/BENCHMARK  dedup                 off                    default
zstore/BENCHMARK  mlslabel              none                   default
zstore/BENCHMARK  sync                  standard               default
zstore/BENCHMARK  dnodesize             legacy                 default
zstore/BENCHMARK  refcompressratio      1.00x                  -
zstore/BENCHMARK  written               176G                   -
zstore/BENCHMARK  logicalused           176G                   -
zstore/BENCHMARK  logicalreferenced     176G                   -
zstore/BENCHMARK  volmode               default                default
zstore/BENCHMARK  filesystem_limit      none                   default
zstore/BENCHMARK  snapshot_limit        none                   default
zstore/BENCHMARK  filesystem_count      none                   default
zstore/BENCHMARK  snapshot_count        none                   default
zstore/BENCHMARK  snapdev               hidden                 default
zstore/BENCHMARK  acltype               posixacl               inherited from zstore
zstore/BENCHMARK  context               none                   default
zstore/BENCHMARK  fscontext             none                   default
zstore/BENCHMARK  defcontext            none                   default
zstore/BENCHMARK  rootcontext           none                   default
zstore/BENCHMARK  relatime              on                     inherited from zstore
zstore/BENCHMARK  redundant_metadata    all                    default
zstore/BENCHMARK  overlay               off                    default

[mabod]

This issue is old. I can not reproduce it anymore. Nowadays I am doing frequent fio benchmarks without any issue. Therefore I am closing this issue.