Unexpectedly low performance of VMs with openzfsonosx backing store

[mauricev]

I have a mirrored pair of regular SATA drives in a thunderbolt enclosure under Mojave. I try to run virtual machines from them. Machines that may run under Virtualbox or VMWare Fusion. They run like elderly turtles with arthritis. Copying outside the virtual machine seems fine. It's that there's some interaction between the virtual machine environment and Mac ZFS that seems to be gimping them.

[Bottacco]

Well, running virtual machines from spinning disks is always very slow. If you have the opportunity to run them from a SSD try it and you will see the difference. Running from a mirrored pair doesn't give you better performance, because VMs change constantly while in execution, so the small gain you could have in speed while reading from both SATA drives is lost in the constant writing to disk.
I don't think the problem is with ZFS. Another thing you can try is to reformat those two drives with disk utility (no ZFS) and create a mirror from CLI (the standard mirroring from macOS/OSX via diskutil in Terminal) and compare the speed, but I am pretty sure it will be almost the same if not worse.

[rottegift]

You haven't told us anything about versions ("zpool version") or whether you're in a dataset, what compression and checksum algorithm you're using, and so forth, so it's hard to pin down what might be at the root of your performance problem.

That said, our mmap performance in datasets is far from great. You might try moving your VMs into a zvol instead:

$ export size=1000g ### for example
$ sudo zfs create -o compression=lz4 -o volblocksize=16k -V $size pool/zvolname
$ ls -l /var/run/zfs/zvol/dsk/pool/zvolname
lrwxr-x-r-x 1 root ... .../pool/zvolname -> /dev/disk44 ### for example
$ diskutil partitiondisk disk44 1 GPT APFS VMVOL R

This creates an APFS volume named "VMVOL" into which you can copy your VMs, using the zvol as a source.

You can experiment with volblocksize - you are likely to get different (not necessarily better) performance as you change the powers-of-two down (4k,8k) or up (32k,64k,128k). It's hard to estimate because lz4 is on modern mac systems is very very fast and the msync() calls from the VM software are sufficiently rare (and handled by the APFS code rather than by our code) that larger volblocks aren't partially-rewritten very often, so there can be a performance gain from larger volblocks.

If the performance of your VMs stored in a zvol rather than a dataset is better (or worse), please add a note to this discussion.