Can ZFS match the performance of NTFS?

[steel3d]

First of all let me start by saying that I really commend this project! It works pretty well, other than a slow start due to lack of docs, and some bugs and lacking features. But it looks like it's getting there!

My issue now is that I'm seeing less than ideal disk performance with ZFS. I want to make sure it's not some misconfiguration on my side, before I put the pool into full service.

Basically, folder copies are 2-3x faster on NTFS, and CrystalDiskMark is showing lower performance on ZFS as well. I'm testing with identical drives in NTFS and ZFS.

To start with CrystalDiskMark:

D is NTFS SSD
J is ZFS SSD
L is NTFS NVME
K is ZFS NVME

Note that random access is quite a bit slower with ZFS.
Sequential is slower on NVME, but faster than theoretical speed (and equivalent to NVME) on SSD.
It's also surprising how all of the ZFS numbers are basically equivalent between SSD and NVME. It feels like CPU is the bottleneck. I'm running a Threadripper 1950x with 64GB RAM.

I also did real world tests copying some large folders, one with lots of small files, and one with 3 large video files.
In the case of small files, NTFS is 2x faster than ZFS. Interestingly, the performance of SSD is the same as NVME on this test even with NTFS.
For large files, NTFS NVME is 3x faster, and NTFS SSD is 2x faster than ZFS. Also, the progress of the transfer is very erratic on ZFS, vs flat on NTFS. At the end of each video file the transfer speed drops to 0MB/s for a few seconds, then resumes for the next file. The funny thing is that even with NTFS the speed on NVME tops out at 1.6GB/s, instead of the 3.2GB/s benchmark speed (due to sharing PCIe lanes between the source and destination drives?).

I tried tuning several things, but I can't get any improvement in performance. I played with recordsize, ashift, compression, cache.

Any improvement to be had? Is it unrealistic to expect performance to match NTFS at this stage? Or ever?

[lundman]

Should be mentioned that I have certainly not had time to get to any performance work, since it has been a scramble to get things stable and adding more features. Once it is stable, developers tend to look at speeding things up. Feel free to give it a try, although my usual tools, like FlameGraph with dtrace probably will not work with Windows.

[guenther-alka]

Low performance propably due the 512B physical setting for the ZFS pool.
#345 (comment)

We need to wait for a 4k setting like Windows is using on ntfs.

[steel3d]

Thanks for the answers! The SSD reports 512B but the perf is lower than NTFS anyway.

[besterino]

Once I find a few hours spare time I will try to tinker with the sourcecode to replace the currently "hardwired" 512B reporting to windows to 4K.

For now, I have seen basically the same results as you did. In addition to NTFS I also ran ReFS benchmarks, both with checksum on data (to have somewhat similar behaviour as ZFS) as well as without.

What I found interesting/promising: If you disable the NVME's write cache, 4k write performance on ZFS is on par if not better compared to ReFS or NTFS (except for reads with high queue depth). So in my mind that means there should be room for performance improvements,

I ran my tests on a Threadripper 3960X, 64GB RAM and 4x Samsung 980 Pro (yeah, don't like them either, but the only drives free to play with atm).

Setup:

PhyiscalDrive0: NTFS (standard settings) = N:
PhyiscalDrive1: ZFS = F:
PhyiscalDrive3: ReFS (FileIntegrity=enabled, i.e. data checksums) = R:

ZFSonWindows: most recent build OpenZFSOnWindows-debug-2.2.99-9-gf886377b36.exe

Pool setup as:
zpool create -O casesensitivity=insensitive -O compression=off -O atime=off -o ashift=12 zfsroot Physicaldrive1

Dataset created as:
zfs create zfsroot/zfsvol

WriteCache enabled:

WriteCache disabled (note most abyssmal results for 1Q/1T ReFS and NTFS for 4K random writes!):

P.S.: ZFS Cache r0xx even on windows. ;)

[besterino]

Here's the original (German) thread, but not really more info there: https://www.hardwareluxx.de/community/threads/napp-it-f%C3%BCr-zfs-on-windows.1349039/post-30286659

[sskras]

Thanks, that's a very interesting research.

If you disable the NVME's write cache, 4k write performance on ZFS is on par if not better compared to ReFS or NTFS (except for reads with high queue depth). So in my mind that means there should be room for performance improvements,

[...]

WriteCache enabled:

I guess this means the improvements' room resides somewhere in the field of NT Cache Manager:
https://www.linkedin.com/pulse/inside-windows-cache-manager-artem-baranov/

EDIT. Maybe that would be a better intro (not sure if needed:)
https://www.osronline.com/article.cfm%5Eid=167.htm

Tried searching for that term in WinBTRFS repo. The only reasonable match, a TODO:
https://github.com/maharmstone/btrfs/blob/01c45ad5ea949060c9821f63585682d50f5af58e/src/write.c#L3329

[lundman]

I threw in some code that looks up ashift to use instead of 512. If you can, check out OpenZFSOnWindows-debug-2.2.99-13-gfddfb6aeb5.exe. I did the most obvious places, but there are quite a few ways to query that.

[besterino]

Awesome! Highly appreciated that you spent precious time on this! Have already changed my testbox (currently comparing ReFS-2SSD-mirror vs ZFS-2SSD-mirror) and can‘t wait to check whether your changes make a difference. Unfortunately, will only be able to test again tomorrow at the earliest.

[besterino]

Ok, couldn't wait to do some testing: fsutil still shows 512B across the bank for a zfs pool or dataset (mounted separately with a driveletter) where a NTFS/ReFS volume shows 512B logical and 4k physical sectors.

Performancewise I could also see no difference. Tried both single drive pool as well as 2-drive-mirror, no palpable difference between -9 or -13:

[lundman]

It uses the ashift now, so if you create the pool with ashift=12 it should return 4096

[besterino]

Weird. Installed most recent built again. Cleaned disk with previous pool (already created with ashift=12) with diskpart and then created the pool again with ashift=12 with the following command (same as before):

zpool create -O casesensitivity=insensitive -O compression=off -O atime=off -o ashift=12 zfsroot Physicaldrive1

NOW it does return 4k!

Performance remains the same though.

Sidenote: Most recent -13.exe install throws an error at the end and Windows Defender identifies & deletes the most recent zpool.exe as malware.

[besterino]

Here's another compare with another bench-tool (atto), which checks performance with different I/O sizes.

Here I'm comparing 2x NVMe mirrors again.

Test file: 256MB

Test file: 8GB

With write cache enabled:

If I am spamming too much, let me know. If there's other tests / benchmarks you'd like to see, also let me know.

[lundman]

Ah yep - I didn't think that 4096 would fix much, but it is good to rule out at least.

No idea about defender, does that here as well at random times. Especially odd when zpool.exe hasn't had any code changes in the last many releases.

[besterino]

Can't thank you enough for bringing ZFS to Windows.

In my opinion ZFS performance is pretty decent already, considering that no performance optimizing has been done yet. In particular, if you look at the performance without write cache enabled compared to ReFS (see atto results above). At least if I set fileintegrity=true, which enables checksums on data in addition to ReFS's standard setting "off" =only on metadata. So with fileintegrity enabled, performance of ReFS and ZFS should be more fairly comparable.

Looking at write speeds with "bypass write cache" enabled, ReFS is destroyed. But I am not sure whether settings like bypass write cache etc. really work with ZFS under the hood - differences between enabed/disabled are minimal (if not in the wrong direction).

However, even if I set sync=always on the ZFS pool, it's still faster than the ReFS volume: