High cpu usage of dbuf_evict for small random reads

[woquflux]

System information

Type	Version/Name
Distribution Name	Redhat
Distribution Version	7.2
Linux Kernel	3.10.0-327.el7.x86_64
Architecture
ZFS Version	0.6.4.2
SPL Version	0.6.4.2

Describe the problem you're observing

Terrible Performance with SSD x 18 (Intel S3500)

Describe how to reproduce the problem

CPU usage of rngd is 100%

Include any warning/errors/backtraces from the system logs

[behlendorf]

This was addressed in the zfs-0.7 series. Alternately if you must run zfs-0.6. you should be able to cherry pick these two patches openzfs/spl@0b43696 and openzfs/spl@0d26756.

[woquflux]

There is another problem in the 0.7.3, dbuf_evict process CPU usage is 100%

90,000 IOPS(8K,random read) is maximum

[behlendorf]

@woquflux that's definitely something we're going to want to look in to further optimizing. This particular small random read test case is pushing that dbuf cache eviction code to the limit on your system. Can you post the output of perf top -p <dbuf_evict pid> from you system when it's running this workload. That should let us isolate exactly which functions are consuming those cycles and why.

I've also changed the title of this issue to reflect the second issue you've mentioned rather than opening a new issue for it.

[woquflux]

@behlendorf Thanks for your reply. I had enough devices for test if you want more information

volblocksize=4k, by the way.

[behlendorf]

@woquflux thanks for the profiling data.

[richardelling]

While not a solution, we are interested in understanding whether and how we can use dbufstats to track dbuf-related pathologies. Could you add the output of cat /proc/spl/kstat/zfs/dbufstats before and after experiments?

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.