OpenZFS 9188 - increase size of dbuf cache to reduce indirect block decompression

[behlendorf]

Description

With compressed ARC (OpenZFS 6950) we use up to 25% of our CPU to decompress
indirect blocks, under a workload of random cached reads. To reduce this
decompression cost, we would like to increase the size of the dbuf cache so
that more indirect blocks can be stored uncompressed.

If we are caching entire large files of recordsize=8K, the indirect blocks
use 1/64th as much memory as the data blocks (assuming they have the same
compression ratio). We suggest making the dbuf cache be 1/32nd of all memory,
so that in this scenario we should be able to keep all the indirect blocks
decompressed in the dbuf cache. (We want it to be more than the 1/64th that
the indirect blocks would use because we need to cache other stuff in the dbuf
cache as well.)

In real world workloads, this won't help as dramatically as the example above,
but we think it's still worth it because the risk of decreasing performance is
low. The potential negative performance impact is that we will be slightly
reducing the size of the ARC (by ~3%).

Porting Notes:

• Added modules options to zfs-module-parameters.5 man page.
• Preserved scaling based on target ARC size rather than max ARC size.

Authored by: George Wilson [email protected]
Reviewed by: Dan Kimmel [email protected]
Reviewed by: Prashanth Sreenivasa [email protected]
Reviewed by: Paul Dagnelie [email protected]
Ported-by: Brian Behlendorf [email protected]

Motivation and Context

OpenZFS-issue: https://www.illumos.org/issues/9188
OpenZFS-commit: openzfs/openzfs#564
Upstream bug: DLPX-46942

And issues #6880 and #7255.

How Has This Been Tested?

Performance testing done for the OpenZFS PR, pending results from Buildbot .

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted and contain Signed-off-by.
• Change has been approved by a ZFS on Linux member.

[codecov]

Codecov Report

Merging #7273 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #7273      +/-   ##
==========================================
- Coverage   76.35%   76.35%   -0.01%     
==========================================
  Files         327      327              
  Lines      103874   103876       +2     
==========================================
+ Hits        79314    79315       +1     
- Misses      24560    24561       +1

Flag	Coverage Δ
#kernel	76.19% <66.66%> (+0.01%)	⬆️
#user	65.51% <100%> (-0.21%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2705ebf...097905b. Read the comment docs.

[behlendorf]

@kpande no sorry. Although, it's not clear to me which issue I intended to reference.