L2ARC interaction with Metadata Allocation Class

[adilger]

Per recent discussion with @behlendorf and @ahrens , if the Metadata Allocation Class (issue #5182) special class is located on a flash device and the system is also using flash for L2ARC, it likely doesn't make sense to cache metadata in the L2ARC, since fetching it from the special device will be just as fast as the L2ARC and avoids writes to the L2ARC. In this case, the L2ARC can be used entirely for caching data from non-special VDEVs instead of metadata.

However, using the secondarycache=data option implies caching both data and metadata, and there is no way to specify only data should be cached. It probably makes more sense to have an option like secondarycache=dataonly, or possibly secondarycache=nonspecial to allow L2ARC to cache anything that is not in the special class. The latter would work better with the various options that add/remove other bits from the special class (DDT, small files, etc), so that anything that is not included in the special class will still be cached in L2ARC.

It might make sense to default to using secondarycache=nonspecial if the special class is on a flash (non-rotational) device.

[allanjude]

Has any code been created for this?

[behlendorf]

@allanjude no, I don't believe anyone has started working on this yet.

[AlurianNighthawk]

As we already have:
secondarycache=all (metadata and data)
secondarycache=metadata (metadata not data)
It is probably reasonable to assume that secondarydata=data would exclude metadata.

[adilger]

There is a patch #12285 off of issue #11761 that may address this issue.

[dioni21]

I would add that an option secondarycache=data could not be the ideal.

Special and Cache vdevs are pool related, not dataset related. So, any option regarding this, IMHO, should be in the pool, not in the dataset.

I imagine a pool property specifying that any data read from an Special vdev should not go to L2ARC. This property is "read-only compatible".

[mattventura]

There is a patch #12285 off of issue #11761 that may address this issue.

I'm reading through the all the issues related to this, but I can't seem to figure out whether or not this issue is actually addressed. It seems like it is, but I don't know enough of the lower-level terminology to say with certainty.

[mwpastore]

Yes @mattventura the feature is available as a module parameter as of a release or two ago