Enable "small file" data to be excluded from L2ARC

[pashford]

Describe the feature would like to see added to OpenZFS

I believe it would be useful to have a per-pool flag to disable placing data residing in a Small File VDEV into the L2ARC.

How will this feature improve OpenZFS?

The use case for this feature is when both L2ARC and Small Files are kept on SSDs of a similar performance. Under these circumstances, the time to get the data from L2ARC is similar to the time to get the data from the Small File VDEV. By not allowing Small File data into the L2ARC, it becomes more able to cache data from spinning disk.

In addition, the L2ARC segment size will be more consistent. By removing the smallest files from consideration for L2ARC, the average block size will increase, causing less memory to be used for L2ARC headers.

[behlendorf]

This was actually a change we discussed making to the default behavior in the original implementation. It looks like it was never included in the finalized version, however I think it's still a good idea. Specifically, any block which is stored on a dedicated "special allocation class" vdev should not be written to the L2ARC.

For anyone interested in picking up this work it looks pretty straight forward. We'd want to update the DBUF_IS_L2CACHEABLE macro so it considers these dbufs to be non-l2arc cacheable. This will prevent them from ever being written to the L2ARC. The blkptr is available in the dbuf and can be used to determine where it's stored in the pool. One option we discussed was adding a new value to the "secondarycache" property to control this. However, it seems like this would always be a good idea as long as we can assume the special vdevs are roughly the same performance as the L2ARC vdev.