[Feature Request] Support the evict on close feature in parallel HDF5

[jhendersonHDF]

Context from John Mainzer for this discussion:

The objective of the evict on close feature is to evict from the metadata cache all metadata associated with the target object (i.e. data set, group) from the metadata cache when that object is closed by the user, and thereby minimize HDF5 library footprint.

However, this creates a problem in the parallel case, as on the one hand, all metadata caches must contain the same set of dirty metadata, and on the other hand, closing an object need not be a collective operation.

At present, we solve this problem by disabling the feature in parallel. However, while we can't implement Evict on Close in parallel exactly as we do in serial, we can come close via the following:

1. Evict the clean entries associated with the target object on close.

2. Mark the dirty entries and move them to the end of the LRU. At the next sync point, flush and evict them. Observe that this will require some changes in our existing sync point code, as at present, while we flush entries at a sync point, we have no facility for evicting them as well.