Metadata not getting evicted from ARC

[Hydrar]

My zfs system has after 17 days uptime managed to use more metadata space in ARC than arc_meta_limit, while simultaneously also going above the ARC max
echoing 3 to drop_caches does nothing, it only seemed to jump up and down ~60MB in metadatasize but never change from that. System crashed last night from OOM
arcstats: http://bpaste.net/show/149591/
kmem/slab: http://bpaste.net/show/149597/

Kernel 3.10.17-gentoo
ZoL ebuild 0.6.2-r2

[behlendorf]

Thank you! The logs you attached show something I've suspected for a while. The ARC is largely filled by dnodes and for some reasons they aren't being reclaimed. If you get in the situation again and have the dbufstat.py scripts in the gentoo packaging it would be useful to run both dbufstat.py -x and dbufstat.py -bx to dump the ARC contents. That may help explain why the dnodes can't be released.

[Hydrar]

I will run it if I see it happening again and report back

[lundman]

My apologies if this is not relevant but we had similar issues on OSX for metadata not being evicted.

In the code snippet:

arc_adjust_meta(int64_t adjustment, boolean_t may_prune)
{
    int64_t delta;

    if (adjustment > 0 && arc_mru->arcs_lsize[ARC_BUFC_METADATA] > 0) {
        delta = MIN(arc_mru->arcs_lsize[ARC_BUFC_METADATA], adjustment);
        arc_evict(arc_mru, 0, delta, FALSE, ARC_BUFC_METADATA);
        adjustment -= delta;
    }

    if (adjustment > 0 && arc_mfu->arcs_lsize[ARC_BUFC_METADATA] > 0) {

The code will try the first arc_evict() as expected, but if arc_evict() can not evict something (for whatever reason), it still removes delta from the adjustment. We found adjustment went to zero before the following 3 ifs, making them effectively dead code.

For a quick hack, we keep the value of adjustment between the 4 ifs in this function, so that metadata eviction happens (perhaps 'too much'). Possibly the more correct answer is if arc_evict() returned the number of bytes successfully evicted.

[behlendorf]

@lundman Nice observation, I think it's absolutely relevant and explains a great deal. Although since adjustment is reset after the first two if conditionals the effect would be to prevent any metadata reclaim while allowing data reclaim.

I've put together an initial patch for testing which takes your suggestion and modified arc_evict() to return the number of bytes evicted. The function was already calculating this so it's pretty straight forward. This also gives us a chance to cleanup the arc_evict() API which isn't as clear as it could be. I'll post a patch after some sanity testing.

[behlendorf]

Closing as out of date. Much as changed in the ARC since late 2013.