bad performance/OOM killer under memory pressure

[nielsdec]

It seems having a small ARC often performs better then having a large ARC and requiring memory reclaim. It seems the spl slab reclaim is mainly freeing slabs that are immediately needed again, and the arc reclaim is ineffective because it just causes fragmentation in the slabs. Since indirect reclaim doesn't stop ARC growing, it seems there is continuing fight over memory.

In kernel 3.8, this shows as high loads for spl_kmem_cache and kswapd, and a drop in performance.

In kernel 3.11, the OOM killer gets quickly invoked in case of memory pressure. There are likely 2 reasons for this: 1 vmalloc performance has increased hugely, and so the arc can grow much faster (ARC growth was limited by vmalloc performance). 2. Changes in kswapd has made it less aggressive, and so memory is reclaimed slower.

It seems the reclaim logic needs to be improved to better work with kswapd and be more effective. I suspect ARC growth needs to stopped when multiple indirect reclaims are happening. (In 3.11 my dd gets killed by the OOM killer without a single direct reclaim happening)

[ryao]

Part of what is happening is that there are three layers involved. The Linux kernel's memory allocator, the SPL SLABs and ZFS (mainly ARC). Freeing space from ARC frees it to the SPL. The SPL does SLAB allocation, such that it can only free memory to the kernel's memory allocator when all objects in a SLAB are freed. Therefore, long-lived objects can prevent fairly substantial amounts of memory from being reclaimed.

The solution to issue #75 should fix that. That is scheduled for 0.9.0, but I am already working on it. I had hoped to have it finished last week, but I simultaneously overestimated the amount of work involved and had hardware problems that prevented me from working on it. I plan/hope/expect to finish it this year.

[tuxoko]

@ryao Hi, I'm very interested in this issue. What is the status of your progress? Is there anything I can help? If you have any public/private branch, I'd like to hack on it. Thanks.

[behlendorf]

Closing, WIP patches in #4332.