Eliminate drbbdup bookkeeping for single-case-forever uses

[derekbruening]

With drbbdup inside drmemtrace (#3995) we're seeing drbbdup use
an extra ~35MB ("Peak vmm virtual memory in use" rstat) for ~150K blocks;
so ~140MB for ~600K blocks. This is not horrible, but is not insignificant,
and is unnecessary: if the user promises to never use more
than just one case (which memtrace can certainly do based on its own
runtime options at init time) drbbdup shouldn't need any bookkeeping.

[johnfxgalea]

This seems okay with me. Note, it would require setting up dup options again for translations and traces, which might be expensive with respect to the user's use-case.

[derekbruening]

Note, it would require setting up dup options again for translations and traces, which might be expensive with respect to the user's use-case.

I'm not sure I understand: if you could spell this out?

[derekbruening]

It is now clear that this overhead is the cause for out-of-memory failures we're seeing in a proprietary app: it has 2.5M blocks, so that's ~583MB extra just from drbbdup tracking even when duplication is turned off. We had vmcode size set to 1G, and client heap is reachable for backward compatibility: meaning that drbbdup's bookkeeping took up over half the limited reachable space. We can bump to 2G, but that's the absolute max and it precludes -vm_base_near_app and so incurs overheads.

Maybe we should change the client heap default, or change drbbdup to explicitly request unreachable heap.

[johnfxgalea]

Note, it would require setting up dup options again for translations and traces, which might be expensive with respect to the user's use-case.

I'm not sure I understand: if you could spell this out?

If I understood you correctly, you would like to remove drbbdup_manager_t structures, which are associated with basic blocks for book-keeping purposes, to improve memory performance. This is possible, provided dynamic handling is not enabled (although I believe I can optimise the structure's memory footprint when enabled as well).

However, this requires setting up a temporary manager, which in turn invokes a user callback, upon every basic block event, including for translations and traces. The manager is thrown away once instrumentation phases are completed. Currently, drbbdup limits this with the use of a hash-table, and only invokes the callback if the manager lookup fails.

[johnfxgalea]

BTW, to be clear, the point I mentioned is not a blocker; just an observation.

[derekbruening]

The proposal is to remove the drbbdup_manager_t structures only when there is zero duplication: i.e., when drbbdup is doing nothing. For drmemtrace, the normal mode is to not duplicate at all, and we'd like to avoid paying for having drbbdup linked in and available for runtime options to turn on when it is not actually turned on.

[johnfxgalea]

I think the approach I mentioned above caters for zero dups as well. I see that your approach is to use unreachable heap; this is a good first step.

[derekbruening]

Even with PR #5417 we're seeing a lot of reachable heap still in use. Looking at drbbdup's per-thread heap, it is much larger than expect: drbbdup_per_thread.hit_counts is 64K*2=128K bytes. We have apps with 10K threads, so that's 1GB right there! As reachable heap that's half the max 2GB you can have. This needs to be made unreachable too, and probably that giant array should be allocated dynamically (since our use case doesn't need it).

[johnfxgalea]

Lazy allocation is okay. I don't think making it unreachable is possible as the code in the cache directly accesses it to keep track of hits. For now, lazy allocation should mitigate the issue for your use-case as it should be enabled only for dynamic case handling.

[derekbruening]

Lazy allocation is okay. I don't think making it unreachable is possible as the code in the cache directly accesses it to keep track of hits. For now, lazy allocation should mitigate the issue for your use-case as it should be enabled only for dynamic case handling.

Hmm, I'm looking at drbbdup_insert_dynamic_handling() where it seems to load from the pointer stored in a TLS slot into a register. So that usage can handle unreachable heap. Is there another usage somewhere that is using PC-relative addressing (or an absolute address)?

[johnfxgalea]

So that usage can handle unreachable heap.

Okay, that works then!