Max heap memory used by Analysis App can be greater than RAM causing UI crashes

[patrickdalla]

When opening IPED from "exe", it starts JVM with max heap size, "-Xmx" parameter, with value of 30G. It seems that JVM believes in this value, even if the real RAM installed is less than 30G, postponing some memory disallocation and halting IPED.

This IPED halting occurred in a case reported by Werner, during background timeline graph cache creation on a machine with 8GB of installed RAM, but testing the same case, I found that occurred also when toogling to timeline table view, even with timeline graph disabled, both processes that demand memory. The case worked when configured to use max heap size of 8GB, even with 3GB, but halted when executed via "exe".

[patrickdalla]

As nassif suggested, this max heap size can be corrected in Bootstrap class. I found that the following code returns the real installed mem size.
((com.sun.management.OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean()).getTotalPhysicalMemorySize()

[lfcnassif]

To make it more clear, when @patrickdalla said halted it means the JVM crashed because it tried to allocate more memory than available in the machine, when it could just run GC.

I was thinking to implement a dynamic memory allocation scheme, since now the Bootstrap class can give us more control over the max used heap. If user doesn't provide -Xmx and the machine has:

• 8GB RAM, use -Xmx4G (or 6G?)
• 16GB RAM, use -Xmx8G
• 32GB RAM, use -Xmx16GB
• 64GB RAM or more, use -Xmx32G

The default of most JVMs is to use just 1/4 of the RAM. Opinions?

[wladimirleite]

• 8GB RAM, use -Xmx4G (or 6G?)

I would use 4G in a 8GB machine.

[lfcnassif]

Thanks @tc-wleite. Half the RAM seems reasonable. When processing, we need memory for other processes. When analyzing, we need memory to generate reports.

[lfcnassif]

Reopening, the switch introduced in #1713 should also handle lowercase units: g, m, k. If they are used, current code is interpreting the value always as bytes.

[patrickdalla]

Maybe this solution can cause some problem in Terminal Servers environment where the real RAM is huge, but there should be some memory usage limit between the different users, as more than one user can use IPED at the same time.
For example, in our TS, the machine has 192GB. This solution would start iped with Xmx as 48GB.

What do you think @lfcnassif? Maybe we can discuss this in a new issue, specific for Terminal Server usage.

[wladimirleite]

@lfcnassif, just a minor detail...
As discussed many times, including in #2047 (reply in thread), it is better to use less than 32 GB (not "-Xmx32G"), or much higher values (like more than 48G in machines with hundreds GB of RAM).
Using JOL, I was able to find out that the exact threshold (in my environment) is 32736M (= 32GB - 32MB).
Usually I open the analysis UI through the EXE and my PC has 96 GB of RAM, so it will use -Xmx32G (%50 of RAM, limited to 32GB), which disables compressed OOPs, wasting some memory (which can be critical for large cases).
I suggest changing the maximum -Xmx value automatically set from 32G to 32500M.