[GR-52400] Revise heuristic for memory usage of Native Image build pr…

…ocess.

PullRequest: graal/19757

docs/reference-manual/native-image/BuildOutput.md

@@ -30,7 +30,7 @@ GraalVM Native Image: Generating 'helloworld' (executable)...
  Garbage collector: Serial GC (max heap size: 80% of RAM)
 --------------------------------------------------------------------------------
  Build resources:
- - 13.24GB of memory (42.7% of 31.00GB system memory, determined at start)
+ - 13.24GB of memory (42.7% of system memory, using available memory)
  - 16 thread(s) (100.0% of 16 available processor(s), determined at start)
 [2/8] Performing analysis...  [****]                             (4.5s @ 0.54GB)
     3,163 reachable types   (72.5% of    4,364 total)
@@ -142,12 +142,13 @@ The `NATIVE_IMAGE_OPTIONS` environment variable is designed to be used by users,
 #### <a name="glossary-build-resources"></a>Build Resources
 The memory limit and number of threads used by the build process.
 
-More precisely, the memory limit of the Java heap, so actual memory consumption can be even higher.
+More precisely, the memory limit of the Java heap, so actual memory consumption can be higher.
 Please check the [peak RSS](#glossary-peak-rss) reported at the end of the build to understand how much memory was actually used.
-By default, the build process tries to only use free memory (to avoid memory pressure on the build machine), and never more than 32GB of memory.
-If less than 8GB of memory are free, the build process falls back to use 85% of total memory.
+By default, the build process uses the dedicated mode (up to 85% of system memory) in containers or CI environments (when the `$CI` environment variable is set to `true`), but never more than 32GB of memory.
+Otherwise, it tries to use available memory to avoid memory pressure on developer machines (shared mode).
+If less than 8GB of memory are available, the build process falls back to the dedicated mode.
 Therefore, consider freeing up memory if your machine is slow during a build, for example, by closing applications that you do not need.
-It is possible to overwrite the default behavior, for example with `-J-XX:MaxRAMPercentage=60.0` or `-J-Xmx16g`.
+It is possible to override the default behavior and set relative or absolute memory limits, for example with `-J-XX:MaxRAMPercentage=60.0` or `-J-Xmx16g`.
 
 By default, the build process uses all available processors to maximize speed, but not more than 32 threads.
 Use the `--parallelism` option to set the number of threads explicitly (for example, `--parallelism=4`).

substratevm/CHANGELOG.md

@@ -5,6 +5,7 @@ This changelog summarizes major changes to GraalVM Native Image.
 ## GraalVM for JDK 25
 * (GR-58668) Enabled [Whole-Program Sparse Conditional Constant Propagation (WP-SCCP)](https://github.com/oracle/graal/pull/9821) by default, improving the precision of points-to analysis in Native Image. This optimization enhances static analysis accuracy and scalability, potentially reducing the size of the final native binary.
 * (GR-59313) Deprecated class-level metadata extraction using `native-image-inspect` and removed option `DumpMethodsData`. Use class-level SBOMs instead by passing `--enable-sbom=class-level,export` to the `native-image` builder. The default value of option `IncludeMethodData` was changed to `false`.  
+* (GR-52400) The build process now uses 85% of system memory in containers and CI environments. Otherwise, it tries to only use available memory. If less than 8GB of memory are available, it falls back to 85% of system memory. The reason for the selected memory limit is now also shown in the build resources section of the build output.
 
 ## GraalVM for JDK 24 (Internal Version 24.2.0)
 * (GR-59717) Added `DuringSetupAccess.registerObjectReachabilityHandler` to allow registering a callback that is executed when an object of a specified type is marked as reachable during heap scanning.

substratevm/mx.substratevm/suite.py

@@ -1003,6 +1003,9 @@
                 "java.base" : [
                     "jdk.internal.jimage",
                 ],
+                "jdk.jfr": [
+                    "jdk.jfr.internal",
+                ],
             },
             "checkstyle": "com.oracle.svm.hosted",
             "workingSets": "SVM",

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/SubstrateOptions.java

@@ -738,6 +738,8 @@ public static boolean hasColorsEnabled(OptionValues values) {
     @Option(help = "Internal option to forward the value of " + NATIVE_IMAGE_OPTIONS_ENV_VAR)//
     public static final HostedOptionKey<String> BuildOutputNativeImageOptionsEnvVarValue = new HostedOptionKey<>(null);
 
+    public static final String BUILD_MEMORY_USAGE_REASON_TEXT_PROPERTY = "svm.build.memoryUsageReasonText";
+
     /*
      * Object and array allocation options.
      */

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/SubstrateUtil.java

@@ -38,7 +38,6 @@
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
-import jdk.graal.compiler.word.Word;
 import org.graalvm.nativeimage.Platform;
 import org.graalvm.nativeimage.Platforms;
 import org.graalvm.nativeimage.c.type.CCharPointer;
@@ -60,6 +59,7 @@
 import jdk.graal.compiler.java.LambdaUtils;
 import jdk.graal.compiler.nodes.BreakpointNode;
 import jdk.graal.compiler.util.Digest;
+import jdk.graal.compiler.word.Word;
 import jdk.vm.ci.meta.ResolvedJavaMethod;
 import jdk.vm.ci.meta.ResolvedJavaType;
 import jdk.vm.ci.meta.Signature;
@@ -82,17 +82,17 @@ public class SubstrateUtil {
 
     public static String getArchitectureName() {
         String arch = System.getProperty("os.arch");
-        switch (arch) {
-            case "x86_64":
-                arch = "amd64";
-                break;
-            case "arm64":
-                arch = "aarch64";
-                break;
-        }
-        return arch;
+        return switch (arch) {
+            case "x86_64" -> "amd64";
+            case "arm64" -> "aarch64";
+            default -> arch;
+        };
     }
 
+    /*
+     * [GR-55515]: Accessing isTerminal() reflectively only for 21 JDK compatibility. After dropping
+     * JDK 21, use it directly.
+     */
     private static final Method IS_TERMINAL_METHOD = ReflectionUtil.lookupMethod(true, Console.class, "isTerminal");
 
     private static boolean isTTY() {
@@ -111,8 +111,12 @@ private static boolean isTTY() {
         }
     }
 
-    public static boolean isRunningInCI() {
-        return !isTTY() || System.getenv("CI") != null;
+    public static boolean isNonInteractiveTerminal() {
+        return isCISetToTrue() || !isTTY();
+    }
+
+    public static boolean isCISetToTrue() {
+        return Boolean.parseBoolean(System.getenv("CI"));
     }
 
     /**

substratevm/src/com.oracle.svm.driver/src/com/oracle/svm/driver/MemoryUtil.java

@@ -24,12 +24,27 @@
  */
 package com.oracle.svm.driver;
 
+import java.io.BufferedReader;
+import java.io.InputStreamReader;
 import java.lang.management.ManagementFactory;
+import java.lang.reflect.Method;
+import java.nio.file.Files;
+import java.nio.file.Paths;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
 
+import com.oracle.svm.core.OS;
+import com.oracle.svm.core.SubstrateOptions;
+import com.oracle.svm.core.SubstrateUtil;
 import com.oracle.svm.core.util.ExitStatus;
+import com.oracle.svm.core.util.VMError;
 import com.oracle.svm.driver.NativeImage.NativeImageError;
+import com.oracle.svm.util.ReflectionUtil;
+
+import jdk.graal.compiler.serviceprovider.JavaVersionUtil;
 
 class MemoryUtil {
     private static final long KiB_TO_BYTES = 1024L;
@@ -39,16 +54,35 @@ class MemoryUtil {
     /* Builder needs at least 512MiB for building a helloworld in a reasonable amount of time. */
     private static final long MIN_HEAP_BYTES = 512L * MiB_TO_BYTES;
 
-    /* If free memory is below 8GiB, use 85% of total system memory (e.g., 7GiB * 85% ~ 6GiB). */
-    private static final long DEDICATED_MODE_THRESHOLD = 8L * GiB_TO_BYTES;
+    /* Use 85% of total system memory (e.g., 7GiB * 85% ~ 6GiB) in dedicated mode. */
     private static final double DEDICATED_MODE_TOTAL_MEMORY_RATIO = 0.85D;
 
+    /* If available memory is below 8GiB, fall back to dedicated mode. */
+    private static final int MIN_AVAILABLE_MEMORY_THRESHOLD_GB = 8;
+
     /*
      * Builder uses at most 32GB to avoid disabling compressed oops (UseCompressedOops).
      * Deliberately use GB (not GiB) to stay well below 32GiB when relative maximum is calculated.
      */
     private static final long MAX_HEAP_BYTES = 32_000_000_000L;
 
+    private static final Method IS_CONTAINERIZED_METHOD;
+    private static final Object IS_CONTAINERIZED_RECEIVER;
+
+    static {
+        IS_CONTAINERIZED_METHOD = ReflectionUtil.lookupMethod(jdk.jfr.internal.JVM.class, "isContainerized");
+        if (JavaVersionUtil.JAVA_SPEC == 21) { // non-static
+            var jvmField = ReflectionUtil.lookupField(jdk.jfr.internal.JVM.class, "jvm");
+            try {
+                IS_CONTAINERIZED_RECEIVER = jvmField.get(null);
+            } catch (IllegalAccessException e) {
+                throw VMError.shouldNotReachHere(e);
+            }
+        } else {
+            IS_CONTAINERIZED_RECEIVER = null; // static
+        }
+    }
+
     public static List<String> determineMemoryFlags(NativeImage.HostFlags hostFlags) {
         List<String> flags = new ArrayList<>();
         if (hostFlags.hasUseParallelGC()) {
@@ -61,9 +95,9 @@ public static List<String> determineMemoryFlags(NativeImage.HostFlags hostFlags)
          * -XX:InitialRAMPercentage or -Xms.
          */
         if (hostFlags.hasMaxRAMPercentage()) {
-            flags.add("-XX:MaxRAMPercentage=" + determineReasonableMaxRAMPercentage());
+            flags.addAll(determineMemoryUsageFlags(value -> "-XX:MaxRAMPercentage=" + value));
         } else if (hostFlags.hasMaximumHeapSizePercent()) {
-            flags.add("-XX:MaximumHeapSizePercent=" + (int) determineReasonableMaxRAMPercentage());
+            flags.addAll(determineMemoryUsageFlags(value -> "-XX:MaximumHeapSizePercent=" + value.intValue()));
         }
         if (hostFlags.hasGCTimeRatio()) {
             /*
@@ -82,23 +116,39 @@ public static List<String> determineMemoryFlags(NativeImage.HostFlags hostFlags)
     }
 
     /**
-     * Returns a percentage (0.0-100.0) to be used as a value for the -XX:MaxRAMPercentage flag of
-     * the builder process. Prefer free memory over total memory to reduce memory pressure on the
-     * host machine. Note that this method uses OperatingSystemMXBean, which is container-aware.
+     * Returns memory usage flags for the build process. Dedicated mode uses a fixed percentage of
+     * total memory and is the default in containers. Shared mode tries to use available memory to
+     * reduce memory pressure on the host machine. Note that this method uses OperatingSystemMXBean,
+     * which is container-aware.
      */
-    private static double determineReasonableMaxRAMPercentage() {
+    private static List<String> determineMemoryUsageFlags(Function<Double, String> toMemoryFlag) {
         var osBean = (com.sun.management.OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();
         final double totalMemorySize = osBean.getTotalMemorySize();
-        double reasonableMaxMemorySize = osBean.getFreeMemorySize();
+        final double dedicatedMemorySize = totalMemorySize * DEDICATED_MODE_TOTAL_MEMORY_RATIO;
 
-        if (reasonableMaxMemorySize < DEDICATED_MODE_THRESHOLD) {
-            /*
-             * When free memory is low, for example in memory-constrained environments or when a
-             * good amount of memory is used for caching, use a fixed percentage of total memory
-             * rather than free memory. In containerized environments, builds are expected to run
-             * more or less exclusively (builder + driver + optional Gradle/Maven process).
-             */
-            reasonableMaxMemorySize = totalMemorySize * DEDICATED_MODE_TOTAL_MEMORY_RATIO;
+        String memoryUsageReason = "unknown";
+        final boolean isDedicatedMemoryUsage;
+        if (SubstrateUtil.isCISetToTrue()) {
+            isDedicatedMemoryUsage = true;
+            memoryUsageReason = "$CI set to 'true'";
+        } else if (isContainerized()) {
+            isDedicatedMemoryUsage = true;
+            memoryUsageReason = "in container";
+        } else {
+            isDedicatedMemoryUsage = false;
+        }
+
+        double reasonableMaxMemorySize;
+        if (isDedicatedMemoryUsage) {
+            reasonableMaxMemorySize = dedicatedMemorySize;
+        } else {
+            reasonableMaxMemorySize = getAvailableMemorySize();
+            if (reasonableMaxMemorySize >= MIN_AVAILABLE_MEMORY_THRESHOLD_GB * GiB_TO_BYTES) {
+                memoryUsageReason = "using available memory";
+            } else { // fall back to dedicated mode
+                memoryUsageReason = "less than " + MIN_AVAILABLE_MEMORY_THRESHOLD_GB + "GB of memory available";
+                reasonableMaxMemorySize = dedicatedMemorySize;
+            }
         }
 
         if (reasonableMaxMemorySize < MIN_HEAP_BYTES) {
@@ -111,6 +161,142 @@ private static double determineReasonableMaxRAMPercentage() {
         /* Ensure max memory size does not exceed upper limit. */
         reasonableMaxMemorySize = Math.min(reasonableMaxMemorySize, MAX_HEAP_BYTES);
 
-        return reasonableMaxMemorySize / totalMemorySize * 100;
+        double reasonableMaxRamPercentage = reasonableMaxMemorySize / totalMemorySize * 100;
+        return List.of(toMemoryFlag.apply(reasonableMaxRamPercentage),
+                        "-D" + SubstrateOptions.BUILD_MEMORY_USAGE_REASON_TEXT_PROPERTY + "=" + memoryUsageReason);
+    }
+
+    private static boolean isContainerized() {
+        if (!OS.LINUX.isCurrent()) {
+            return false;
+        }
+        /*
+         * [GR-55515]: Accessing isContainerized() reflectively only for 21 JDK compatibility
+         * (non-static vs static method). After dropping JDK 21, use it directly.
+         */
+        try {
+            return (boolean) IS_CONTAINERIZED_METHOD.invoke(IS_CONTAINERIZED_RECEIVER);
+        } catch (ReflectiveOperationException | ClassCastException e) {
+            throw VMError.shouldNotReachHere(e);
+        }
+    }
+
+    private static double getAvailableMemorySize() {
+        return switch (OS.getCurrent()) {
+            case LINUX -> getAvailableMemorySizeLinux();
+            case DARWIN -> getAvailableMemorySizeDarwin();
+            case WINDOWS -> getAvailableMemorySizeWindows();
+        };
+    }
+
+    /**
+     * Returns the total amount of available memory in bytes on Linux based on
+     * <code>/proc/meminfo</code>, otherwise <code>-1</code>. Note that this metric is not
+     * container-aware (does not take cgroups into account) and may report available memory of the
+     * host.
+     *
+     * @see <a href=
+     *      "https://github.com/torvalds/linux/blob/865fdb08197e657c59e74a35fa32362b12397f58/mm/page_alloc.c#L5137">page_alloc.c#L5137</a>
+     */
+    private static long getAvailableMemorySizeLinux() {
+        try {
+            String memAvailableLine = Files.readAllLines(Paths.get("/proc/meminfo")).stream().filter(l -> l.startsWith("MemAvailable")).findFirst().orElse("");
+            Matcher m = Pattern.compile("^MemAvailable:\\s+(\\d+) kB").matcher(memAvailableLine);
+            if (m.matches()) {
+                return Long.parseLong(m.group(1)) * KiB_TO_BYTES;
+            }
+        } catch (Exception e) {
+        }
+        return -1;
+    }
+
+    /**
+     * Returns the total amount of available memory in bytes on Darwin based on
+     * <code>vm_stat</code>, otherwise <code>-1</code>.
+     *
+     * @see <a href=
+     *      "https://opensource.apple.com/source/system_cmds/system_cmds-496/vm_stat.tproj/vm_stat.c.auto.html">vm_stat.c</a>
+     */
+    private static long getAvailableMemorySizeDarwin() {
+        try {
+            Process p = Runtime.getRuntime().exec(new String[]{"vm_stat"});
+            try (BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()))) {
+                String line1 = reader.readLine();
+                if (line1 == null) {
+                    return -1;
+                }
+                Matcher m1 = Pattern.compile("^Mach Virtual Memory Statistics: \\(page size of (\\d+) bytes\\)").matcher(line1);
+                long pageSize = -1;
+                if (m1.matches()) {
+                    pageSize = Long.parseLong(m1.group(1));
+                }
+                if (pageSize <= 0) {
+                    return -1;
+                }
+                String line2 = reader.readLine();
+                Matcher m2 = Pattern.compile("^Pages free:\\s+(\\d+).").matcher(line2);
+                long freePages = -1;
+                if (m2.matches()) {
+                    freePages = Long.parseLong(m2.group(1));
+                }
+                if (freePages <= 0) {
+                    return -1;
+                }
+                String line3 = reader.readLine();
+                if (!line3.startsWith("Pages active")) {
+                    return -1;
+                }
+                String line4 = reader.readLine();
+                Matcher m4 = Pattern.compile("^Pages inactive:\\s+(\\d+).").matcher(line4);
+                long inactivePages = -1;
+                if (m4.matches()) {
+                    inactivePages = Long.parseLong(m4.group(1));
+                }
+                if (inactivePages <= 0) {
+                    return -1;
+                }
+                assert freePages > 0 && inactivePages > 0 && pageSize > 0;
+                return (freePages + inactivePages) * pageSize;
+            } finally {
+                p.waitFor();
+            }
+        } catch (Exception e) {
+        }
+        return -1;
+    }
+
+    /**
+     * Returns the total amount of available memory in bytes on Windows based on <code>wmic</code>,
+     * otherwise <code>-1</code>.
+     *
+     * @see <a href=
+     *      "https://learn.microsoft.com/en-us/windows/win32/cimwin32prov/win32-operatingsystem">Win32_OperatingSystem
+     *      class</a>
+     */
+    private static long getAvailableMemorySizeWindows() {
+        try {
+            Process p = Runtime.getRuntime().exec(new String[]{"cmd.exe", "/c", "wmic", "OS", "get", "FreePhysicalMemory"});
+            try (BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()))) {
+                String line1 = reader.readLine();
+                if (line1 == null || !line1.startsWith("FreePhysicalMemory")) {
+                    return -1;
+                }
+                String line2 = reader.readLine();
+                if (line2 == null) {
+                    return -1;
+                }
+                String line3 = reader.readLine();
+                if (line3 == null) {
+                    return -1;
+                }
+                Matcher m = Pattern.compile("^(\\d+)\\s+").matcher(line3);
+                if (m.matches()) {
+                    return Long.parseLong(m.group(1)) * KiB_TO_BYTES;
+                }
+            }
+            p.waitFor();
+        } catch (Exception e) {
+        }
+        return -1;
     }
 }