java.lang.OutOfMemoryError: Java heap space #2265

timrijckaert · 2022-07-11T14:02:23Z

Describe the bug
Hello,

As we have been adding screenshot tests to more and more Android modules we wanted to parallelise our CI build to first build the test APK's.
After all modules have created their APK we run ./gradlew runFlank

We have about 14 modules thus-far containing screenshot tests.
Averaging in size of approximately 60MB

The build stops with a OutOfMemory.
See full logs in Build summary section below.

Build summary

Task :configureFulladle
Task :validateFladleConfig
Task :writeConfigProps
Task :execFlank
version: v21.11.0
revision: 1bf9b6a
session id: 08044f2b-9127-402c-9a6a-f5b6e7b51c13
AndroidArgs
gcloud:
results-bucket: test-lab-nxzauzcrb50h6-m6yq6i7d7dwni
results-dir: 2022-07-11_06-29-04.516092_hCGC
record-video: false
timeout: 15m
async: false
client-details:
network-profile: null
results-history-name: null
# Android gcloud
app: /xxx/xxxx/xxxxx/testlab/testlab-debug.apk
test: /xxx/xxxx/xxxxx/features/xxxx/build/outputs/apk/androidTest/debug/xxxx-debug-androidTest.apk
additional-apks:
auto-google-login: false
use-orchestrator: false
directories-to-pull:
- /storage/emulated/0/Download/screenshots/
grant-permissions: all
type: null
other-files:
scenario-numbers:
scenario-labels:
obb-files:
obb-names:
performance-metrics: false
num-uniform-shards: null
test-runner-class: null
test-targets:
- annotation com.xxxx.xxxxx.xxxxx.ShotTest
robo-directives:
robo-script: null
device:
- model: Pixel2
version: 29
locale: en
orientation: portrait
num-flaky-test-attempts: 0
test-targets-for-shard:
fail-fast: false
parameterized-tests: default
flank:
max-test-shards: 1
shard-time: -1
num-test-runs: 1
smart-flank-gcs-path:
smart-flank-disable-upload: false
default-test-time: 120.0
use-average-test-time-for-new-tests: false
files-to-download:
- .*.png$
test-targets-always-run:
disable-sharding: false
project: neba-android
local-result-dir: /xxx/xxxx/xxxxx/flank
full-junit-result: false
# Android Flank Yml
keep-file-path: false
additional-app-test-apks:
- test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
- test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
- test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
- test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
- test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
- test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
- test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
- test: /xxx/xxxx/xxxxx/libraries/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
- test: /xxx/xxxx/xxxxx/libraries/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
- test: /xxx/xxxx/xxxxx/libraries/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
- test: /xxx/xxxx/xxxxx/libraries/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
- test: /xxx/xxxx/xxxxx/libraries/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk
max-test-shards: 5
run-timeout: 30m
legacy-junit-result: false
ignore-failed-tests: false
output-style: verbose
disable-results-upload: false
default-class-test-time: 240.0
disable-usage-statistics: false
output-report: none
skip-config-validation: false
custom-sharding-json:
RunTests
Smart Flank cache hit: 0% (0 / 23)
Shard times: 2760s
Smart Flank cache hit: 0% (0 / 1)
Shard times: 120s
Smart Flank cache hit: 0% (0 / 37)
Shard times: 4440s
Smart Flank cache hit: 0% (0 / 8)
Shard times: 960s
Smart Flank cache hit: 0% (0 / 3)
Shard times: 360s
Smart Flank cache hit: 0% (0 / 9)
Shard times: 1080s
Smart Flank cache hit: 0% (0 / 2)
Shard times: 240s
Smart Flank cache hit: 0% (0 / 4)
Shard times: 480s
Smart Flank cache hit: 0% (0 / 3)
Shard times: 360s
Smart Flank cache hit: 0% (0 / 1)
Shard times: 120s
Smart Flank cache hit: 0% (0 / 13)
Shard times: 1560s
Smart Flank cache hit: 0% (0 / 5)
Shard times: 600s
Smart Flank cache hit: 0% (0 / 211)
Shard times: 5040s, 5040s, 5040s, 5040s, 5160s
Saved 13 shards to /xxx/xxxx/xxxxx/flank/android_shards.json
Uploading [android_shards.json] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/...
Uploading [testlab-debug.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/...
Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/.... Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/.. Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/.. Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/...... Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/... Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/...........................
Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/...
Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/.....
java.lang.OutOfMemoryError: Java heap space
at java.base/java.nio.file.Files.read(Files.java:3153)
at java.base/java.nio.file.Files.readAllBytes(Files.java:3213)
at ftl.util.FileReferenceKt.upload(FileReference.kt:33)
at ftl.util.FileReferenceKt.uploadIfNeeded(FileReference.kt:28)
at ftl.run.platform.android.UploadApksKt.upload(UploadApks.kt:35)
at ftl.run.platform.android.UploadApksKt.upload(UploadApks.kt:27)
at ftl.run.platform.android.UploadApksKt.access$upload(UploadApks.kt:1)
at ftl.run.platform.android.UploadApksKt$upload$2$1$1.invokeSuspend(UploadApks.kt:23)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Total run duration: 2m 40s
- Preparation: 0m 6s
Task :execFlank FAILED
FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':execFlank'.

Process 'command '/usr/lib/jvm/java-11-openjdk-amd64/bin/java'' finished with non-zero exit value 15

Try:

Run with --stacktrace option to get the stack trace.
Run with --info or --debug option to get more log output.
Run with --scan to get full insights.

Get more help at https://help.gradle.org
BUILD FAILED in 6m 27s
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings
4 actionable tasks: 4 executed

java.lang.OutOfMemoryError: Java heap space
	at java.base/java.nio.file.Files.read(Files.java:3153)
	at java.base/java.nio.file.Files.readAllBytes(Files.java:3213)
	at ftl.util.FileReferenceKt.upload(FileReference.kt:33)
	at ftl.util.FileReferenceKt.uploadIfNeeded(FileReference.kt:28)
	at ftl.run.platform.android.UploadApksKt.upload(UploadApks.kt:35)
	at ftl.run.platform.android.UploadApksKt.upload(UploadApks.kt:27)
	at ftl.run.platform.android.UploadApksKt.access$upload(UploadApks.kt:1)
	at ftl.run.platform.android.UploadApksKt$upload$2$1$1.invokeSuspend(UploadApks.kt:23)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Any idea on how to fix this?

Additional context

Running on Gitlab shared runners
Flank version: v21.11.0
I saw possible related ticket

Thanks in advance.

The text was updated successfully, but these errors were encountered:

bootstraponline · 2022-07-12T19:13:59Z

@timrijckaert Hey, for this use case I'd recommend uploading the apks to Google Cloud Storage via gsutil, and then passing Flank the GCS path to the apks. That'll ensure there's no memory problems as the apks get larger over time. The tests will also be a bit faster as the apk only has to be uploaded once.

Alternatively, you could try increasing the Java heap space made available to flank. Something like: java -jar -Xms2048M -Xmx2048M flank.jar

timrijckaert · 2022-07-12T19:23:54Z

Hi @bootstraponline

Thanks for the quick reply!
I'm using the Fladle plugin.
Any idea or more detailed information on how to configure it to use a combination of gsutil and the plugin?

Or pass the memory options to the runFlank task?

Thanks in advance

bootstraponline · 2022-07-13T09:45:39Z

@timrijckaert I don't think Fladle supports the gsutil strategy I mentioned.

Try to increase the heap space by setting gradle.properties
https://stackoverflow.com/a/47454640

timrijckaert · 2022-07-13T10:08:38Z

Hmm same result unfortunately

bootstraponline · 2022-07-14T14:14:40Z

@timrijckaert I'd open an issue on https://github.com/runningcode/fladle/issues to see if they can advise.

bootstraponline · 2022-07-14T17:08:59Z

@timrijckaert The fladle maintainer suggested to set the maxHeapSize on the FlankExec task:

https://docs.gradle.org/current/dsl/org.gradle.api.tasks.JavaExec.html#org.gradle.api.tasks.JavaExec:maxHeapSize

timrijckaert · 2022-07-20T09:51:47Z

I applied the following

task flankMaxHeapSize(type: com.osacky.flank.gradle.FlankJavaExec) {
    maxHeapSize = "4g"
}

I get

Task :flankMaxHeapSize FAILED
Error: Could not find or load main class ftl.Main
Caused by: java.lang.ClassNotFoundException: ftl.Main

loki666 · 2022-07-28T11:12:18Z

issue seems to be related to the Files.readAllBytes() call when a file need to be uploaded

flank/test_runner/src/main/kotlin/ftl/util/FileReference.kt

Line 33 in 3748bce

RemoteStorage.Data(file, Files.readAllBytes(Paths.get(file)))

maybe a new type should be introduced (ie: RemoteStorage.File instead of RemoteStorage.Data which takes the ByteArray) to delay the loading and use a buffered read/write with GcStorage (ie: using a WriterChannel)

timrijckaert · 2022-07-28T13:28:06Z

To add on this

flank/test_runner/src/main/kotlin/ftl/run/platform/android/UploadApks.kt

Lines 56 to 62 in 33d6f7a

    
           private suspend fun List<String>.uploadToGcloudIfNeeded( 
        
               runGcsPath: String, 
        
               resultsBucket: String 
        
           ) = coroutineScope { 
        
               map { 
        
                   async { it.asFileReference().uploadIfNeeded(resultsBucket, runGcsPath).remote } 
        
               }.awaitAll()

This gets amplified by the amount of APK's to upload.

bootstraponline · 2022-07-29T13:42:29Z

@loki666 @timrijckaert If you'd like to submit a patch, I'll happily review it.

timrijckaert added the Bug label Jul 11, 2022

timrijckaert mentioned this issue Jul 20, 2022

java.lang.OutOfMemoryError: Java heap space runningcode/fladle#303

Closed

joshafeinberg mentioned this issue Aug 12, 2022

fix: stop loading entire file into memory to fix OOM errors #2274

Merged

3 tasks

adamvduke closed this as completed in #2274 Dec 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

java.lang.OutOfMemoryError: Java heap space #2265

java.lang.OutOfMemoryError: Java heap space #2265

timrijckaert commented Jul 11, 2022 •

edited

Loading

bootstraponline commented Jul 12, 2022

timrijckaert commented Jul 12, 2022 •

edited

Loading

bootstraponline commented Jul 13, 2022

timrijckaert commented Jul 13, 2022

bootstraponline commented Jul 14, 2022

bootstraponline commented Jul 14, 2022 •

edited

Loading

timrijckaert commented Jul 20, 2022 •

edited

Loading

loki666 commented Jul 28, 2022

timrijckaert commented Jul 28, 2022

bootstraponline commented Jul 29, 2022

java.lang.OutOfMemoryError: Java heap space #2265

java.lang.OutOfMemoryError: Java heap space #2265

Comments

timrijckaert commented Jul 11, 2022 • edited Loading

bootstraponline commented Jul 12, 2022

timrijckaert commented Jul 12, 2022 • edited Loading

bootstraponline commented Jul 13, 2022

timrijckaert commented Jul 13, 2022

bootstraponline commented Jul 14, 2022

bootstraponline commented Jul 14, 2022 • edited Loading

timrijckaert commented Jul 20, 2022 • edited Loading

loki666 commented Jul 28, 2022

timrijckaert commented Jul 28, 2022

bootstraponline commented Jul 29, 2022

timrijckaert commented Jul 11, 2022 •

edited

Loading

timrijckaert commented Jul 12, 2022 •

edited

Loading

bootstraponline commented Jul 14, 2022 •

edited

Loading

timrijckaert commented Jul 20, 2022 •

edited

Loading