[Failing Test]: Java_GCP_IO_Direct on Github Actions is very flaky due to OOMs when running FhirIOSearchIT tests

[ahmedabu98]

What happened?

Looks like both testFhirIOSearchWithGenericParameters and testFhirIOSearch are failing due to OOM errors. Seeing these errors very frequently when running Java_GCP_IO_Direct on Github Actions but not so much for the Jenkins jobs.
Example runs: https://github.com/apache/beam/actions/runs/6281472530/job/17059883679?pr=28634, https://github.com/apache/beam/actions/runs/6278950376/job/17053724245?pr=28631
Not to say it doesn't happen on Jenkins (see this failing test), just not seeing it as much

Stacktrace dump for testFhirIOSearch:

com.google.gson.JsonParseException: Failed parsing JSON source: JsonReader at line 1 column 518561 path $[121].resource.item[1].adjudication[1].category.coding[0].code to Json
	at com.google.gson.JsonParser.parseReader(JsonParser.java:106)
	at com.google.gson.JsonParser.parseReader(JsonParser.java:70)
	at com.google.gson.JsonParser.parseString(JsonParser.java:51)
	at org.apache.beam.sdk.io.gcp.healthcare.JsonArrayCoder.decode(JsonArrayCoder.java:46)
	at org.apache.beam.sdk.io.gcp.healthcare.JsonArrayCoder.decode(JsonArrayCoder.java:28)
	at org.apache.beam.sdk.coders.Coder.decode(Coder.java:154)
	at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:84)
	at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:37)
	at org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:142)
	at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:102)
	at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:96)
	at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:168)
	at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodifiedThrowingCheckedExceptions(MutationDetectors.java:139)
	at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodified(MutationDetectors.java:128)
	at org.apache.beam.runners.direct.ImmutabilityEnforcementFactory$ImmutabilityCheckingEnforcement.verifyUnmodified(ImmutabilityEnforcementFactory.java:146)
	at org.apache.beam.runners.direct.ImmutabilityEnforcementFactory$ImmutabilityCheckingEnforcement.afterElement(ImmutabilityEnforcementFactory.java:131)
	at org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:175)
	at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:129)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.Arrays.copyOfRange(Arrays.java:3664)
	at java.lang.String.<init>(String.java:207)
	at com.google.gson.stream.JsonReader.nextQuotedValue(JsonReader.java:1014)
	at com.google.gson.stream.JsonReader.nextName(JsonReader.java:798)
	at com.google.gson.internal.bind.TypeAdapters$28.read(TypeAdapters.java:783)
	at com.google.gson.internal.bind.TypeAdapters$28.read(TypeAdapters.java:725)
	at com.google.gson.internal.Streams.parse(Streams.java:49)
	at com.google.gson.JsonParser.parseReader(JsonParser.java:102)

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[svetakvsundhar]

.take-issue

[Abacn]

dup #28505

We are working on moving this test suite to large-memory worker set which would likely mitigate the issue, so no need to do with heap space for now; for test itself, may either decrease the data size since this is just an integration test; or optimize the IO

[svetakvsundhar]

fixed via #28719

[svetakvsundhar]

.close-issue