Seg Fault on Java 21 JRE

[thai-op]

Describe the bug

This is an error message from a job that I run in a container in AWS batch. I don't have the core dump and just the message:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f3394e82bdb, pid=1, tid=92
#
 Environment Temurin-21.0.1+12 (21.0.1+12) (build 21.0.1+12-LTS)
 Environment Temurin-21.0.1+12 (21.0.1+12) (build 21.0.1+12-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.1+12 (21.0.1+12-LTS, mixed mode, sharing, tiered, compressed class ptrs, z gc, linux-amd64)
# Problematic frame:
# C [AWSCRT_9028473956044655129libaws-crt-jni.so+0x8bbdb] Java_software_amazon_awssdk_crt_http_HttpStreamBase_httpStreamBaseIncrementWindow+0x1b
#
# Core dump will be written. Default location: //core.1
#
# An error report file with more information is saved as:
# //hs_err_pid1.log
[144.782s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
# https://github.com/adoptium/adoptium-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f38bb2ea898]

The core interaction with AWS involves DynamoDB as a cache, SSM parameter stores, secret values, etc. nothing special there. We heavily use virtual threads for non-blocking network IO with high parallelism. This was working fine using the default Apache Http client but we wanted the newer non-blocking http client so this is disappointing.

JRE Version: Temurin-21.0.1+12 running on Redhat UBI9 image.

Expected Behavior

It should not seg fault.

Current Behavior

It killed the vm.

Reproduction Steps

No way to reproduce thus far.

Possible Solution

No response

Additional Information/Context

No response

aws-crt-java version used

2.22.13

Java version used

21.0.1+12

Operating System and version

Redhat UBI9

[TingDaoK]

aws/aws-sdk-java-v2#4833 this PR from aws-sdk-java-v2 probably fixes the issue

[thai-op]

Thanks, if there’s a newer sdk version I’ll try it today to see if things won’t crash again

[ravi-signal]

Hi, I also hit this (on 2.23.1 initially). I tried upgrading to 2.23.5, which AIUI includes aws/aws-sdk-java-v2#4833 , and I continue to see it:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f07fc28c4eb, pid=1, tid=131
#
# JRE version: OpenJDK Runtime Environment Temurin-21.0.1+12 (21.0.1+12) (build 21.0.1+12-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.1+12 (21.0.1+12-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [AWSCRT_15418525997656330583libaws-crt-jni.so+0x8c4eb]  Java_software_amazon_awssdk_crt_http_HttpStreamBase_httpStreamBaseIncrementWindow+0x1b
#

[jmklix]

The most recent merged PR should fix this Seg Fault. Please use the most recent version of the java-v2 sdk and let us know if you run into any problems

[TingDaoK]

To add more detail. aws/aws-sdk-java-v2#4839 PR from java SDK is supposed to fix the issue. They plan to release another version today.

[zoewangg]

Hey all, a fix has been released as part of 2.23.8. Could you try it and let us know if you continue to see the issue?

[ravi-signal]

I have not been able to reproduce the issue on 2.23.8. FWIW, I was able to reproduce the issue fairly quickly with the previous builds.

Thank you for the quick fix/release cycle!

[porwalu]

Hello Team,
We are seeing this on version 2.22.0 and our main use case it for accessing S3
However, we are seeing this issue on some of our environments and not all. It occurs after 18 hours - 36 hours on different environments.
While it is good to see this resolved, is there a way to reproduce it at will so that we can test it sooner and confidently rather than waiting for a long interval based on historic crashes?