JDK 19 MiniMix_5m_1 hang in Thread.getStateImpl #16036

pshipton · 2022-10-04T13:29:24Z

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_1/19
MiniMix_5m_1

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_1/19/system_test_output.tar.gz

LT  04:08:01.390 - Completed 13.4%. Number of tests started=10586 (+3937)
STF 04:12:19.367 - Heartbeat: Process LT  is still running
STF 04:17:19.211 - Heartbeat: Process LT  is still running
STF 04:22:19.069 - Heartbeat: Process LT  is still running
STF 04:27:19.047 - Heartbeat: Process LT  is still running
STF 04:32:19.283 - Heartbeat: Process LT  is still running
STF 04:37:19.023 - Heartbeat: Process LT  is still running
STF 04:42:19.203 - Heartbeat: Process LT  is still running
STF 04:47:19.411 - Heartbeat: Process LT  is still running
STF 04:52:19.009 - Heartbeat: Process LT  is still running
STF 04:57:19.398 - Heartbeat: Process LT  is still running
STF 05:02:19.487 - Heartbeat: Process LT  is still running
STF 05:07:19.427 - Heartbeat: Process LT  is still running
STF 05:12:19.456 - Heartbeat: Process LT  is still running
STF 05:12:20.457 - **FAILED** Process LT  has timed out

3XMTHREADINFO      "load-9" J9VMThread:0x000000000060BE00, omrthread_t:0x000003FF8C5B5040, java/lang/Thread:0x000000003A012EF0, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x21, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x0000000039A7D7F8)
3XMTHREADINFO1            (native thread ID:0xEE4E, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00001001)
3XMTHREADINFO2            (native stack address range from:0x000003FF6C1BC000, to:0x000003FF6C1FC000, size:0x40000)
3XMCPUTIME               CPU usage total: 3.111664071 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=2331328 (0x2392C0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Thread.getStateImpl(Native Method)
4XESTACKTRACE                at java/lang/Thread.threadState(Thread.java:2864(Compiled Code))
5XESTACKTRACE                   (entered lock: java/lang/Object@0x000000003A07F8C8, entry count: 1)
4XESTACKTRACE                at java/lang/Thread.isTerminated(Thread.java:2872(Compiled Code))
4XESTACKTRACE                at java/lang/Thread.getThreadGroup(Thread.java:2025(Compiled Code))
4XESTACKTRACE                at java/lang/ThreadGroup.enumerate(ThreadGroup.java:450)
4XESTACKTRACE                at gnu/testlet/java/lang/ThreadGroup/enumerate.test(enumerate.java:48)
4XESTACKTRACE                at java/lang/invoke/LambdaForm$DMH/0x00000000d80054f0.invokeVirtual(LambdaForm$DMH(Compiled Code))
4XESTACKTRACE                at java/lang/invoke/LambdaForm$MH/0x00000000d8005be0.invoke(LambdaForm$MH(Compiled Code))
4XESTACKTRACE                at java/lang/invoke/LambdaForm$MH/0x000000008c40ee70.invokeExact_MT(LambdaForm$MH(Compiled Code))
4XESTACKTRACE                at jdk/internal/reflect/DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:155(Compiled Code))
4XESTACKTRACE                at jdk/internal/reflect/DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104(Compiled Code))
4XESTACKTRACE                at java/lang/reflect/Method.invoke(Method.java:578(Compiled Code))
4XESTACKTRACE                at net/adoptopenjdk/loadTest/adaptors/MauveAdaptor.executeTest(MauveAdaptor.java:74(Compiled Code))
4XESTACKTRACE                at net/adoptopenjdk/loadTest/LoadTestRunner$2.run(LoadTestRunner.java:182)
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
4XESTACKTRACE                at java/lang/Thread.run(Thread.java:1573)

The text was updated successfully, but these errors were encountered:

pshipton · 2022-10-04T13:29:43Z

@babsingh @tajila fyi

tajila · 2022-10-04T13:39:06Z

@ChengJin01 Please take a look at this

ChengJin01 · 2022-10-05T03:13:39Z

I went through the dumps at https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_1/19/system_test_output.tar.gz and didn't find any suspicious thread that might trigger the hang in the failing job. In addition, there was no deadlock detected via jdmpview.

As for the J9VMThread load-9 mentioned in the description, it was waiting for an unowned mutex lock, which doesn't seem to be hung in there.

2LKREGMON          Thread public flags mutex lock (0x000003FF8C421268): <unowned> <------
3LKNOTIFYQ            Waiting to be notified:
3LKWAITNOTIFY            "load-9" (J9VMThread:0x000000000060BE00)

I manually ran the failing test suite MiniMix_5m_1 with the failing build over 10 times on one of Jenkins zLinux machines and it always passed without any issue.
There is no hang detected so far in the Grinder (X50/still ongoing) at https://openj9-jenkins.osuosl.org/job/Grinder/1321/consoleText. Will wait and see from there how frequently the hang occurs.

pshipton · 2022-10-05T03:22:56Z

There are a couple of javacore files, taken a minute apart, and both showed the same stack for the "load-9" thread.

ChengJin01 · 2022-10-05T04:03:35Z

Assuming there was something happening for "load-9" thread according to the javacores,

3XMTHREADINFO      "load-9" J9VMThread:0x000000000060BE00, omrthread_t:0x000003FF8C5B5040, java/lang/Thread:0x000000003A012EF0, state:R, prio=5
...
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Thread.getStateImpl(Native Method) <------
4XESTACKTRACE                at java/lang/Thread.threadState(Thread.java:2864(Compiled Code))
5XESTACKTRACE                   (entered lock: java/lang/Object@0x000000003A07F8C8, entry count: 1)
4XESTACKTRACE                at java/lang/Thread.isTerminated(Thread.java:2872(Compiled Code))
4XESTACKTRACE                at java/lang/Thread.getThreadGroup(Thread.java:2025(Compiled Code))
4XESTACKTRACE                at java/lang/ThreadGroup.enumerate(ThreadGroup.java:450)
4XESTACKTRACE                at gnu/testlet/java/lang/ThreadGroup/enumerate.test(enumerate.java:48)
...
3XMTHREADINFO3           Native callstack:
...
4XENATIVESTACK                (0x000003FF6C1F9B10 [<unknown>+0x0])
4XENATIVESTACK               pthread_cond_wait+0x138 (0x000003FF9108C3B0 [libpthread.so.0+0xc3b0])
4XENATIVESTACK                (0x000003FF90607B26 [libj9thr29.so+0x7b26])
4XENATIVESTACK               omrthread_monitor_wait+0x60 (0x000003FF9060B690 [libj9thr29.so+0xb690])
4XENATIVESTACK                (0x000003FF90818D9A [libj9vm29.so+0x98d9a]) <--- internalAcquireVMAccessNoMutexWithMask
4XENATIVESTACK                (0x000003FF9081C8EA [libj9vm29.so+0x9c8ea]) <--- haltThreadForInspection (via objdump)
4XENATIVESTACK               Java_java_lang_Thread_getStateImpl+0x90 (0x000003FF8AC55E08 [libjclse29.so+0x55e08])
4XENATIVESTACK                (0x000003FF6FB5839A [<unknown>+0x0])

it should be stuck at the native method getStateImpl() in Thread.threadState at https://github.com/ibmruntimes/openj9-openjdk-jdk19/blob/b9e3daf582ff40b3a187281241c3cb691e4fec73/src/java.base/share/classes/java/lang/Thread.java#L2864

    boolean isTerminated() {
        return threadState() == State.TERMINATED;
    }

    State threadState() {
        synchronized (interruptLock) {
            if (eetop == NO_REF) {
                if (isDead()) {
                    return State.TERMINATED;
                }
                return State.NEW;
            }
            return State.values()[getStateImpl(eetop)]; <-------
        }
    }

where the native code of getStateImpl was at

openj9/runtime/jcl/common/thread.cpp

Line 87 in 96fe9b8

    
           currentThread->javaVM->internalVMFunctions->haltThreadForInspection(currentThread, vmThread);

Java_java_lang_Thread_getStateImpl(JNIEnv *env, jobject recv, jlong threadRef)
{
...
	Trc_JCL_Thread_getStateImpl_Entry(currentThread, vmThread);
	currentThread->javaVM->internalVMFunctions->internalEnterVMFromJNI(currentThread);
	currentThread->javaVM->internalVMFunctions->haltThreadForInspection(currentThread, vmThread); <------

and

openj9/runtime/vm/VMAccess.cpp

Line 1296 in e674448

    
           internalAcquireVMAccessNoMutexWithMask(currentThread, J9_PUBLIC_FLAGS_HALT_THREAD_ANY - J9_PUBLIC_FLAGS_HALT_THREAD_INSPECTION);

haltThreadForInspection(J9VMThread * currentThread, J9VMThread * vmThread)
{

_tryAgain:

	Assert_VM_mustHaveVMAccess(currentThread);

	/* Inspecting the current thread does not require any halting */
	if (currentThread != vmThread) {
		VM_VMAccess::setPublicFlags(currentThread, J9_PUBLIC_FLAGS_NOT_AT_SAFE_POINT);

		omrthread_monitor_enter(vmThread->publicFlagsMutex);

		/* increment the inspection count but don't try to short circuit -- the thread might not actually be halted yet */
		vmThread->inspectionSuspendCount += 1;

		/* Now halt the thread for inspection */
		setHaltFlag(vmThread, J9_PUBLIC_FLAGS_HALT_THREAD_INSPECTION);

		/* If the thread doesn't have VM access and it not queued for exclusive we can proceed immediately */
		if (vmThread->publicFlags & (J9_PUBLIC_FLAGS_VM_ACCESS | J9_PUBLIC_FLAGS_QUEUED_FOR_EXCLUSIVE)) {
			/* Release VM access while waiting */
			/* (We must release the other thread's publicFlagsMutex to avoid deadlock here) */
			omrthread_monitor_exit(vmThread->publicFlagsMutex);
			internalReleaseVMAccess(currentThread);
			omrthread_monitor_enter(vmThread->publicFlagsMutex);
#if defined(J9VM_INTERP_ATOMIC_FREE_JNI)
#if defined(J9VM_INTERP_ATOMIC_FREE_JNI_USES_FLUSH)
			flushProcessWriteBuffers(currentThread->javaVM);
#endif /* J9VM_INTERP_ATOMIC_FREE_JNI_USES_FLUSH */
			VM_AtomicSupport::readWriteBarrier(); // necessary?
#endif /* J9VM_INTERP_ATOMIC_FREE_JNI */
			if (VM_VMAccess::mustWaitForVMAccessRelease(vmThread)) {
				while (J9_ARE_ANY_BITS_SET(vmThread->publicFlags, J9_PUBLIC_FLAGS_VM_ACCESS | J9_PUBLIC_FLAGS_QUEUED_FOR_EXCLUSIVE)) {
					omrthread_monitor_wait(vmThread->publicFlagsMutex);
				}
			}
			omrthread_monitor_exit(vmThread->publicFlagsMutex);

			/* Thread is halted - reacquire VM access */
	
			omrthread_monitor_enter(currentThread->publicFlagsMutex);
----------->     internalAcquireVMAccessNoMutexWithMask(currentThread, J9_PUBLIC_FLAGS_HALT_THREAD_ANY - J9_PUBLIC_FLAGS_HALT_THREAD_INSPECTION);
			omrthread_monitor_exit(currentThread->publicFlagsMutex);

ChengJin01 · 2022-10-05T04:08:14Z

It seems haltThreadForInspection was only modified via a PR at #15227 recently to resolve a couple of issues related to hang/halted thread. So I am wondering whether there was any connection between this issue and #15227.

ChengJin01 · 2022-10-05T04:08:58Z

FYI: @amicic, @gacholio

ChengJin01 · 2022-10-06T16:24:14Z

Launched another Grinder (x200) at https://openj9-jenkins.osuosl.org/job/Grinder/1327/ with the latest passing build at https://openj9-jenkins.osuosl.org/job/Pipeline-Build-Test-JDK19/167/ to see how it goes.

tajila · 2022-10-07T13:27:30Z

@ChengJin01 Do you know if this fails on JDK17?

ChengJin01 · 2022-10-07T14:52:01Z

@ChengJin01 Do you know if this fails on JDK17?

This job never fails on JDK17 in recent histories at https://openj9-jenkins.osuosl.org/view/Test_System_Nightly/job/Test_openjdk17_j9_extended.system_s390x_linux_Nightly/, but we can launch a Grinder with the latest build to double-check.

ChengJin01 · 2022-10-07T14:58:18Z

Launched a Grinder (x200) at https://openj9-jenkins.osuosl.org/job/Grinder/1329/ with the latest build at https://openj9-jenkins.osuosl.org/view/Test_System_Nightly/job/Test_openjdk17_j9_extended.system_s390x_linux_Nightly/328/.

ChengJin01 · 2022-10-07T15:12:37Z

The previously completed Grinder at https://openj9-jenkins.osuosl.org/job/Grinder/1327/ ended without any issue (killed by after 98 runs due to timeout) at https://openj9-jenkins.osuosl.org/job/Grinder_testList_0/161/consoleText

[2022-10-07T02:18:56.782Z] LT  02:18:54.278 - Completed 46.9%. Number of tests started=34837 (+5415)
[2022-10-07T02:19:15.668Z] Cancelling nested steps due to timeout
[2022-10-07T02:19:15.835Z] Sending interrupt signal to process
[2022-10-07T02:19:17.071Z] LT  02:19:14.265 - Completed 53.6%. Number of tests started=39086 (+4249)

So it seems the hang was only triggered by the changes between https://openj9-jenkins.osuosl.org/job/Pipeline-Build-Test-JDK19/168/

OpenJ9: [c19e4267798](https://github.com/eclipse-openj9/openj9/commit/c19e4267798b13e6ee21c71290a2877c07fc4abb)
OMR: [b8f2f89464c](https://github.com/eclipse-openj9/openj9-omr/commit/b8f2f89464ca71f38dd7d63f49bd3fc0f945224d)
OpenJDK19: [b9e3daf582f](https://github.com/ibmruntimes/openj9-openjdk-jdk19/commit/b9e3daf582ff40b3a187281241c3cb691e4fec73) - openj9-openjdk-jdk19

and https://openj9-jenkins.osuosl.org/job/Pipeline-Build-Test-JDK19/167/

OpenJ9: [710fd952924](https://github.com/eclipse-openj9/openj9/commit/710fd952924c673ee702fcbc311c99d70095ecde)
OMR: [f99c9dd2248](https://github.com/eclipse-openj9/openj9-omr/commit/f99c9dd2248e6ee35e40536b3c243eee33924407)
OpenJDK19: [05db3773ec3](https://github.com/ibmruntimes/openj9-openjdk-jdk19/commit/05db3773ec323ac48a721cc22e59b0a768e5d4e7) - openj9-openjdk-jdk19

gacholio · 2022-10-07T16:34:18Z

If I understand the above comments, this rules out #15227 as the cause (committed in June).

It would be helpful to see the !threads stack and !threads flags output.

ChengJin01 · 2022-10-11T15:18:12Z

There was no hang detected in the Grinder at https://openj9-jenkins.osuosl.org/job/Grinder/1329/ for JDK17. So the problem is most likely related to Project Loom in JDK19 for the merged changes in OpenJ9 between c19e426 and 710fd95 given there is nothing suspicious in OMR changes.

FYI: @fengxue-IS

gacholio · 2022-10-11T15:26:34Z

710fd95...c19e426

gacholio · 2022-10-11T15:35:10Z

Nothing immediately suspicious - does the test fail with -Xint or on platforms other than 390?

ChengJin01 · 2022-10-11T15:51:44Z

Just launched a couple of Grinders (x100) with the latest build on other platforms plus the failing build with -Xint on Linux/s390x see how it goes.
https://openj9-jenkins.osuosl.org/job/Grinder/1349/ (Linux/x86_64)
https://openj9-jenkins.osuosl.org/job/Grinder/1350/ (macOS/Aarch64)
https://openj9-jenkins.osuosl.org/job/Grinder/1351/ (AIX/PPC64)
https://openj9-jenkins.osuosl.org/job/Grinder/1352/ (with -Xint on Linux/s390x)

pshipton · 2022-10-11T18:32:45Z

So far we've only ever seen this hang once, in jdk19 nightly builds.

ChengJin01 · 2022-10-12T17:25:22Z

All grinders at #16036 (comment) passed without any issue:
https://openj9-jenkins.osuosl.org/job/Grinder/1352/tapTestReport/ (with -Xint on Linux/s390x)
https://openj9-jenkins.osuosl.org/job/Grinder/1351/tapTestReport/ (AIX/PPC64)
https://openj9-jenkins.osuosl.org/job/Grinder/1350/tapTestReport/ (macOS/Aarch64)
https://openj9-jenkins.osuosl.org/job/Grinder/1349/tapTestReport/ (Linux/x86_64)

So it seems the hang only occurred on Linux/s390x with JIT enabled and was captured only once in Grinder(x50) at https://openj9-jenkins.osuosl.org/job/Grinder/1321/consoleText.

gacholio · 2022-10-14T16:46:04Z

If that's the case, getting the information I requested earlier (thread flags, java stacks and native stacks) will help us figure out what's going on.

ChengJin01 · 2022-10-14T22:31:21Z

The flags of threads in the core don't seem to be problematic as follows (in which the majority of threads parked in there)

> !threads flags
Attached Threads List. For more options, run !threads help

    !j9vmthread 0x69700 publicFlags=1 privateFlags=1008 inNative=0 // main <---J9_PUBLIC_FLAGS_HALT_THREAD_EXCLUSIVE 
    !j9vmthread 0x6bd00 publicFlags=1 privateFlags=101a inNative=0 // JIT Compilation Thread-000
    !j9vmthread 0x6e300 publicFlags=1 privateFlags=101a inNative=0 // JIT Compilation Thread-001
    !j9vmthread 0x204900 publicFlags=49000 privateFlags=0 inNative=0 // load-0 <-- J9_PUBLIC_FLAGS_EXCLUSIVE_SET_NOT_SAFE  |  J9_PUBLIC_FLAGS_HALT_THREAD_INSPECTION  |  J9_PUBLIC_FLAGS_NOT_AT_SAFE_POINT 
    !j9vmthread 0x5f9000 publicFlags=1 privateFlags=0 inNative=0 // load-1
    !j9vmthread 0x5fb600 publicFlags=1 privateFlags=0 inNative=0 // load-2
    !j9vmthread 0x5fdb00 publicFlags=20001 privateFlags=0 inNative=0 // load-3 <--- J9_PUBLIC_FLAGS_THREAD_PARKED  | J9_PUBLIC_FLAGS_HALT_THREAD_EXCLUSIVE 
    !j9vmthread 0x600100 publicFlags=20001 privateFlags=0 inNative=0 // load-4
    !j9vmthread 0x602700 publicFlags=1 privateFlags=0 inNative=0 // load-5
    !j9vmthread 0x604d00 publicFlags=20001 privateFlags=0 inNative=0 // load-6
    !j9vmthread 0x607300 publicFlags=20001 privateFlags=0 inNative=0 // load-7
    !j9vmthread 0x609900 publicFlags=20001 privateFlags=0 inNative=0 // load-8
----> !j9vmthread 0x60be00 publicFlags=1001 privateFlags=0 inNative=0 // load-9 <---- J9_PUBLIC_FLAGS_NOT_AT_SAFE_POINT |  J9_PUBLIC_FLAGS_HALT_THREAD_EXCLUSIVE 
    !j9vmthread 0x6ef800 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-1
    !j9vmthread 0x6fe300 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-2
    !j9vmthread 0x737500 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-3
    !j9vmthread 0x745c00 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-4
    !j9vmthread 0x748200 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-5
    !j9vmthread 0x74e800 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-6
    !j9vmthread 0x754e00 publicFlags=1 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-8
...

#define J9_PUBLIC_FLAGS_HALT_THREAD_EXCLUSIVE 0x1
#define J9_PUBLIC_FLAGS_NOT_AT_SAFE_POINT 0x1000
#define J9_PUBLIC_FLAGS_THREAD_PARKED 0x20000
#define J9_PUBLIC_FLAGS_EXCLUSIVE_SET_NOT_SAFE 0x40000
#define J9_PUBLIC_FLAGS_HALT_THREAD_INSPECTION 0x8000

But the java stacktrace of thread load-9 mentioned in the description indicates it was totally in JIT:

> !stackslots 0x60be00
<60be00> *** BEGIN STACK WALK, flags = 00400001 walkThread = 0x000000000060BE00 ***
<60be00>        ITERATE_O_SLOTS
<60be00>        RECORD_BYTECODE_PC_OFFSET
<60be00> Initial values: walkSP = 0x00000000006227E0, PC = 0x0000000000000006, literals = 0x0000000000000000, A0 = 0x0000000000622948, j2iFrame = 0x00000000006229C8, ELS = 0x000003FF6C1FAA30, decomp = 0x0000000000000000
<60be00> JIT JNI call-out frame: bp = 0x0000000000622800, sp = 0x00000000006227E0, pc = 0x0000000000000006, cp = 0x00000000000B8C90, arg0EA = 0x0000000000622948, flags = 0x0000000020000000
<60be00>        Method: java/lang/Thread.getStateImpl(J)I !j9method 0x00000000000BB130
<60be00> JIT inline frame: bp = 0x00000000006228D8, pc = 0x000003FF6FB5839A, unwindSP = 0x0000000000622808, cp = 0x00000000000B8C90, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6C260738
<60be00>        Method: java/lang/Thread.threadState()Ljava/lang/Thread$State; !j9method 0x00000000000BAD90
<60be00>        Bytecode index = 43, inlineDepth = 2, PC offset = 0x000003FF6FB5764D
<60be00> JIT inline frame: bp = 0x00000000006228D8, pc = 0x000003FF6FB5839A, unwindSP = 0x0000000000622808, cp = 0x00000000000B8C90, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6C260738
<60be00>        Method: java/lang/Thread.isTerminated()Z !j9method 0x00000000000BADB0
<60be00>        Bytecode index = 1, inlineDepth = 1, PC offset = 0xFFFFFFFFFFFFF2A0
<60be00> JIT frame: bp = 0x00000000006228D8, pc = 0x000003FF6FB5839A, unwindSP = 0x0000000000622808, cp = 0x00000000000B8C90, arg0EA = 0x00000000006228E0, jitInfo = 0x000003FF6C260738
<60be00>        Method: java/lang/Thread.getThreadGroup()Ljava/lang/ThreadGroup; !j9method 0x00000000000BA990
<60be00>        Bytecode index = 1, inlineDepth = 0, PC offset = 0x00000000000001A0
<60be00>        stackMap=0x000003FF6C260B78, slots=I16(0x0001) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0001), localBaseOffset=I16(0xFFB8)
<60be00>        Described JIT args starting at 0x00000000006228E0 for U16(0x0001) slots
<60be00>                O-Slot: : a0[0x00000000006228E0] = 0x000000003A012430
<60be00>        Described JIT temps starting at 0x0000000000622890 for IDATA(0x0000000000000009) slots
<60be00>                O-Slot: : t8[0x0000000000622890] = 0x000000003A07F8C8
<60be00>                O-Slot: : t7[0x0000000000622898] = 0x000000003A07F8C8
<60be00>                I-Slot: : t6[0x00000000006228A0] = 0x00000000000BACD0
<60be00>                O-Slot: : t5[0x00000000006228A8] = 0x000000003A012430
<60be00>                I-Slot: : t4[0x00000000006228B0] = 0x00000000006228E8
<60be00>                O-Slot: : t3[0x00000000006228B8] = 0x000000003A012430
<60be00>                O-Slot: : t2[0x00000000006228C0] = 0x000000003A07F8C8
<60be00>                O-Slot: : t1[0x00000000006228C8] = 0x00000000B0472C00
<60be00>                O-Slot: : t0[0x00000000006228D0] = 0x000000003A012430
<60be00>        JIT-RegisterMap = UDATA(0x0000000000000000)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622838] = UDATA(0x000003FF6F644F70) (jit_r6)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622840] = UDATA(0x0000000000E16900) (jit_r7)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622848] = UDATA(0x00000000B0472108) (jit_r8)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622850] = UDATA(0x0000000000342C00) (jit_r9)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622858] = UDATA(0x0000000000000000) (jit_r10)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622860] = UDATA(0x00000000B0472108) (jit_r11)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622868] = UDATA(0x00000000B0472050) (jit_r12)
<60be00> I2J values: PC = 0x000003FF354F5CA5, A0 = 0x0000000000622948, walkSP = 0x00000000006228F0, literals = 0x00000000000DC7B8, JIT PC = 0x000003FF8BCDD7C0, pcAddress = 0x000003FF6C1FAA58, decomp = 0x0000000000000000
<60be00> Bytecode frame: bp = 0x0000000000622900, sp = 0x00000000006228F0, pc = 0x000003FF354F5CA5, cp = 0x00000000000DBD90, arg0EA = 0x0000000000622948, flags = 0x0000000000000000
<60be00>        Method: java/lang/ThreadGroup.enumerate([Ljava/lang/Thread;Z)I !j9method 0x00000000000DC7B8
<60be00>        Bytecode index = 45
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622948 for 0x0000000000000009 slots
<60be00>                O-Slot: a0[0x0000000000622948] = 0x00000000B046DC10
<60be00>                O-Slot: a1[0x0000000000622940] = 0x00000000B0471C18
<60be00>                I-Slot: a2[0x0000000000622938] = 0x00000001B046F558
<60be00>                I-Slot: t3[0x0000000000622930] = 0x0000000000000000
<60be00>                O-Slot: t4[0x0000000000622928] = 0x00000000B0472598
<60be00>                I-Slot: t5[0x0000000000622920] = 0x000000E1B0472598
<60be00>                I-Slot: t6[0x0000000000622918] = 0x0000001200000000
<60be00>                O-Slot: t7[0x0000000000622910] = 0x000000003A012430
<60be00>                I-Slot: t8[0x0000000000622908] = 0x0000000039A7EFB0
<60be00> J2I frame: bp = 0x00000000006229C8, sp = 0x0000000000622950, pc = 0x000003FF0011E54F, cp = 0x0000000000E61870, arg0EA = 0x0000000000622A08, flags = 0x0000000010000000
<60be00>        Method: gnu/testlet/java/lang/ThreadGroup/enumerate.test(Lgnu/testlet/TestHarness;)V !j9method 0x0000000000E612C8
<60be00>        Bytecode index = 75
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622A08 for 0x0000000000000008 slots
<60be00>                I-Slot: a0[0x0000000000622A08] = 0x00000000B046D0B0
<60be00>                O-Slot: a1[0x0000000000622A00] = 0x00000000B046DAA0
<60be00>                I-Slot: t2[0x00000000006229F8] = 0x0000000039A7EFB0
<60be00>                I-Slot: t3[0x00000000006229F0] = 0x00000000B046DC10
<60be00>                I-Slot: t4[0x00000000006229E8] = 0x00000000B046DC50
<60be00>                I-Slot: t5[0x00000000006229E0] = 0x00000000B046E460
<60be00>                I-Slot: t6[0x00000000006229D8] = 0x0000000A00000000
<60be00>                I-Slot: t7[0x00000000006229D0] = 0x00000000B0471C18
<60be00>        JIT-J2I-RegisterMap[0x00000000006229A8] = UDATA(0x000000003C4EC4B8) (jit_r6)
<60be00>        JIT-J2I-RegisterMap[0x00000000006229A0] = UDATA(0x000003FF6F7482F4) (jit_r7)
<60be00>        JIT-J2I-RegisterMap[0x0000000000622998] = UDATA(0x000000000008E200) (jit_r8)
<60be00>        JIT-J2I-RegisterMap[0x0000000000622990] = UDATA(0x000000003C4EC4B8) (jit_r9)
<60be00>        JIT-J2I-RegisterMap[0x0000000000622988] = UDATA(0x00000000B046DAA0) (jit_r10)
<60be00>        JIT-J2I-RegisterMap[0x0000000000622980] = UDATA(0x000000003C4EC4B8) (jit_r11)
<60be00>        JIT-J2I-RegisterMap[0x0000000000622978] = UDATA(0x0000000039A7EF00) (jit_r12)
<60be00> JIT frame: bp = 0x0000000000622A80, pc = 0x000003FF6FA4876A, unwindSP = 0x0000000000622A00, cp = 0x00000000005FF4C0, arg0EA = 0x0000000000622A98, jitInfo = 0x000003FF6D0C99F8
<60be00>        Method: java/lang/invoke/LambdaForm$DMH/0x00000000d80054f0.invokeVirtual(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V !j9method 0x00000000005FF5C8
<60be00>        Bytecode index = 11, inlineDepth = 0, PC offset = 0x000000000000017C
<60be00>        stackMap=0x000003FF6D0C9B20, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFFF8)
<60be00>        Described JIT args starting at 0x0000000000622A88 for U16(0x0003) slots
<60be00>                O-Slot: : a2[0x0000000000622A88] = 0x00000000B046DAA0
<60be00>                O-Slot: : a1[0x0000000000622A90] = 0x00000000B046D0B0
<60be00>                O-Slot: : a0[0x0000000000622A98] = 0x000000003C4EC548
<60be00>        Described JIT temps starting at 0x0000000000622A78 for IDATA(0x0000000000000001) slots
<60be00>                O-Slot: : t0[0x0000000000622A78] = 0x000000003C4EC5D8
<60be00>        JIT-RegisterMap = UDATA(0x0000000000000000)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622A20] = UDATA(0x000000003C4EC4B8) (jit_r6)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622A28] = UDATA(0x000003FF6F7482F4) (jit_r7)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622A30] = UDATA(0x000000000008E200) (jit_r8)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622A38] = UDATA(0x000000003C4EC4B8) (jit_r9)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622988] = UDATA(0x00000000B046DAA0) (jit_r10)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622980] = UDATA(0x000000003C4EC4B8) (jit_r11)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622978] = UDATA(0x0000000039A7EF00) (jit_r12)
<60be00> JIT frame: bp = 0x0000000000622B78, pc = 0x000003FF6F9A761C, unwindSP = 0x0000000000622A88, cp = 0x00000000006010C0, arg0EA = 0x0000000000622B90, jitInfo = 0x000003FF6D0D78B8
<60be00>        Method: java/lang/invoke/LambdaForm$MH/0x00000000d8005be0.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x00000000006011E8
<60be00>        Bytecode index = 52, inlineDepth = 0, PC offset = 0x000000000000022E
<60be00>        stackMap=0x000003FF6D0D7A30, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFF78)
<60be00>        Described JIT args starting at 0x0000000000622B80 for U16(0x0003) slots
<60be00>                O-Slot: : a2[0x0000000000622B80] = 0x00000000B046DAA0
<60be00>                O-Slot: : a1[0x0000000000622B88] = 0x00000000B046D0B0
<60be00>                O-Slot: : a0[0x0000000000622B90] = 0x000000003C4EC4B8
<60be00>        Described JIT temps starting at 0x0000000000622AF0 for IDATA(0x0000000000000011) slots
<60be00>                O-Slot: : t16[0x0000000000622AF0] = 0x00000000B046DAA0
<60be00>                O-Slot: : t15[0x0000000000622AF8] = 0x00000000B046D0B0
<60be00>                O-Slot: : t14[0x0000000000622B00] = 0x000000003C4EC548
<60be00>                O-Slot: : t13[0x0000000000622B08] = 0x00000000B046DAA0
<60be00>                O-Slot: : t12[0x0000000000622B10] = 0x00000000B046D0B0
<60be00>                O-Slot: : t11[0x0000000000622B18] = 0x000000003C4EC548
<60be00>                O-Slot: : t10[0x0000000000622B20] = 0x00000000B046DAA0
<60be00>                O-Slot: : t9[0x0000000000622B28] = 0x00000000B046DAA0
<60be00>                O-Slot: : t8[0x0000000000622B30] = 0x000000003C4EC578
<60be00>                O-Slot: : t7[0x0000000000622B38] = 0x00000000B046D0B0
<60be00>                O-Slot: : t6[0x0000000000622B40] = 0x00000000B046D0B0
<60be00>                O-Slot: : t5[0x0000000000622B48] = 0x000000003C4EC5A8
<60be00>                O-Slot: : t4[0x0000000000622B50] = 0x000000003C4EC548
<60be00>                O-Slot: : t3[0x0000000000622B58] = 0x00000000B046DAA0
<60be00>                O-Slot: : t2[0x0000000000622B60] = 0x000000003C4EC578
<60be00>                O-Slot: : t1[0x0000000000622B68] = 0x00000000B046D0B0
<60be00>                O-Slot: : t0[0x0000000000622B70] = 0x000000003C4EC5A8
<60be00>        JIT-RegisterMap = UDATA(0x0000000000000000)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622AA0] = UDATA(0x000003FF6F9A7400) (jit_r6)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622AA8] = UDATA(0x00000000B046DAA0) (jit_r7)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622AB0] = UDATA(0x00000000B046D0B0) (jit_r8)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622AB8] = UDATA(0x000000003C4EC4B8) (jit_r9)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622AC0] = UDATA(0x00000000B046DAA0) (jit_r10)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622980] = UDATA(0x000000003C4EC4B8) (jit_r11)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622978] = UDATA(0x0000000039A7EF00) (jit_r12)
<60be00> JIT inline frame: bp = 0x0000000000622C88, pc = 0x000003FF6FA8C726, unwindSP = 0x0000000000622B80, cp = 0x00000000004107C0, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6D0F2C78
<60be00>        Method: java/lang/invoke/LambdaForm$MH/0x000000008c40ee70.invokeExact_MT(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x00000000004108D8
<60be00>        Bytecode index = 20, inlineDepth = 4, PC offset = 0x000003FF6FA8C321
<60be00> JIT inline frame: bp = 0x0000000000622C88, pc = 0x000003FF6FA8C726, unwindSP = 0x0000000000622B80, cp = 0x00000000001F9EC0, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6D0F2C78
<60be00>        Method: jdk/internal/reflect/DirectMethodHandleAccessor.invokeImpl(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x00000000001FA798
<60be00>        Bytecode index = 55, inlineDepth = 3, PC offset = 0x000000000018D538
<60be00> JIT inline frame: bp = 0x0000000000622C88, pc = 0x000003FF6FA8C726, unwindSP = 0x0000000000622B80, cp = 0x00000000001F9EC0, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6D0F2C78
<60be00>        Method: jdk/internal/reflect/DirectMethodHandleAccessor.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x00000000001FA758
<60be00>        Bytecode index = 23, inlineDepth = 2, PC offset = 0x00000000001F9058
<60be00> JIT inline frame: bp = 0x0000000000622C88, pc = 0x000003FF6FA8C726, unwindSP = 0x0000000000622B80, cp = 0x00000000000B0850, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6D0F2C78
<60be00>        Method: java/lang/reflect/Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x00000000000B1C70
<60be00>        Bytecode index = 104, inlineDepth = 1, PC offset = 0x000000000043BED8
<60be00> JIT frame: bp = 0x0000000000622C88, pc = 0x000003FF6FA8C726, unwindSP = 0x0000000000622B80, cp = 0x0000000000597740, arg0EA = 0x0000000000622C90, jitInfo = 0x000003FF6D0F2C78
<60be00>        Method: net/adoptopenjdk/loadTest/adaptors/MauveAdaptor.executeTest()Lnet/adoptopenjdk/loadTest/adaptors/AdaptorInterface$ResultStatus; !j9method 0x0000000000597D20
<60be00>        Bytecode index = 104, inlineDepth = 0, PC offset = 0x0000000000000E2C
<60be00>        stackMap=0x000003FF6D0F5074, slots=I16(0x0001) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0001), localBaseOffset=I16(0xFF88)
<60be00>        Described JIT args starting at 0x0000000000622C90 for U16(0x0001) slots
<60be00>                O-Slot: : a0[0x0000000000622C90] = 0x000000003A18A568
<60be00>        Described JIT temps starting at 0x0000000000622C10 for IDATA(0x000000000000000F) slots
<60be00>                I-Slot: : t14[0x0000000000622C10] = 0x000000003C1DD598
<60be00>                I-Slot: : t13[0x0000000000622C18] = 0x0000000039A7D878
<60be00>                O-Slot: : t12[0x0000000000622C20] = 0x000000003C4EC468
<60be00>                O-Slot: : t11[0x0000000000622C28] = 0x00000000B046DBB8
<60be00>                I-Slot: : t10[0x0000000000622C30] = 0x0000000039A7D7F8
<60be00>                O-Slot: : t9[0x0000000000622C38] = 0x00000000B046DB60
<60be00>                I-Slot: : t8[0x0000000000622C40] = 0x0000000039F004D8
<60be00>                O-Slot: : t7[0x0000000000622C48] = 0x00000000B046D0B0
<60be00>                O-Slot: : t6[0x0000000000622C50] = 0x00000000B046DB60
<60be00>                I-Slot: : t5[0x0000000000622C58] = 0x000000003D716568
<60be00>                I-Slot: : t4[0x0000000000622C60] = 0x0000000039F00468
<60be00>                I-Slot: : t3[0x0000000000622C68] = 0x00000000B046DAA0
<60be00>                I-Slot: : t2[0x0000000000622C70] = 0x0000000039ADA6D8
<60be00>                I-Slot: : t1[0x0000000000622C78] = 0x00000000B046DAA0
<60be00>                I-Slot: : t0[0x0000000000622C80] = 0x0000000039F004D8
<60be00>        JIT-RegisterMap = UDATA(0x0000000000001C00)
<60be00>                JIT-RegisterMap-I-Slot[0x0000000000622AA0] = UDATA(0x000003FF6F9A7400) (jit_r6)
<60be00>                JIT-RegisterMap-I-Slot[0x0000000000622AA8] = UDATA(0x00000000B046DAA0) (jit_r7)
<60be00>                JIT-RegisterMap-I-Slot[0x0000000000622AB0] = UDATA(0x00000000B046D0B0) (jit_r8)
<60be00>                JIT-RegisterMap-I-Slot[0x0000000000622AB8] = UDATA(0x000000003C4EC4B8) (jit_r9)
<60be00>                JIT-RegisterMap-O-Slot[0x0000000000622AC0] = 0x00000000B046DAA0 (jit_r10)
<60be00>                JIT-RegisterMap-O-Slot[0x0000000000622980] = 0x000000003C4EC4B8 (jit_r11)
<60be00>                JIT-RegisterMap-O-Slot[0x0000000000622978] = 0x0000000039A7EF00 (jit_r12)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BB0] = UDATA(0x00000000B046C910) (jit_r6)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BB8] = UDATA(0x00000000B046CAB8) (jit_r7)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BC0] = UDATA(0x000000003C4CC980) (jit_r8)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BC8] = UDATA(0x000003FF6FA48600) (jit_r9)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BD0] = UDATA(0x000003FF6F748300) (jit_r10)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BD8] = UDATA(0x000003FF6F7482F4) (jit_r11)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BE0] = UDATA(0x0000000039A7EF00) (jit_r12)
<60be00> I2J values: PC = 0x000003FF8C550DA5, A0 = 0x0000000000622D18, walkSP = 0x0000000000622CA0, literals = 0x00000000005DE878, JIT PC = 0x000003FF8BCDD7C0, pcAddress = 0x0000000000622968, decomp = 0x0000000000000000
<60be00> Bytecode frame: bp = 0x0000000000622CB0, sp = 0x0000000000622CA0, pc = 0x000003FF8C550DA5, cp = 0x00000000005DE2C0, arg0EA = 0x0000000000622D18, flags = 0x0000000000000000
<60be00>        Method: net/adoptopenjdk/loadTest/LoadTestRunner$2.run()V !j9method 0x00000000005DE878
<60be00>        Bytecode index = 129
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622D18 for 0x000000000000000D slots
<60be00>                O-Slot: a0[0x0000000000622D18] = 0x000000003A012FA0
<60be00>                I-Slot: t1[0x0000000000622D10] = 0x00000000B49CD598
<60be00>                I-Slot: t2[0x0000000000622D08] = 0x00000000B49A7760
<60be00>                I-Slot: t3[0x0000000000622D00] = 0x0000000000000000
<60be00>                I-Slot: t4[0x0000000000622CF8] = 0x00000000005DE878
<60be00>                I-Slot: t5[0x0000000000622CF0] = 0x0000000000000000
<60be00>                O-Slot: t6[0x0000000000622CE8] = 0x000000003A012FC8
<60be00>                O-Slot: t7[0x0000000000622CE0] = 0x000000003A012FE0
<60be00>                O-Slot: t8[0x0000000000622CD8] = 0x000000003A18A568
<60be00>                I-Slot: t9[0x0000000000622CD0] = 0x0000000000000000
<60be00>                I-Slot: t10[0x0000000000622CC8] = 0x000000013A0101A8
<60be00>                I-Slot: t11[0x0000000000622CC0] = 0x000000000060BC88
<60be00>                I-Slot: t12[0x0000000000622CB8] = 0x000003FF3554DC6E
<60be00> Bytecode frame: bp = 0x0000000000622D30, sp = 0x0000000000622D20, pc = 0x000003FF35944416, cp = 0x0000000000518B10, arg0EA = 0x0000000000622D70, flags = 0x0000000000000000
<60be00>        Method: java/util/concurrent/ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V !j9method 0x0000000000519920
<60be00>        Bytecode index = 94
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622D70 for 0x0000000000000008 slots
<60be00>                O-Slot: a0[0x0000000000622D70] = 0x000000003A0123E0
<60be00>                O-Slot: a1[0x0000000000622D68] = 0x000000003A013000
<60be00>                O-Slot: t2[0x0000000000622D60] = 0x000000003A012EF0
<60be00>                O-Slot: t3[0x0000000000622D58] = 0x000000003A012FA0
<60be00>                I-Slot: t4[0x0000000000622D50] = 0x00000001B49A7788
<60be00>                I-Slot: t5[0x0000000000622D48] = 0x0000000000000000
<60be00>                I-Slot: t6[0x0000000000622D40] = 0x0000000000000000
<60be00>                I-Slot: t7[0x0000000000622D38] = 0x0000000000000000
<60be00> Bytecode frame: bp = 0x0000000000622D88, sp = 0x0000000000622D78, pc = 0x000003FF35A3B395, cp = 0x00000000005EAA00, arg0EA = 0x0000000000622D90, flags = 0x0000000000000000
<60be00>        Method: java/util/concurrent/ThreadPoolExecutor$Worker.run()V !j9method 0x00000000005EABC0
<60be00>        Bytecode index = 5
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622D90 for 0x0000000000000001 slots
<60be00>                I-Slot: a0[0x0000000000622D90] = 0x00000000B49A7788
<60be00> Bytecode frame: bp = 0x0000000000622DA8, sp = 0x0000000000622D98, pc = 0x000003FF354BB3E7, cp = 0x00000000000B8C90, arg0EA = 0x0000000000622DB8, flags = 0x0000000000000000
<60be00>        Method: java/lang/Thread.run()V !j9method 0x00000000000BA710
<60be00>        Bytecode index = 15
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622DB8 for 0x0000000000000002 slots
<60be00>                I-Slot: a0[0x0000000000622DB8] = 0x00000000B49A77B8
<60be00>                I-Slot: t1[0x0000000000622DB0] = 0x00000000B49A7788
<60be00> JNI call-in frame: bp = 0x0000000000622DE0, sp = 0x0000000000622DC0, pc = 0x000003FF909AAC86, cp = 0x0000000000000000, arg0EA = 0x0000000000622DE0, flags = 0x0000000000000000
<60be00>        New ELS = 0x0000000000000000
<60be00> JNI native method frame: bp = 0x0000000000622E08, sp = 0x0000000000622DE8, pc = 0x0000000000000007, cp = 0x0000000000000000, arg0EA = 0x0000000000622E08, flags = 0x0000000000000000
<60be00> <end of stack>
<60be00> *** END STACK WALK (rc = NONE) ***

I went through the code changes listed at 710fd95...c19e426, in which the only PR that looks suspicious is Z JIT Pinning Support #15752 (which was to support Virtual Threads in JIT on zLinux) given the hang never happens on other platforms and never occurs with JIT off on zLinux.

So I will revert the changes to see how it goes in Grinder.

ChengJin01 · 2022-10-17T03:25:41Z

So far the problem can only be produced on the failing build at https://openj9-jenkins.osuosl.org/job/Grinder/1408/consoleText as I tried both with the latest build and with the reverted change at https://github.com/ChengJin01/openj9/tree/revert_jit_zlinux_thrd_pin_15752 in which case the hang never occurred. I think the JIT team still need to take a look to see what happened in there.

FYI: @r30shah

gacholio · 2022-10-17T15:28:56Z

The impression I get from @pshipton is that this happened exactly once and has never been reproduced, so I'm not sure there's any reason to blame any change. Was the pinning support updated in the next build?

ChengJin01 · 2022-10-17T15:54:11Z

The impression I get from @pshipton is that this happened exactly once and has never been reproduced

As I mentioned above, the hang is only reproduced on the failing build but never on the build right before the failing build or on the latest build.

Was the pinning support updated in the next build?

I didn't see any update on these two files in #15752 or anything related to the pinning support after it was merged.

ChengJin01 · 2022-10-18T16:11:01Z

Just talked to GAC, we close the issue given there is no way to reproduce it in the latest build and no further evidence to prove the hang was caused by any change unless the problem shows up again.

pshipton added comp:vm test failure jdk19 labels Oct 4, 2022

pshipton added this to the Java 19 milestone Oct 4, 2022

ChengJin01 closed this as completed Oct 18, 2022

tajila mentioned this issue Oct 19, 2022

jdk19 MiniMix_aot_5m_0 hang #15917

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JDK 19 MiniMix_5m_1 hang in Thread.getStateImpl #16036

JDK 19 MiniMix_5m_1 hang in Thread.getStateImpl #16036

pshipton commented Oct 4, 2022

pshipton commented Oct 4, 2022

tajila commented Oct 4, 2022

ChengJin01 commented Oct 5, 2022 •

edited

Loading

pshipton commented Oct 5, 2022

ChengJin01 commented Oct 5, 2022

ChengJin01 commented Oct 5, 2022 •

edited

Loading

ChengJin01 commented Oct 5, 2022 •

edited

Loading

ChengJin01 commented Oct 6, 2022

tajila commented Oct 7, 2022

ChengJin01 commented Oct 7, 2022 •

edited

Loading

ChengJin01 commented Oct 7, 2022

ChengJin01 commented Oct 7, 2022

gacholio commented Oct 7, 2022

ChengJin01 commented Oct 11, 2022 •

edited

Loading

gacholio commented Oct 11, 2022

gacholio commented Oct 11, 2022

ChengJin01 commented Oct 11, 2022 •

edited

Loading

pshipton commented Oct 11, 2022

ChengJin01 commented Oct 12, 2022

gacholio commented Oct 14, 2022

ChengJin01 commented Oct 14, 2022 •

edited

Loading

ChengJin01 commented Oct 17, 2022 •

edited

Loading

gacholio commented Oct 17, 2022

ChengJin01 commented Oct 17, 2022 •

edited

Loading

ChengJin01 commented Oct 18, 2022

JDK 19 MiniMix_5m_1 hang in Thread.getStateImpl #16036

JDK 19 MiniMix_5m_1 hang in Thread.getStateImpl #16036

Comments

pshipton commented Oct 4, 2022

pshipton commented Oct 4, 2022

tajila commented Oct 4, 2022

ChengJin01 commented Oct 5, 2022 • edited Loading

pshipton commented Oct 5, 2022

ChengJin01 commented Oct 5, 2022

ChengJin01 commented Oct 5, 2022 • edited Loading

ChengJin01 commented Oct 5, 2022 • edited Loading

ChengJin01 commented Oct 6, 2022

tajila commented Oct 7, 2022

ChengJin01 commented Oct 7, 2022 • edited Loading

ChengJin01 commented Oct 7, 2022

ChengJin01 commented Oct 7, 2022

gacholio commented Oct 7, 2022

ChengJin01 commented Oct 11, 2022 • edited Loading

gacholio commented Oct 11, 2022

gacholio commented Oct 11, 2022

ChengJin01 commented Oct 11, 2022 • edited Loading

pshipton commented Oct 11, 2022

ChengJin01 commented Oct 12, 2022

gacholio commented Oct 14, 2022

ChengJin01 commented Oct 14, 2022 • edited Loading

ChengJin01 commented Oct 17, 2022 • edited Loading

gacholio commented Oct 17, 2022

ChengJin01 commented Oct 17, 2022 • edited Loading

ChengJin01 commented Oct 18, 2022

ChengJin01 commented Oct 5, 2022 •

edited

Loading

ChengJin01 commented Oct 5, 2022 •

edited

Loading

ChengJin01 commented Oct 5, 2022 •

edited

Loading

ChengJin01 commented Oct 7, 2022 •

edited

Loading

ChengJin01 commented Oct 11, 2022 •

edited

Loading

ChengJin01 commented Oct 11, 2022 •

edited

Loading

ChengJin01 commented Oct 14, 2022 •

edited

Loading

ChengJin01 commented Oct 17, 2022 •

edited

Loading

ChengJin01 commented Oct 17, 2022 •

edited

Loading