JDK15 cmake : DDR is not enabled for this core file

[JasonFengJ9]

Failure link

From an internal build Test_openjdknext_j9_extended.functional_x86-64_linux_cm_Nightly/4:

23:40:50  openjdk version "15-internal" 2020-09-15
23:40:50  OpenJDK Runtime Environment (build 15-internal+0-adhoc.jenkins.BuildJDKnextx86-64linuxcmNightly)
23:40:50  Eclipse OpenJ9 VM (build ibm_sdk-f67b1960a5, JRE 15 Linux amd64-64-Bit Compressed References 20200418_4 (JIT enabled, AOT enabled)
23:40:50  OpenJ9   - f67b1960a5
23:40:50  OMR      - 2c11feebc
23:40:50  JCL      - 637204c3307 based on jdk-15+18)

Optional info

Failure output (captured from console output)

Testing: Run !allclasses
Test start time: 2020/04/19 01:44:34 Eastern Standard Time
Running command: /home/jenkins/workspace/Test_openjdknext_j9_extended.functional_x86-64_linux_cm_Nightly/openjdkbinary/j2sdk-image/bin/jdmpview -core j9core.dmp
Time spent starting: 7 milliseconds
Time spent executing: 902 milliseconds
Test result: FAILED
 [OUT] DTFJView version 4.29.5, using DTFJ version 1.12.29003
 [OUT] Loading image from DTFJ...
 [OUT] 
 [OUT] Could not load dump file and/or could not load XML file: Could not create a valid ImageFactory
 [OUT] For a list of commands, type "help"; for how to use "help", type "help help"
 [OUT] > DDR is not enabled for this core file, '!' commands are disabled
 [OUT] > 
>> Success condition was not found: [Output match: java/lang/Object]
>> Failure condition was not found: [Output match: DDRInteractiveCommandException]
>> Failure condition was not found: [Output match: unable to read]
>> Failure condition was not found: [Output match: could not read]
>> Failure condition was not found: [Output match: dump event]

cmdLineTester_classesdbgddrext_1_FAILED

For example, to rebuild the failed tests in =https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder, use the following links:
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/parambuild/?JDK_VERSION=15&JDK_IMPL=openj9&BUILD_LIST=functional&PLATFORM=x86-64_linux&TARGET=cmdLineTester_classesdbgddrext_1

[pshipton]

Unless this happens regularly, it's probably just a bad core file problem. Core file was truncated due to lack of disk space or similar intermittent issue.

[JasonFengJ9]

It happened quit a bit. Will post more for references.

[pshipton]

Does it occur non-cmake as well?

[pshipton]

Might be worth tracking machines where it fails, it could be a machine issue.

[JasonFengJ9]

Does it occur non-cmake as well?

Not seen in other build specs last nightly builds.

Might be worth tracking machines where it fails, it could be a machine issue.

It happened at past three builds in different machines: build 16 at rhel6x64rt13-1, build 15 at ub16x64cudavm2, build 14 at cent6x64rt2-1.

[dnakamura]

For whatever reason we appear to disable ddr on jdk14/next. Testing fix now

[sharon-wang]

Has this been fixed? Seeing a similar issue for a mixed refs JDK 11 CMake build and want to see if this issue is related.

From: https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_s390x_linux_mixed_cm_Nightly/2/tapResults/

Testing: Run !printallcallsites
Test start time: 2020/12/10 20:49:41 Coordinated Universal Time
Running command: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_mixed_cm_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/jdmpview -core j9core.dmp
Time spent starting: 2 milliseconds
Time spent executing: 3048 milliseconds
Test result: FAILED
Output from test:
 [OUT] DTFJView version 4.29.5, using DTFJ version 1.12.29003
 [OUT] Loading image from DTFJ...
 [OUT] 
 [OUT] Warning: dump file was produced by a different, possibly incompatible, VM.
 [OUT]   core file: (unknown)
 [OUT]   this VM:   Eclipse OpenJ9 VM
 [OUT] 
 [OUT] For a list of commands, type "help"; for how to use "help", type "help help"
 [OUT] Available contexts (* = currently selected context) : 
 [OUT] 
 [OUT] Source : file:///home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_mixed_cm_Nightly_testList_1/openjdk-tests/TKG/test_output_16076320077235/cmdLineTester_callsitedbgddrext_openj9_0/j9core.dmp
 [OUT] 	*0 : JRE 11 Linux s390x-64 (build 11.0.10-internal+0-adhoc.jenkins.BuildJDK11s390xlinuxmixedcmNightly)
 [OUT] 
 [OUT] > DDR is not enabled for this core file, '!' commands are disabled
 [OUT] > 
>> Success condition was not found: [Output match: jvminit.c]
>> Failure condition was not found: [Output match: DDRInteractiveCommandException]
>> Failure condition was not found: [Output match: no shared cache]
>> Failure condition was not found: [Output match: unable to read]
>> Failure condition was not found: [Output match: could not read]
>> Failure condition was not found: [Output match: dump event]

Testing: Run !findallcallsites
Test start time: 2020/12/10 20:49:44 Coordinated Universal Time
Running command: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_mixed_cm_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/jdmpview -core j9core.dmp
Time spent starting: 0 milliseconds
Time spent executing: 1680 milliseconds
Test result: FAILED
Output from test:
 [OUT] DTFJView version 4.29.5, using DTFJ version 1.12.29003
 [OUT] Loading image from DTFJ...
 [OUT] 
 [OUT] Warning: dump file was produced by a different, possibly incompatible, VM.
 [OUT]   core file: (unknown)
 [OUT]   this VM:   Eclipse OpenJ9 VM
 [OUT] 
 [OUT] For a list of commands, type "help"; for how to use "help", type "help help"
 [OUT] Available contexts (* = currently selected context) : 
 [OUT] 
 [OUT] Source : file:///home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_mixed_cm_Nightly_testList_1/openjdk-tests/TKG/test_output_16076320077235/cmdLineTester_callsitedbgddrext_openj9_0/j9core.dmp
 [OUT] 	*0 : JRE 11 Linux s390x-64 (build 11.0.10-internal+0-adhoc.jenkins.BuildJDK11s390xlinuxmixedcmNightly)
 [OUT] 
 [OUT] > DDR is not enabled for this core file, '!' commands are disabled
 [OUT] > 
>> Success condition was not found: [Output match: jvminit.c]
>> Failure condition was not found: [Output match: DDRInteractiveCommandException]
>> Failure condition was not found: [Output match: no shared cache]
>> Failure condition was not found: [Output match: unable to read]
>> Failure condition was not found: [Output match: could not read]
>> Failure condition was not found: [Output match: dump event]

[sharon-wang]

May also be related to: adoptium/run-aqa#50

[pshipton]

I expect this Issue was a different problem from #9284 (comment). For this Issue DDR just wasn't enabled in the build. For #9284 (comment), I believe DDR is enabled. The JVM was found in the core since *0 : JRE 11 Linux s390x-64 (build 11.0.10-internal+0-adhoc.jenkins.BuildJDK11s390xlinuxmixedcmNightly) is shown. It's not clear why DDR isn't enabled, I suspect it can't read certain things it needs or reads as a sanity check.

I found the following DDR logging info. I also found CombinedContext.java which sets a ddrAvailable flag and uses a logger called com.ibm.jvm.dtfjview.logger. I'd start by turning that logger on in order to get info about why it's not available.

I was able to get some logging output by keeping the default ConsoleHanlder, setting java.util.logging.ConsoleHandler.level to ALL, and turning on some of the loggers (setting them to ALL).

Enabling logging output from DDR code

Add these lines to conf/logging.properties, selecting which parts of DDR you want to log, see /DDR_VM/src/com/ibm/j9ddr/logging/LoggerNames.java

# ConsoleHandler is the default. Add FileHandler, to route detailed log output to a file
handlers= java.util.logging.FileHandler, java.util.logging.ConsoleHandler

# default file output is in user's home directory.
java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
java.util.logging.FileHandler.level=ALL

# select which parts of DDR you want to log, see /DDR_VM/src/com/ibm/j9ddr/logging/LoggerNames.java
j9ddr.walkers.level = ALL
j9ddr.structure_reader.level = ALL
j9ddr.stackwalker.level = ALL
j9ddr.stackwalker.localmap.level = ALL
j9ddr.stackwalker.stackmap.level = ALL
j9ddr.walkers.pool.level = ALL
j9ddr.view.dtfj.level = ALL
j9ddr.interactive.context.level = ALL

[pshipton]

I'm going to close this obsolete issue, we should create another one for the mixed build DDR problem.

[pshipton]

Interesting to note for the mixed build problem, that only BE platforms failed the DDR test, zlinux and AIX.

[pshipton]

What worked to show the problem was running jdmpview with -verbose, which shows a stack trace for UnsupportedOperationException: GC policy not supported.

java.lang.ExceptionInInitializerError
at java.base/java.lang.J9VMInternals.ensureError(J9VMInternals.java:185)
at java.base/java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:174)
at com.ibm.j9ddr.vm29.tools.ddrinteractive.GetCommandsTask.run(GetCommandsTask.java:184)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.ibm.j9ddr.vm29.j9.VMData.bootstrapClass(VMData.java:65)
at com.ibm.j9ddr.vm29.j9.VMData.bootstrapRelative(VMData.java:44)
at com.ibm.j9ddr.tools.ddrinteractive.Context.getContextCommands(Context.java:97)
at com.ibm.j9ddr.tools.ddrinteractive.Context.refreshCommandList(Context.java:87)
at com.ibm.j9ddr.tools.ddrinteractive.Context.(Context.java:68)
at com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive.addDDRContextFromDTFJ(DDRInteractive.java:226)
at com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive.(DDRInteractive.java:155)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.startDDRInteractiveSession(CombinedContext.java:219)
at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.OpenCommand.createCombinedContext(OpenCommand.java:251)
at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.OpenCommand.createContexts(OpenCommand.java:200)
at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.OpenCommand.imagesFromCommandLine(OpenCommand.java:123)
at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.OpenCommand.run(OpenCommand.java:83)
at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.tryCommand(Context.java:141)
at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.execute(Context.java:97)
at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.execute(CombinedContext.java:173)
at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.execute(CombinedContext.java:157)
at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.imageFromCommandLine(Session.java:610)
at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.sessionInit(Session.java:226)
at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.(Session.java:175)
at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.getInstance(Session.java:171)
at openj9.dtfjview/com.ibm.jvm.dtfjview.DTFJView.launch(DTFJView.java:51)
at openj9.dtfjview/com.ibm.jvm.dtfjview.DTFJView.main(DTFJView.java:46)
Caused by: java.lang.UnsupportedOperationException: GC policy not supported
at com.ibm.j9ddr.vm29.j9.gc.GCHeapMap.from(GCHeapMap.java:104)
at com.ibm.j9ddr.vm29.tools.ddrinteractive.commands.MarkMapCommand.(MarkMapCommand.java:51)
... 31 more

[sharon-wang]

Opened #11469 to track the mixed refs failure.

Thank you for digging into this! GC policy not supported is interesting to see. My initial hunch from looking into this error is that there might be some conflicting build flags in the mixed refs build, since it tries to use both compressed and full build flags. I'll look into this more and post updates in the new issue.

Interesting to note for the mixed build problem, that only BE platforms failed the DDR test, zlinux and AIX.

Ah, I tried to reproduce on xlinux and couldn't...so this makes since. I'll do further investigation on zlinux or AIX.