jdk_management tests consistently fail on test-marist-ubuntu2204-s390x-1

[sophia-guo]

Some jdk_managment jdp tests consitently fail on test-marist-ubuntu2204-s390x-1

sun/management/jdp/JdpDefaultsTest.java.JdpDefaultsTest
sun/management/jdp/JdpJmxRemoteDynamicPortTest.java.JdpJmxRemoteDynamicPortTest
sun/management/jdp/JdpSpecificAddressTest.java.JdpSpecificAddressTest

'2023-09-12 22:15:58 FINE    Test started. 
''2023-09-12 22:15:58 FINE    Listening for multicast packets at 224.0.1.2:1234 
''2023-09-12 22:15:58 FINE    Waiting for 3 packets with jdp.name=6c239bc6-87ae-4392-a444-ebaf30a5f4df 
''2023-09-12 22:15:58 FINE    Pause in between packets is: 1 seconds. 
''2023-09-12 22:15:58 FINE    Timeout set to 10 seconds. 
''2023-09-12 22:16:08 SEVERE  Timed out waiting for JDP packet. Should arrive within 1 seconds, but waited for 10 seconds. 
''2023-09-12 22:16:08 SEVERE  Timed out waiting for JDP packet. Should arrive within 1 seconds, but waited for 10 seconds. 
'Exception in thread "main" java.lang.Exception: Timed out waiting for JDP packet. Should arrive within 1 seconds, but waited for 10 seconds.
	at JdpOnTestCase.onSocketTimeOut(JdpOnTestCase.java:75)
	at JdpTestCase.run(JdpTestCase.java:77)
	at JdpOnTestCase.main(JdpOnTestCase.java:105)
Caused by: java.net.SocketTimeoutException: Receive timed out
	at java.base/sun.nio.ch.DatagramChannelImpl.trustedBlockingReceive(DatagramChannelImpl.java:744)
	at java.base/sun.nio.ch.DatagramChannelImpl.blockingReceive(DatagramChannelImpl.java:673)
	at java.base/sun.nio.ch.DatagramSocketAdaptor.receive(DatagramSocketAdaptor.java:241)
	at java.base/java.net.DatagramSocket.receive(DatagramSocket.java:714)
	at JdpTestCase.run(JdpTestCase.java:74)
	... 1 more

 stdout: [];
 stderr: ['2023-09-12 22:15:58 FINE    Test started. 
''2023-09-12 22:15:58 FINE    Listening for multicast packets at 224.0.1.2:1234 
''2023-09-12 22:15:58 FINE    Waiting for 3 packets with jdp.name=6c239bc6-87ae-4392-a444-ebaf30a5f4df 
''2023-09-12 22:15:58 FINE    Pause in between packets is: 1 seconds. 
''2023-09-12 22:15:58 FINE    Timeout set to 10 seconds. 
''2023-09-12 22:16:08 SEVERE  Timed out waiting for JDP packet. Should arrive within 1 seconds, but waited for 10 seconds. 
''2023-09-12 22:16:08 SEVERE  Timed out waiting for JDP packet. Should arrive within 1 seconds, but waited for 10 seconds. 
'Exception in thread "main" java.lang.Exception: Timed out waiting for JDP packet. Should arrive within 1 seconds, but waited for 10 seconds.
	at JdpOnTestCase.onSocketTimeOut(JdpOnTestCase.java:75)
	at JdpTestCase.run(JdpTestCase.java:77)
	at JdpOnTestCase.main(JdpOnTestCase.java:105)
Caused by: java.net.SocketTimeoutException: Receive timed out
	at java.base/sun.nio.ch.DatagramChannelImpl.trustedBlockingReceive(DatagramChannelImpl.java:744)
	at java.base/sun.nio.ch.DatagramChannelImpl.blockingReceive(DatagramChannelImpl.java:673)
	at java.base/sun.nio.ch.DatagramSocketAdaptor.receive(DatagramSocketAdaptor.java:241)
	at java.base/java.net.DatagramSocket.receive(DatagramSocket.java:714)
	at JdpTestCase.run(JdpTestCase.java:74)
	... 1 more
]
 exitValue = 1

java.lang.RuntimeException: Expected to get exit value of [0], exit value is: [1]
	at jdk.test.lib.process.OutputAnalyzer.shouldHaveExitValue(OutputAnalyzer.java:490)
	at DynamicLauncher.run(DynamicLauncher.java:61)
	at JdpSpecificAddressTest.main(JdpSpecificAddressTest.java:47)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
	at java.base/java.lang.Thread.run(Thread.java:1583)

JavaTest Message: Test threw exception: java.lang.RuntimeException: Expected to get exit value of [0], exit value is: [1]
JavaTest Message: shutting down test

STATUS:Failed.`main' threw exception: java.lang.RuntimeException: Expected to get exit value of [0], exit value is: [1]

[sxa]

Multicast/s390x generic issue - can we fix it

[sxa]

TODO: Run these on a full set of machines similar to #2807 (comment)

[sxa]

Machine	Time	Result
test-docker-U2404	22m	4 failures [*]
test-docker-sles12	22m	4 failures [*]
test-marist-sles15		32 failures. No route to host and multicast failures
test-docker-UBI9	21m	4 failures [*]
test-marist-rhel7		32 failures - similar to sles15
test-marist-U2204	23m	10 failures
test-marist-U2404	29m	10 failures

[*] Systems with four failures are two tests repeated twice with a Compilation failed: Compilation failed error:
- sun/management/jmxremote/startstop/JMXStatusPerfCountersTest.java.JMXStatusPerfCountersTestsun/management/jmxremote/startstop/JMXStartStopTest.java.JMXStartStopTest 0.48 sec 1

• sun/management/jmxremote/startstop/JMXStatusPerfCountersTest.java.JMXStatusPerfCountersTest

The systems with 10 failures have an additional three (repeated twice) which are the ones in the original description:

• sun/management/jdp/JdpDefaultsTest.java.JdpDefaultsTest
• sun/management/jdp/JdpJmxRemoteDynamicPortTest.java.JdpJmxRemoteDynamicPortTest
• sun/management/jdp/JdpSpecificAddressTest.java.JdpSpecificAddressTest

Re-grind link with the 7 tests is here (change LABEL to the machine you want to run on)

[sxa]

Compilation failures should be checked for any corresponding issue upstream or in AQA tests and then have a short look to see if there's a specific compilation failure message that we can look into (as it seems quite likely that it is s390x specific and therefore we may be the first people hitting it)

[jiekang]

The compilation failure is logged in the console output:

16:37:16  /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/management/jmxremote/startstop/JMXStartStopTest.java:75: error: incompatible types: String cannot be converted to Process
16:37:16      private static ManagementAgentJcmd jcmd = new ManagementAgentJcmd(TEST_APP_NAME, verbose);

When reviewing this against the latest source code in jdk17u-dev, etc. it seems that the test material is out of date or out of sync in some way? The latest source has this relevant fix, which will be in the January 2025 release, 17.0.14:

openjdk/jdk17u-dev@848d8e8

[smlambert]

https://ci.adoptium.net/job/Grinder/11803/ is likely a rerun from a release build, so is using USE_TESTENV_PROPERTIES=true, so is pinning to a particular tagged version of test material which is shown in the testenv.properties file which would have originally been attached as an artifact to that Grinder run, though our Jenkins settings don't keep those artifacts forever, and it is no longer attached. One can also see the SHAs that were used at the top of TAP files attached to the job.

# Fetch URL: https://github.com/adoptium/jdk17u.git
# sha:
# 2695db6d1f6c773ea746ecf6c57eda586a581a10

[sxa]

Thanks for the investigation @jiekang - and good spot on that option @smlambert !

Re-runs on other machines:

• test-docker-sles12-s390x-1 ✅
• test-marist-sles12-s390x-1 ❌ 30 failures
• test-marist-ubuntu2204-s390x-1 ❌ 6 failures
• test-marist-ubuntu2404-s390x-1 ❌ 6 failures

The six failures are the same three Jdp ones from the original description in this issue (three repeated twice)

[sxa]

test-marist-sles12-s390x has now been taken offline.