[harfbuzz] jdk_beans_1 FAILED in Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1 with UnsatisfiedLinkError

[smlambert]

jdk_beans testcases fail on JDK11 AIX with java.lang.UnsatisfiedLinkError rtld: 0712-001 Symbol _ZNK2OT10postV2Tail6subsetEP19hb_subset_context_t was referenced from module /home/jenkins/workspace/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1/openjdkbinary/j2sdk-image/lib/libfontmanager.so(), but a runtime definition of the symbol wa

[2023-07-01T22:05:22.084Z] ACTION: main -- Failed. Execution failed: `main' threw exception: java.lang.UnsatisfiedLinkError: /home/jenkins/workspace/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1/openjdkbinary/j2sdk-image/lib/libfontmanager.so: /home/jenkins/workspace/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1/openjdkbinary/j2sdk-image/lib/libfontmanager.so, LIBPATH=/home/jenkins/workspace/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1/openjdkbinary/j2sdk-image/lib/server:/home/jenkins/workspace/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1/openjdkbinary/j2sdk-image/lib:/home/jenkins/workspace/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1/openjdkbinary/j2sdk-image/lib/jli:/home/jenkins/workspace/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1/openjdkbinary/j2sdk-image/../lib, LD_LIBRARY_PATH= : rtld: 0712-001 Symbol _ZNK2OT10postV2Tail6subsetEP19hb_subset_context_t was referenced from module /home/jenkins/workspace/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1/openjdkbinary/j2sdk-image/lib/libfontmanager.so(), but a runtime definition of the symbol wa

Test Info
Test Name: jdk_beans_1
Test Duration: 11 min 45 sec
Machine: test-osuosl-aix72-ppc64-6
TRSS link for the test output: https://trss.adoptium.net/output/test?id=64b8535d17052c6715871023

Build Info
Build Name: Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1
Jenkins Build start time: Jul 19 2023, 07:15 am
Jenkins Build URL: https://ci.adoptium.net/job/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1/78/
TRSS link for the build: https://trss.adoptium.net/allTestsInfo?buildId=64b8527817052c6715870b1d

Java Version
openjdk version "11.0.20" 2023-07-18
OpenJDK Runtime Environment Temurin-11.0.20+8 (build 11.0.20+8)
OpenJDK 64-Bit Server VM Temurin-11.0.20+8 (build 11.0.20+8, mixed mode)

This test has been failed 3 times since Jul 01 2023, 02:20 pm
Java Version when the issue first seen
openjdk version "11.0.20-beta" 2023-07-18
OpenJDK Runtime Environment Temurin-11.0.20+7-202307011712 (build 11.0.20-beta+7-202307011712)
OpenJDK 64-Bit Server VM Temurin-11.0.20+7-202307011712 (build 11.0.20-beta+7-202307011712, mixed mode)
Jenkins Build URL: https://ci.adoptium.net/job/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_1/76/

The test failed on machine test-osuosl-aix72-ppc64-6 2 times
The test failed on machine test-osuosl-aix72-ppc64-5 1 times

Rerun in Grinder

[andrew-m-leonard]

So hb_subset_context_t is a Harfbuzz symbol: https://github.com/harfbuzz/harfbuzz/blob/60db142784bb145426e4a02635bbad4f2f260d16/src/hb-subset.hh#L40
and I am suspecting the change in Harfbuzz version 7.0.1 might be the culprit: openjdk/jdk17u-dev@f969f81
i'm trying to figure why we don't have the right Harfbuzz dynamic library?

[tellison]

See also eclipse-openj9/openj9#17759

[andrew-m-leonard]

As a wild guess, i'm going to try re-building, without this optimization: https://github.com/adoptium/jdk17u/blob/3197a9b02992c9f95154b4b5e2b420e82f420cf2/make/modules/java.desktop/lib/Awt2dLibraries.gmk#L494
As the failing symbol is in that source.

[andrew-m-leonard]

See harfbuzz issue: harfbuzz/harfbuzz#4138

[andrew-m-leonard]

@ShelleyLambert @tellison
The harfbuzz issue harfbuzz/harfbuzz#4138 is still open, with no resolution yet.
Openj9 have reverted Harfbuzz 7.0.1 to 4.4.1, eclipse-openj9/openj9#17759 (comment)
We need to make the decision on whether to do the same?

[andrew-m-leonard]

Note, the revert is several hundred files...

[andrew-m-leonard]

I'm going to have a go at reverting this change: https://github.com/adoptium/jdk17u/blob/3197a9b02992c9f95154b4b5e2b420e82f420cf2/src/java.desktop/share/native/libharfbuzz/hb-subset.cc#L47
which seems a bit too much, and will see if the original problem can be work'd around some other way

[andrew-m-leonard]

The root compilation failure in
https://github.com/adoptium/jdk17u/blob/3197a9b02992c9f95154b4b5e2b420e82f420cf2/src/java.desktop/share/native/libharfbuzz/hb-ot-post-table-v2subset.hh#L81
is described in this comment:
harfbuzz/harfbuzz#4138 (comment)
Which fails specifically with Xlc 16 on AIX.
This needs focus from Harfbuzz upstream, who require access to an AIX box to debug..

[andrew-m-leonard]

Raised OpenJDK bug for this issue: https://bugs.openjdk.org/browse/JDK-8313240

[andrew-m-leonard]

Build with new Harfbuzz patch completes successfully: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-aix-ppc64-temurin/276/

Running jdk_beans_1 group to confirm: https://ci.adoptium.net/job/Grinder/7411/

[andrew-m-leonard]

jdk_beans_1 now passes: https://ci.adoptium.net/job/Grinder/7411/
So fix looks good.

[andrew-m-leonard]

So the patch applied to harfbuzz 7.0.1 does not compile with jdk-17.0.9+9 based on Harfbuzz 7.2.0.

I am trying a Harfbuzz 8.2.2 base on jdk-17.0.9+9, to verify a complete upgrade at least builds: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-aix-ppc64-temurin/321/

[pshipton]

FYI I believe these are the changes OpenJ9 used to fix Harfbuzz 7.20 for AIX.

ibmruntimes/openj9-openjdk-jdk17@55e75c010
ibmruntimes/openj9-openjdk-jdk17@66679e2dc

[andrew-m-leonard]

FYI I believe these are the changes OpenJ9 used to fix Harfbuzz 7.20 for AIX.

ibmruntimes/openj9-openjdk-jdk17@55e75c010 ibmruntimes/openj9-openjdk-jdk17@66679e2dc

Thank you @pshipton that 2nd commit is probably the one I am missing, i'll give that a try, thanks

[andrew-m-leonard]

test build with additional commit patch: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-aix-ppc64-temurin/323/

[andrew-m-leonard]

So the patch applied to harfbuzz 7.0.1 does not compile with jdk-17.0.9+9 based on Harfbuzz 7.2.0.

I am trying a Harfbuzz 8.2.2 base on jdk-17.0.9+9, to verify a complete upgrade at least builds: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-aix-ppc64-temurin/321/

Upgrading harfbuzz is not straightfoward, as it's not a straight copy....

[andrew-m-leonard]

test build with additional commit patch: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-aix-ppc64-temurin/323/

Successful build, so patch looks good from compilation perspective.

[smlambert]

Plan to exclude testcase until the upstream fix lands for April CPU, check via https://ci.adoptium.net/job/Grinder/9026 what failures relate to this issue.

[smlambert]

Appears the testcase is already excluded as Grinder/9026 is green. So will return to this ahead of April, to see what can be re-enabled, moving to a later iteration.