Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump native-maven-plugin from 0.10.2 to 0.10.3 #1143

Merged
merged 6 commits into from
Oct 9, 2024

Conversation

cstamas
Copy link
Member

@cstamas cstamas commented Sep 20, 2024

The plugin introduced some "windows fix"[1] that causes current arg parsing to fail. Solution seems to be to split each arg in own line, but that introduced another set of challenges.

[1] graalvm/native-build-tools#609

Supersedes #1133

The plugin introduced some "windows fix"[1] that causes
current arg parsing to fail. Solution seems to be to
split each arg in own line, but that introduced another
set of challenges.

[1] graalvm/native-build-tools#609
@cstamas cstamas requested a review from gnodet September 20, 2024 14:18
@cstamas cstamas self-assigned this Sep 20, 2024
@cstamas cstamas changed the base branch from master to mvnd-1.x September 20, 2024 14:19
@cstamas cstamas changed the base branch from mvnd-1.x to master September 20, 2024 14:20
@cstamas cstamas added the dependencies Pull requests that update a dependency file label Sep 20, 2024
@cstamas cstamas added this to the 2.0.0-beta-2 milestone Sep 20, 2024
@cstamas
Copy link
Member Author

cstamas commented Oct 7, 2024

@gzm55 help please! The glibc patching test fails now, but this made me look in it more deeper and questions just grow...

Your related commit: da20df0

We use GraalVM 22 to build (and release) mvnd, and I checked on my box GraalVM 22.0.2-graalce installation (Fedora 40 WS x86_64) the contents of the directory referenced from CI script "$GRAALVM_HOME/lib/static/linux-amd64/glibc" and the referenced files are not existing as I see? My Linux box has these files:

[cstamas@angeleyes 22.0.2-graalce]$ ll lib/static/linux-amd64/glibc/
total 93220
-rw-r--r--. 1 cstamas cstamas     9340 Jun 29 20:14 libattach.a
-rw-r--r--. 1 cstamas cstamas  1312360 Jun 29 20:14 libawt.a
-rw-r--r--. 1 cstamas cstamas    83056 Jun 29 20:14 libawt_headless.a
-rw-r--r--. 1 cstamas cstamas  1227658 Jun 29 20:14 libawt_xawt.a
-rw-r--r--. 1 cstamas cstamas    44062 Jun 29 20:14 libdt_socket.a
-rw-r--r--. 1 cstamas cstamas    15934 Jun 29 20:14 libextnet.a
-rw-r--r--. 1 cstamas cstamas  2877468 Jun 29 20:14 libfontmanager.a
-rw-r--r--. 1 cstamas cstamas   119826 Jun 29 20:14 libinstrument.a
-rw-r--r--. 1 cstamas cstamas    97556 Jun 29 20:14 libj2gss.a
-rw-r--r--. 1 cstamas cstamas    20208 Jun 29 20:14 libj2pcsc.a
-rw-r--r--. 1 cstamas cstamas   194676 Jun 29 20:14 libj2pkcs11.a
-rw-r--r--. 1 cstamas cstamas     3918 Jun 29 20:14 libjaas.a
-rw-r--r--. 1 cstamas cstamas   342684 Jun 29 20:14 libjava.a
-rw-r--r--. 1 cstamas cstamas   391470 Jun 29 20:14 libjavajpeg.a
-rw-r--r--. 1 cstamas cstamas     2496 Jun 29 20:14 libjawt.a
-rw-r--r--. 1 cstamas cstamas   859762 Jun 29 20:14 libjdwp.a
-rw-r--r--. 1 cstamas cstamas    60462 Jun 29 20:14 libjimage.a
-rw-r--r--. 1 cstamas cstamas   148078 Jun 29 20:14 libjli.a
-rw-r--r--. 1 cstamas cstamas    18350 Jun 29 20:14 libjpackageapplauncher.a
-rw-r--r--. 1 cstamas cstamas   570102 Jun 29 20:14 libjpackageapplauncheraux.a
-rw-r--r--. 1 cstamas cstamas    10378 Jun 29 20:14 libjsig.a
-rw-r--r--. 1 cstamas cstamas   146176 Jun 29 20:14 libjsound.a
-rw-r--r--. 1 cstamas cstamas  1066052 Jun 29 20:14 libjsvml.a
-rw-r--r--. 1 cstamas cstamas 82384838 Jun 29 20:14 libjvm.a
-rw-r--r--. 1 cstamas cstamas   987482 Jun 29 20:14 liblcms.a
-rw-r--r--. 1 cstamas cstamas    13138 Jun 29 20:14 lible.a
-rw-r--r--. 1 cstamas cstamas    46946 Jun 29 20:14 libmanagement.a
-rw-r--r--. 1 cstamas cstamas     3032 Jun 29 20:14 libmanagement_agent.a
-rw-r--r--. 1 cstamas cstamas    56800 Jun 29 20:14 libmanagement_ext.a
-rw-r--r--. 1 cstamas cstamas   754928 Jun 29 20:14 libmlib_image.a
-rw-r--r--. 1 cstamas cstamas   121346 Jun 29 20:14 libnet.a
-rw-r--r--. 1 cstamas cstamas   227622 Jun 29 20:14 libnio.a
-rw-r--r--. 1 cstamas cstamas     4176 Jun 29 20:14 libprefs.a
-rw-r--r--. 1 cstamas cstamas     1908 Jun 29 20:14 librmi.a
-rw-r--r--. 1 cstamas cstamas   131498 Jun 29 20:14 libsaproc.a
-rw-r--r--. 1 cstamas cstamas    44392 Jun 29 20:14 libsctp.a
-rw-r--r--. 1 cstamas cstamas   106168 Jun 29 20:14 libsimdsort.a
-rw-r--r--. 1 cstamas cstamas   726368 Jun 29 20:14 libsplashscreen.a
-rw-r--r--. 1 cstamas cstamas     1650 Jun 29 20:14 libsyslookup.a
-rw-r--r--. 1 cstamas cstamas    82566 Jun 29 20:14 libverify.a
-rw-r--r--. 1 cstamas cstamas    57848 Jun 29 20:14 libzip.a
[cstamas@angeleyes 22.0.2-graalce]$ 

And CI script refers to files libz.a, liblibchelper.a and those are not present.

Can you help me here?
Also, why is all this needed? Could we instead just use "older" OS like ubuntu-20 that uses glibc 2.31 that is before breaking change you mention in your commit 2.34?

@gzm55
Copy link
Contributor

gzm55 commented Oct 7, 2024

@gzm55 help please! The glibc patching test fails now, but this made me look in it more deeper and questions just grow...

Your related commit: da20df0

We use GraalVM 22 to build (and release) mvnd, and I checked on my box GraalVM 22.0.2-graalce installation (Fedora 40 WS x86_64) the contents of the directory referenced from CI script "$GRAALVM_HOME/lib/static/linux-amd64/glibc" and the referenced files are not existing as I see? My Linux box has these files:

[cstamas@angeleyes 22.0.2-graalce]$ ll lib/static/linux-amd64/glibc/
total 93220
-rw-r--r--. 1 cstamas cstamas     9340 Jun 29 20:14 libattach.a
-rw-r--r--. 1 cstamas cstamas  1312360 Jun 29 20:14 libawt.a
-rw-r--r--. 1 cstamas cstamas    83056 Jun 29 20:14 libawt_headless.a
-rw-r--r--. 1 cstamas cstamas  1227658 Jun 29 20:14 libawt_xawt.a
-rw-r--r--. 1 cstamas cstamas    44062 Jun 29 20:14 libdt_socket.a
-rw-r--r--. 1 cstamas cstamas    15934 Jun 29 20:14 libextnet.a
-rw-r--r--. 1 cstamas cstamas  2877468 Jun 29 20:14 libfontmanager.a
-rw-r--r--. 1 cstamas cstamas   119826 Jun 29 20:14 libinstrument.a
-rw-r--r--. 1 cstamas cstamas    97556 Jun 29 20:14 libj2gss.a
-rw-r--r--. 1 cstamas cstamas    20208 Jun 29 20:14 libj2pcsc.a
-rw-r--r--. 1 cstamas cstamas   194676 Jun 29 20:14 libj2pkcs11.a
-rw-r--r--. 1 cstamas cstamas     3918 Jun 29 20:14 libjaas.a
-rw-r--r--. 1 cstamas cstamas   342684 Jun 29 20:14 libjava.a
-rw-r--r--. 1 cstamas cstamas   391470 Jun 29 20:14 libjavajpeg.a
-rw-r--r--. 1 cstamas cstamas     2496 Jun 29 20:14 libjawt.a
-rw-r--r--. 1 cstamas cstamas   859762 Jun 29 20:14 libjdwp.a
-rw-r--r--. 1 cstamas cstamas    60462 Jun 29 20:14 libjimage.a
-rw-r--r--. 1 cstamas cstamas   148078 Jun 29 20:14 libjli.a
-rw-r--r--. 1 cstamas cstamas    18350 Jun 29 20:14 libjpackageapplauncher.a
-rw-r--r--. 1 cstamas cstamas   570102 Jun 29 20:14 libjpackageapplauncheraux.a
-rw-r--r--. 1 cstamas cstamas    10378 Jun 29 20:14 libjsig.a
-rw-r--r--. 1 cstamas cstamas   146176 Jun 29 20:14 libjsound.a
-rw-r--r--. 1 cstamas cstamas  1066052 Jun 29 20:14 libjsvml.a
-rw-r--r--. 1 cstamas cstamas 82384838 Jun 29 20:14 libjvm.a
-rw-r--r--. 1 cstamas cstamas   987482 Jun 29 20:14 liblcms.a
-rw-r--r--. 1 cstamas cstamas    13138 Jun 29 20:14 lible.a
-rw-r--r--. 1 cstamas cstamas    46946 Jun 29 20:14 libmanagement.a
-rw-r--r--. 1 cstamas cstamas     3032 Jun 29 20:14 libmanagement_agent.a
-rw-r--r--. 1 cstamas cstamas    56800 Jun 29 20:14 libmanagement_ext.a
-rw-r--r--. 1 cstamas cstamas   754928 Jun 29 20:14 libmlib_image.a
-rw-r--r--. 1 cstamas cstamas   121346 Jun 29 20:14 libnet.a
-rw-r--r--. 1 cstamas cstamas   227622 Jun 29 20:14 libnio.a
-rw-r--r--. 1 cstamas cstamas     4176 Jun 29 20:14 libprefs.a
-rw-r--r--. 1 cstamas cstamas     1908 Jun 29 20:14 librmi.a
-rw-r--r--. 1 cstamas cstamas   131498 Jun 29 20:14 libsaproc.a
-rw-r--r--. 1 cstamas cstamas    44392 Jun 29 20:14 libsctp.a
-rw-r--r--. 1 cstamas cstamas   106168 Jun 29 20:14 libsimdsort.a
-rw-r--r--. 1 cstamas cstamas   726368 Jun 29 20:14 libsplashscreen.a
-rw-r--r--. 1 cstamas cstamas     1650 Jun 29 20:14 libsyslookup.a
-rw-r--r--. 1 cstamas cstamas    82566 Jun 29 20:14 libverify.a
-rw-r--r--. 1 cstamas cstamas    57848 Jun 29 20:14 libzip.a
[cstamas@angeleyes 22.0.2-graalce]$ 

And CI script refers to files libz.a, liblibchelper.a and those are not present.

Can you help me here? Also, why is all this needed? Could we instead just use "older" OS like ubuntu-20 that uses glibc 2.31 that is before breaking change you mention in your commit 2.34?

Hi @cstamas , the following codes are used to patch the static libs using newer api, and the paths are fixed for ubuntu OS since the linux runner always runs on ubuntu-latest.

The libs requiring patching are:

  • all .a files in $GRAALVM_HOME/lib/static/linux-amd64/glibc/ folder
  • libz.a, on ubuntu it locates at /lib/x86_64-linux-gnu/ folder
  • some .a files in $GRAALVM_HOME/lib/svm/clibraries/linux-amd64/ folder, it is different with folder of the 1st entry

Could we instead just use "older" OS like [ubuntu-20]

It works if github still provides ubuntu-20. but locking the runner os may introduce package conflicts on building process when the ubuntu-20 is dropped by other required tools.

          : patch common libraries
          ( find "$GRAALVM_HOME/lib/static/linux-amd64/glibc" -name '*.a'
            ls -1 /lib/x86_64-linux-gnu/libz.a
            ls -1 "$GRAALVM_HOME/lib/svm/clibraries/linux-amd64/libjvm.a"
            ls -1 "$GRAALVM_HOME/lib/svm/clibraries/linux-amd64/liblibchelper.a"
          ) | while IFS= read -r input; do
                output="client/target/graalvm-libs-for-glibc-2.12/$(basename -- "$input")"
                objcopy --redefine-syms=client/src/main/resources/glibc/glibc.redef -- "$input" "$output" 2>/dev/null
              done

@gzm55
Copy link
Contributor

gzm55 commented Oct 7, 2024

@cstamas from the failure output, it seems that the patched libs are all not linked when generating the native mvnd, no new glibc symbols are introduced by the new JVM.

This pr tries to merge two lists from two profiles, but the combine.self="append" attributes should only be used between parent poms and childs poms, not two parallel profiles. So we still need property placeholders method to inject build args.

@cstamas
Copy link
Member Author

cstamas commented Oct 9, 2024

@gzm55 Thanks for helping! Appending the arguments list seems solved. But you did not answer my original question: does the CI patching script needs any update or it is okay as it is? (for both, master and mvnd-1.x) TIA

@cstamas cstamas merged commit 06eb5fd into apache:master Oct 9, 2024
5 checks passed
@cstamas cstamas deleted the bump-native-maven-plugin branch October 9, 2024 09:12
@cstamas
Copy link
Member Author

cstamas commented Oct 9, 2024

@gzm55 despite merged (just to make it move), can you pls review the CI script?

@gzm55
Copy link
Contributor

gzm55 commented Oct 14, 2024

@gzm55 despite merged (just to make it move), can you pls review the CI script?

@cstamas CI script for patching .a libs is still OK. i can list the .a files with fixed paths on a Ubuntu 24 box and graalvm-ce-java17-22.3.0:

ls -l /lib/x86_64-linux-gnu/libz.a
cd "$GRAALVM_HOME"
ls -l ./lib/svm/clibraries/linux-amd64/liblibchelper.a
ls -l ./lib/svm/clibraries/linux-amd64/libjvm.a

the folder of lib/static/linux-amd64/glibc/ also exists, all .a libs under this folder are patched.

@cstamas
Copy link
Member Author

cstamas commented Oct 15, 2024

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants