Add support for creating a native-image using -jar option

[jjbustamante]

Summary

Extend the current native-image Buildpack to allow building an image using the syntax

native-image [options] -jar jarfile [imagename] [options]

See doc

Use Cases

Right now, if we try to build a native image from a Quarkus project we faced several problems. Quarkus allows us to generate a folder with all the files require to build a native image when we use the variable: -Dquarkus.package.type=native-sources for example in Maven, the output of this build is a folder in my build directory with a structure similar to this:

target/native-sources
├── getting-started-1.0.0-SNAPSHOT-runner.jar
├── lib
└── native-image.args

Assuming the issue 76 is fixed an these artifacts are restore in the /workspace, this PR is trying to provide the functionality to build the image from those sources

This PR Fixes #109

Checklist

• I have viewed, signed, and submitted the Contributor License Agreement.
• I have linked issue(s) that this PR should close using keywords or the Github UI (See docs)
• I have added an integration test, if necessary.
• I have reviewed the styleguide for guidance on my code quality.
• I'm happy with the commit history on this PR (I have rebased/squashed as needed).

[dmikusa]

This seems totally reasonable.

I think the reason that we are building without the -jar option at the moment is for two reasons.

1. When you pack build -p some.jar and push a compiled artifact, pack is not giving the buildpacks the JAR. It gives them the contents of the JAR. Thus native-image never sees the JAR, and needs to build from the class files.
2. When you build from source, Maven/Gradle run and produce a single output EAR/JAR/WAR. This is then extracted to /workspace so that /workspace has the exact same contents as if you pushed a compiled artifact. This simplifies native-image in that you don't have to worry about what the user pushed, but again, you're left building from class files.

At any rate, when we open up libbs (Maven/Gradle/etc..) to allow multiple output artifacts, the behavior of building from class files is not going to be sufficient.

My only ask would be to wait until we've resolved that issue with Maven before merging this PR. It'll make testing/validation easier. Is that OK with you @jjbustamante?

[jjbustamante]

Hi @dmikusa-pivotal

Yes, I am 100% agree to wait for the fix of multiple artifacts, in fact, I will be adding some unit tests to this PR, but I wanted to share what I had before doing that work.

I am almost done to push another PR for the multiple artifacts enhancement.

[jjbustamante]

I created the PR in libbs to handle multiple artifacts after the build

[jjbustamante]

Thanks @dmikusa-pivotal.

Let me fix the unit tests!

[dmikusa]

Thanks @jjbustamante The tests are passing. I'll give this a review shortly using the https://github.com/quarkusio/quarkus-quickstarts. Unless you have any other suggestions for demo apps to use.

[jjbustamante]

Hi @dmikusa-pivotal.

Using the getting started guide is fine. Let me put here the command I was using to run it locally.

I cloned the Quarkus examples application, then I went into quarkus-quickstarts/getting-started and from there I ran the following command:

pack build quarkus-getting-started-native \
  -e BP_NATIVE_IMAGE=true 
  -e BP_MAVEN_BUILD_ARGUMENTS="-Dquarkus.package.type=native-sources -Dmaven.test.skip=true package" \
  -e BP_MAVEN_BUILT_ARTIFACT="target/native-sources" \
  -e BP_NATIVE_IMAGE_BUILD_ARGUMENTS="-J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=CO -J-Dfile.encoding=UTF-8 -H:+JNI -H:+AllowFoldMethods -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-UseServiceLoaderFeature -H:+StackTrace"  \ 
  -e BP_NATIVE_IMAGE_BUILD_JAR=native-sources/getting-started-1.0.0-SNAPSHOT-runner.jar \
  -p . \ 
  -b /Users/jbustamante/go/src/github.com/jjbustamante/ca-certificates \ 
  -b paketo-buildpacks/bellsoft-liberica -b paketo-buildpacks/syft \
  -b /Users/jbustamante/go/src/github.com/jjbustamante/maven \
  -b paketo-buildpacks/executable-jar  \ 
  -b /Users/jbustamante/go/src/github.com/jjbustamante/native-image \ 
  -b paketo-buildpacks/spring-boot \ 
  --clear-cache

From the previous command, the only thing is that value of the env variable BP_NATIVE_IMAGE_BUILD_ARGUMENTS, I wanted to take the native-image.args generated by Quarkus but that file include the value of jar to be used to create the native-image, I wanted to use something like BP_NATIVE_IMAGE_BUILD_ARGUMENTS=$(cat native-image.args) and probably remove the -jar... part but I didn't spend much time working on how to do that with an awk or sed command, so I basically copy the value of the file and manually remove the -jar ... part

[dmikusa]

@jjbustamante - FYI. I've been playing with this today. I have been using your PR as the nucleus for a new PR because there are some things I needed to change for consistency and to make testing easier (I'm also adding in something so you don't have to manually set all those args). I'm just running short of time today, but I can hopefully finish this tomorrow and share the PR I'm proposing. Just wanted to give you a heads up.

[dmikusa]

@jjbustamante - Here's the PR -> #136.

We released the enhancement for the Maven buildpack so it now supports multiple artifacts. A quick way to test this is to download the branch for the PR and run:

pack build quarkus-app -e BP_NATIVE_IMAGE=true -e BP_MAVEN_BUILD_ARGUMENTS="-Dquarkus.package.type=native-sources -Dmaven.test.skip=true package" -e BP_MAVEN_BUILT_ARTIFACT="target/native-sources" -e BP_NATIVE_IMAGE_BUILD_ARGUMENTS_FILE="native-sources/native-image.args" -e BP_NATIVE_IMAGE_BUILT_ARTIFACT="native-sources/websockets-quickstart-1.0.0-SNAPSHOT-runner.jar" -b paketo-buildpacks/[email protected] -b paketo-buildpacks/syft -b paketo-buildpacks/maven -b paketo-buildpacks/executable-jar -b <path-to-native-image-bp>

I tested with a couple of Quarkus apps and they all built OK. Let me know your experiences & if you think there's anything else we need before we merge this. Thanks!

[jjbustamante]

Hi @dmikusa-pivotal

I tested the branch with different projects from the Quarkus examples and they are working great. I found one thing trying to test it with a personal Quarkus project but I managed to work around the problem following guidelines from here.

Just to let you know my issue.

my application native-source folder looks like this:

target/native-sources/
├── aidpass-backend-1.2.0-SNAPSHOT-runner.jar
├── lib
├── native-image.args
├── proxy-config.json
└── proxy-resources.json

The native-image.args file includes those JSON files using a Quarkus value in the application.properties file\

quarkus.native.additional-build-args=--allow-incomplete-classpath,--report-unsupported-elements-at-runtime,-H:DynamicProxyConfigurationFiles=proxy-config.json,-H:DynamicProxyConfigurationResources=proxy-resources.json

The problem was that those files were not available in the classpath during the build and I had errors like this:

com.oracle.svm.core.util.UserError$UserException: Could not find dynamic proxy configuration resource "/workspace/native-sources/proxy-resources.json".
	at com.oracle.svm.core.util.UserError.abort(UserError.java:73)

I tried different ways to do it, and the only one that works was adding the classpath parameter -cp but for some reason the native-image process never ends. Finally, I managed to fix the problem and build the image, as I mentioned, following some guidelines in the graalvm configuration.

I think the feature is great! just le me know when you merged this PR and I will test it directly with all the buildpacks!! thanks a lot

[dmikusa]

@jjbustamante Classpath is an interesting issue here. In your PR, you had been setting the CLASSPATH but in my observation, the JAR file that gets produced by Quarkus has the CLASSPATH set within the MANIFEST.MF file, so it doesn't seem necessary to set this manually. In the PR I submitted, when building from a JAR we don't set anything for the CLASSPATH. That could be why it can't find the files.

I suspect that if you wanted to set the classpath, you'd need to completely override the CLASSPATH. I don't know if this would cause other issues though, since you'd be adding the native-sources/ directory to the classpath which in turn has the JAR we're trying to process.

How do those json files end up in that location? Is that a standard convention with a Quarkus app? I could look at handling that more gracefully or at the least documenting that, if it's a common case.

Thanks for testing!

[jjbustamante]

@dmikusa-pivotal

I think it's actually related to Graalvm and native-image creation, in my case, those files end up in the native-sources folder during the build, they were at src/main/resources.

In order to solve my particular case, what I did was to follow the recommendations from here to include the required configuration inside the jar that is going to be used to create the image. The steps I did were these:

• I created a folder at `src/main/resources/META-INF/native-image/[my groupd id]/[my artifact id]
• I copy my JSON files from src/main/resources to the folder created before
• I created a file named native-image.properties
• I add the arguments from native-image. args into that properties file created before, using -H:ResourceConfigurationResources={.}/proxy-config.json

When I built using pack I didn't use the BP_NATIVE_IMAGE_BUILD_ARGUMENTS_FILE env because that configuration was now embedded into the jar file created during the maven package. the native-image executable knows how to deal with that configuration and is able to build the image correctly.

I am not 100% sure if it is something we need to take care in Buildpacks, the use case for building a Quarkus app was out of the box with what we did, and these particular cases can also be handled by checking the documentation from Graalvm, but adding some FAQ or similar documentation in some place could be very useful for end-users.

[dmikusa]

OK, I'll make a note to add this to our docs. Thanks for sharing the details on how you got that use case working.

It seems like we're all set though, so I'm going to close out this PR & merge the one I submitted. This work should be out on Friday when we cut the weekly releases.