Unable to compile qute codestart to native on windows

[jedla97]

Describe the bug

When I try compile resteasy-qute and rest-qute (resteasy-reactive-qute) codestart to native on windows the compilation fail with this error:

[1/8] Initializing...
(11,5s @ 0,14GB)
Java version: 21.0.2+13-LTS, vendor version: Oracle GraalVM 21.0.2+13.1
Graal compiler: optimization level: 2, target machine: x86-64-v3, PGO: ML-inferred
C compiler: cl.exe (microsoft, x64, 19.38.33130)
Garbage collector: Serial GC (max heap size: 80% of RAM)
4 user-specific feature(s):
- com.oracle.svm.thirdparty.gson.GsonFeature
- io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
- org.eclipse.angus.activation.nativeimage.AngusActivationFeature
 ------------------------------------------------------------------------------------------------------------------------
3 experimental option(s) unlocked:
- '-H:+AllowFoldMethods' (origin(s): command line)
- '-H:BuildOutputJSONFile' (origin(s): command line)
- '-H:-UseServiceLoaderFeature' (origin(s): command line)
------------------------------------------------------------------------------------------------------------------------
Build resources:
- 12,09GB of memory (75,6% of 16,00GB system memory, determined at start)
- 2 thread(s) (100,0% of 2 available processor(s), determined at start)
[2/8] Performing analysis...  [*]  (76,8s @ 1,08GB)
 11 652 reachable types   (86,9% of   13 416 total)
 16 854 reachable fields  (58,6% of   28 750 total)
 63 411 reachable methods (58,6% of  108 179 total)
 3 693 types,   155 fields, and 3 238 methods registered for reflection
 4 native libraries: crypt32, ncrypt, psapi, winhttp
------------------------------------------------------------------------------------------------------------------------
11,3s (12,6% of total time) in 111 GCs | Peak RSS: 1,88GB | CPU load: 1,94
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
C:\Users\jedla\Downloads\code-with-quarkus\target\code-with-quarkus-1.0.0-SNAPSHOT-native-image-source-jar\svm_err_b_20240412T200838.238_pid2728.md (build_info)
========================================================================================================================
Failed generating 'code-with-quarkus-1.0.0-SNAPSHOT-runner' after 1m 28s.
The build process encountered an unexpected error:
com.oracle.svm.core.util.VMError$HostedError: InternalFeature defined by com.oracle.svm.hosted.ResourcesFeature unexpectedly failed with a(n) java.util.regex.PatternSyntaxException
at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.VMError.shouldNotReachHere(VMError.java:86)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.handleFeatureError(FeatureHandler.java:287)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:92)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$11(NativeImageGenerator.java:786)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.AbstractAnalysisEngine.runAnalysis(AbstractAnalysisEngine.java:181)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:783)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:592)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)

Caused by: java.util.regex.PatternSyntaxException: Unknown character property name {a} near index 11
templates\page.qute.html
           ^

at java.base/java.util.regex.Pattern.error(Pattern.java:2204)
at java.base/java.util.regex.Pattern.family(Pattern.java:3089)
at java.base/java.util.regex.Pattern.sequence(Pattern.java:2332)
at java.base/java.util.regex.Pattern.expr(Pattern.java:2245)
at java.base/java.util.regex.Pattern.compile(Pattern.java:1945)
at java.base/java.util.regex.Pattern.<init>(Pattern.java:1576)
at java.base/java.util.regex.Pattern.compile(Pattern.java:1101)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ResourcesFeature.makeResourcePattern(ResourcesFeature.java:364)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3573)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ResourcesFeature.compilePatterns(ResourcesFeature.java:357)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ResourcesFeature.duringAnalysis(ResourcesFeature.java:335)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$10(NativeImageGenerator.java:786)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:90)
... 9 more

Expected behavior

Finish build without any error.

Actual behavior

Caused by: java.util.regex.PatternSyntaxException: Unknown character property name {a} near index 11
templates\page.qute.html

How to Reproduce?

1. Download https://code.quarkus.io/d?e=resteasy-qute&cn=code.quarkus.io
2. Compile to native by mvn.cmd install -Dnative
3. During the stage 2 error appear

Output of uname -a or ver

Windows

Output of java -version

No response

Mandrel or GraalVM version (if different from Java)

GraalVM 23.1 JDK 21.0.2+13-LTS-jvmci-23.1-b30

Quarkus version or git rev

3.8.3+, 3.9.0.CR2+

Build tool (ie. output of mvnw --version or gradlew --version)

Apache maven 3.8.8

Additional information

Latest version where this is working are 3.8.2 and 3.9.0.CR1 (just tested if all 3.9 is affected)
Tried it on Windows 10 but this happening also on github runners. Tested it on Linux and build fine.

[quarkus-bot]

/cc @Karm (mandrel), @galderz (mandrel), @ia3andy (codestart), @mkouba (qute), @quarkusio/devtools (codestart), @zakkak (mandrel,native-image)

[ebullient]

This is a problem with your config. Try either two backslashes (to escape the backslash), or a forward slash? Path separators on windows are confusing. The OS uses backslash, but not much else does.

[jedla97]

TBH as this is what was generated by code.quarkus.io I would expect it work out of the box without any modification.

+ just tested it again and 3.8.2 work and 3.8. not working. So I think it's not wrong config. Looking at commits between these version it's possible that this is caused by #39434 but not sure

[mkouba]

TBH as this is what was generated by code.quarkus.io I would expect it work out of the box without any modification.

Yes, I agree that it should work.

• just tested it again and 3.8.2 work and 3.8. not working. So I think it's not wrong config. Looking at commits between these version it's possible that this is caused by Qute: fix the NoRestartTemplatesDevModeTest on Windows #39434 but not sure

The problem with NativeImageResourceBuildItem registration on Windows should be fixed in #40067. So you can give it a try.

However, the PatternSyntaxException comes from the graalvm package. So I'm not really sure...

[galderz]

However, the PatternSyntaxException comes from the graalvm package. So I'm not really sure...

What is it you are unsure about exactly?

[mkouba]

However, the PatternSyntaxException comes from the graalvm package. So I'm not really sure...

What is it you are unsure about exactly?

I'm not sure this error is caused by the NativeImageResourceBuildItem. I would expect a more actionable error message? Something like "You've registered a resource templates\page.qute.html but the value has to be a valid regex pattern. Make sure all reserved chars are escaped.".

[gsmet]

I would bet the problem is that you have an unescaped \.

Either you need to escape it (together with any control character if you expect your path to not contain any so maybe a Pattern.quote call is in order) or you use / (I don't know what GraalVM is supposed to consume).

[aloubyansky]

I should have another go at re-writing it using PathTree API.
https://github.com/quarkusio/quarkus/blob/main/extensions/qute/deployment/src/main/java/io/quarkus/qute/deployment/QuteProcessor.java#L2172-L2180 looks like redundant, given the code block above it.

[jedla97]

@mkouba sorry to take this long to try your PR. I tried build native with #40067 and main (2 or 3 hours ago) and still see the same error.

[aloubyansky]

https://github.com/quarkusio/quarkus/blob/main/extensions/qute/deployment/src/main/java/io/quarkus/qute/deployment/QuteProcessor.java#L2172-L2180 looks like redundant, given the code block above it.

That's not redundant, sorry.

@mkouba can we trust our current testsuite if I suggest a refactoring of this scanning impl? I already made an attempt yesterday and the CI is green.

[aloubyansky]

I mean with the Windows paths and case-sensitive paths. I think previously it caught these issues. Just to be sure.

[gsmet]

I had a closer look and I think that NativeImageResourceBuildItem is quite problematic in its current form as things are passed as regexp patterns, after a glob transformation, whereas in your case you want to pass the exact file path, thus why you need escaping.

I think we should have had a specific method for the glob -> pattern thing and that by default it should have considered the path as is.

It's too late for that as this build item is massively used.

Now two things:

• I'm surprised you have to pass the OS specific path to GraalVM. I would have expected / to work.
• Maybe we should have the ability in NativeImageResourceBuildItem to pass the exact path and have it avoid the Glob thing + use Pattern.quote to make sure the path is tested as is and not considered a regexp.

[mkouba]

https://github.com/quarkusio/quarkus/blob/main/extensions/qute/deployment/src/main/java/io/quarkus/qute/deployment/QuteProcessor.java#L2172-L2180 looks like redundant, given the code block above it.

That's not redundant, sorry.

@mkouba can we trust our current testsuite if I suggest a refactoring of this scanning impl? I already made an attempt yesterday and the CI is green.

Our test suite does not have a lot of tests covering the native image 🤷. Also there's only one test that asserts the produced NativeImageResourceBuildItems - io.quarkus.qute.deployment.templateroot.AdditionalTemplateRootTest. The JVM mode should be covered well.

What is this refactoring about? I mean, does it solve a particular problem, is it worth the risk?

[mkouba]

I had a closer look and I think that NativeImageResourceBuildItem is quite problematic in its current form as things are passed as regexp patterns, after a glob transformation, whereas in your case you want to pass the exact file path, thus why you need escaping.

Hm, the javadoc of NativeImageResourceBuildItem does not even describe the required/supported format of the accepted resource string... This is very bad and should be fixed IMO.

[gsmet]

Well, I think it's worse than that because it somehow evolved along the way because GraalVM changed their inputs. Thus why we have something to translate globs to regexps as I think it used to be supporting globs...

For pattern and globs, NativeImageResourcePatternsBuildItem contract is a lot better.

Problem is I don't think we can change NativeImageResourceBuildItem now.

I wonder if we should mark it as deprecated and have:

• NativeImageResourcePatternsBuildItem <- existing
• NativeImageResourceDirectoryBuildItem <- existing but we need to fix it to use the one below when adding the files
• NativeImageResourceFileBuildItem <- following the same naming as directory. Plain files, no pattern or glob support.

[aloubyansky]

https://github.com/quarkusio/quarkus/blob/main/extensions/qute/deployment/src/main/java/io/quarkus/qute/deployment/QuteProcessor.java#L2172-L2180 looks like redundant, given the code block above it.

That's not redundant, sorry.
@mkouba can we trust our current testsuite if I suggest a refactoring of this scanning impl? I already made an attempt yesterday and the CI is green.

Our test suite does not have a lot of tests covering the native image 🤷. Also there's only one test that asserts the produced NativeImageResourceBuildItems - io.quarkus.qute.deployment.templateroot.AdditionalTemplateRootTest. The JVM mode should be covered well.

What is this refactoring about? I mean, does it solve a particular problem, is it worth the risk?

Sorry for hijacking the native issue discussion.

1. walking directories isn't always a correct way of looking for content. The whole purpose of .getContentTree() methods is to provide a view of the filesystem content that makes up an artifact content (it could be seen as an edge case but there could be filtering applied to directories and that's why this API was introduced);
2. PathTree provides the same API to process archive and directory content, which often simplifies code that needs to do both.

Here is the change I tried main...aloubyansky:quarkus:qute-template-scan

[galderz]

I think this might be relevant here, but I've been doing some fiddling with the resource patterns as part of my work in #36826. In particular, I had to this to the JSON builder:

        /**
         * Skips escaping characters in string values.
         * This option should be enabled when transforming JSON input,
         * whose string values are already escaped.
         * In situations like this, the option avoids escaping characters
         * that are already escaped.
         */
        protected final boolean skipEscapeCharacters;

I did this because the native image agent can generate resource patterns like this:

    "pattern":"\\QMETA-INF/services/java.nio.channels.spi.SelectorProvider\\E"

And I don't want those being escaped when filtering resources to produce a new resource-config.json. Why? Because the \\Q..\\E signals a pattern that is escaped and no additional escape is required in between. Not sure it helps here but just putting it out in case it does :)

Also, seems like this resource format might be changing sometime soon, see oracle/graal#7487

[mkouba]

https://github.com/quarkusio/quarkus/blob/main/extensions/qute/deployment/src/main/java/io/quarkus/qute/deployment/QuteProcessor.java#L2172-L2180 looks like redundant, given the code block above it.

That's not redundant, sorry.
@mkouba can we trust our current testsuite if I suggest a refactoring of this scanning impl? I already made an attempt yesterday and the CI is green.

Our test suite does not have a lot of tests covering the native image 🤷. Also there's only one test that asserts the produced NativeImageResourceBuildItems - io.quarkus.qute.deployment.templateroot.AdditionalTemplateRootTest. The JVM mode should be covered well.
What is this refactoring about? I mean, does it solve a particular problem, is it worth the risk?

Sorry for hijacking the native issue discussion.

1. walking directories isn't always a correct way of looking for content. The whole purpose of  `.getContentTree()` methods is to provide a view of the filesystem content that makes up an artifact content (it could be seen as an edge case but there could be filtering applied to directories and that's why this API was introduced);

2. `PathTree` provides the same API to process archive and directory content, which often simplifies code that needs to do both.

Here is the change I tried main...aloubyansky:quarkus:qute-template-scan

@aloubyansky Ok, pls send a PR.