Integrate support for Panama Vector and FFI

[ChrisHegarty]

This issue proposed to add support to Elasticsearch for the previewing/incubating Panama Vector and FFI.

The most fruitful approach is to follow that we've already done in Lucene - it is tried, tested, and proven to work well. That is:

• Update the build to generate the JDK-specific apijars - this contains the JDK packages with just the stubs of the classes and methods. It also strips the Incubating annotation - a.k.a "the api jar"
• Add grade java plugin support for patching java.base with the apijar
• Add support to building MR JARs, so that we can have version specific source built into the appropriate version section of the jar artifact
• Add reflective support for loading different implementations on different JDK versions
• Add Gradle "magic" so that the IDE can recognise the use the version specific source, e.g. src/java21.
  It's worth nothing that in Lucene my IDE setup for version specific source code, e.g. src/java21, is not great - I get red squiggly lines under the code that is newer than JDK 17. But it's not too bad - there are workarounds by setting local JDKs. But maybe we can do something better?

blocks #98749

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mark-vieira]

• Update the build to generate the JDK-specific apijars - this contains the JDK packages with just the stubs of the classes and methods. It also strips the Incubating annotation - a.k.a "the api jar"

Is there a reason to do this rather than just use the real JDK? Our build can already auto-provision JDKs so this isn't a big deal.

[ChrisHegarty]

• Update the build to generate the JDK-specific apijars - this contains the JDK packages with just the stubs of the classes and methods. It also strips the Incubating annotation - a.k.a "the api jar"

Is there a reason to do this rather than just use the real JDK? Our build can already auto-provision JDKs so this isn't a big deal.

Yes, but of course we have options.

There are two issues that the api jar solves beyond the compile JDK:

1. For JDK 21, FFI is still a preview feature. The means that the compile has to enable preview, which taints the resulting class files - they have "the preview bit" set so can only be loaded by a runtime if --enable-preview is set. We do not want to allow preview features to be generally enabled at run time, and the java launcher does not support fine grain enablement.

2. Panama Vector is incubating. Compiling against incubating APIs emits non-suppressible warnings - cannot be used with -Werror :-(

3. The api jar ensures that only limited apis are used from the newer JDK, i.e. no new language features or other APIs that may not be present in the older JDK.

For no.1 we could just support jdk22+, but that delays work waiting for this. Maybe we can look at this in the future. Alternatively, we could compile with --enable-preview, and strip "the preview bit". If we can do this at a fine grain level - per class or per Gradle sub project. Lucene does this stripping too.

For no.2 we could just disable -Werror for limited compiles or something. If we could do this at a fine-grain level - per class. Or maybe compile with -Werror, capture the process return code and output, parse it for expected warning, and fail is anything else.

I chose to "just do the same as Lucene" so as to avoid creating something new, but the above does not seem too onerous.

No.3 is a little more sensitive, and is probably the strongest reason to use the api jar.

[mark-vieira]

Understood, that makes sense.

[rjernst]

@ChrisHegarty I think this issue has been addressed by the mrjar plugin, would you agree?

[ChrisHegarty]

We have have FFI - yay!! But we don't have the incubating Vector API. Maybe we can live without that for now.