Native libraries are searched in lib/polyglot project dir #11874

Akirathan · 2024-12-16T11:23:05Z

Pull Request Description

Native libraries used via JNI from polyglot Java code can be located in polyglot/lib directories in projects. New docs is in

enso/docs/polyglot/java.md

Lines 68 to 93 in a9c60a5

    
           ## Native libraries 
        
           Java can load native libraries using, e.g., the 
        
           [System.loadLibrary](<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/System.html#loadLibrary(java.lang.String)>) 
        
           or 
        
           [ClassLoader.findLibrary](<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/ClassLoader.html#findLibrary(java.lang.String)>) 
        
           methods. If a Java method loaded from the `polyglot/java` directory in project 
        
           `Proj` tries to load a native library via one of the aforementioned mechanisms, 
        
           the runtime system will look for the native library in the `polyglot/lib` 
        
           directory within the project `Proj`. The runtime system implements this by 
        
           overriding the 
        
           [ClassLoader.findLibrary](<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/ClassLoader.html#findLibrary(java.lang.String)>) 
        
           method on the `ClassLoader` used to load the Java class. 
        
           The algorithm used to search for the native libraries within the `polyglot/lib` 
        
           directory hierarchy conforms to the 
        
           [NetBeans JNI specification](https://bits.netbeans.org/23/javadoc/org-openide-modules/org/openide/modules/doc-files/api.html#jni): 
        
           Lookup of library with name `native` works roughly in these steps: 
        
           - Add platform-specific prefix and/or suffix to the library name, e.g., 
        
             `libnative.so` on Linux. 
        
           - Search for the library in the `polyglot/lib` directory. 
        
           - Search for the library in the `polyglot/lib/<arch>` directory, where `<arch>` 
        
             is the name of the architecture. 
        
           - Search for the library in the `polyglot/lib/<arch>/<os>` directory, where 
        
             `<os>` is the name of the operating system.

Important Notes

Standard.Image depends on opencv.jar which contains all native libraries for all platforms:

> tree opencv-4.7.0-0/nu/pattern/opencv
opencv-4.7.0-0/nu/pattern/opencv/
├── linux
│   ├── ARMv7
│   │   ├── libopencv_java470.so
│   │   └── README.md
│   ├── ARMv8
│   │   ├── libopencv_java470.so
│   │   └── README.md
│   ├── x86_32
│   │   └── README.md
│   └── x86_64
│       ├── libopencv_java470.so
│       └── README.md
├── osx
│   ├── ARMv8
│   │   ├── libopencv_java470.dylib
│   │   └── README.md
│   └── x86_64
│       ├── libopencv_java470.dylib
│       └── README.md
└── windows
    ├── x86_32
    │   ├── opencv_java470.dll
    │   └── README.md
    └── x86_64
        ├── opencv_java470.dll
        └── README.md

12 directories, 15 files

All native libraries have 352 MB, but we only need a single one. For example:

> du -sh opencv-4.7.0-0/nu/pattern/opencv
352M	opencv-4.7.0-0/nu/pattern/opencv
> du -sh opencv-4.7.0-0/nu/pattern/opencv/linux/x86_64/
62M	opencv-4.7.0-0/nu/pattern/opencv/linux/x86_64/

In this PR, sbt extracts all the native libraries from opencv.jar and puts them into Standard/Image/.../polyglot/lib directory. In subsequent PRs, we may want to drop all the native libraries for different platforms.

Native image building modification

EnsoLibraryFeature called during native image build scans all the native libraries inside std libraries, and explicitly registers them for usage in java.library.path during runtime:

enso/engine/runner/src/main/java/org/enso/runner/EnsoLibraryFeature.java

Lines 92 to 105 in 5a44328

    
             if (pkg.nativeLibraryDir().exists()) { 
        
               var nativeLibs = 
        
                   NativeLibraryFinder.listAllNativeLibraries(pkg, FileSystem$.MODULE$.defaultFs()); 
        
               for (var nativeLib : nativeLibs) { 
        
                 assert nativeLib.exists() && nativeLib.isFile(); 
        
                 var dir = nativeLib.toPath().getParent().toFile(); 
        
                 assert dir.exists() && dir.isDirectory(); 
        
                 nativeLibPaths.add(dir.getAbsolutePath()); 
        
                 var current = System.getProperty("java.library.path"); 
        
                 RuntimeSystemProperties.register( 
        
                     "java.library.path", current + File.pathSeparator + dir.getAbsolutePath()); 
        
               } 
        
             } 
        
           }

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

MacOS engine tests succeeds on Image_Tests on native image
The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.
If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
or the Snowflake database integration, a run of the Extra Tests has been scheduled.
- If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

This reverts commit 84be175.

This reverts commit c98bdaf.

Akirathan · 2024-12-17T17:03:31Z

It was a mistake to try to create a different HostClassLoader for each project. The underlying exception of https://github.com/enso-org/enso/actions/runs/12372797379/job/34531804458?pr=11874#step:7:5854 is NoClassDefFoundError: org/enso/base/polyglot/Polyglot_Utils throw from

enso/std-bits/table/src/main/java/org/enso/table/data/table/Column.java

Line 119 in 1e0649f

Object converted = Polyglot_Utils.convertPolyglotValue(v);

. Column class was loaded by a HostClassLoader for Standard.Table pkg, whereas Polyglot_Utils was loaded by a HostClassLoader for Base pkg. Let's just stick with a single HostClassLoader that will override findLibrary.

Reverting in 1649ddf and 05851f5

GitHub
Native libraries are searched in lib/polyglot project dir · enso-org/enso@84be175
Enso Analytics is a self-service data prep and analysis platform designed for data teams. - Native libraries are searched in lib/polyglot project dir · 84be175

…from a dll

So that no extraction is done unnecessarily.

JaroslavTulach

Looks very nice. Looking forward to see the CI green.

docs/polyglot/java.md

engine/runtime/src/main/java/org/enso/interpreter/runtime/util/TruffleFileSystem.java

lib/scala/pkg/src/main/java/org/enso/pkg/NativeLibraryFinder.java

JaroslavTulach · 2024-12-21T05:50:29Z

lib/scala/pkg/src/main/scala/org/enso/filesystem/FileSystem.scala

+    * @param file
+    * @return
+    */
+  def getAbsolutePath(file: F): String


As usually I have to note: It would be nice to rewrite FileSystem into Java. Not necessarily in this PR.

JaroslavTulach · 2024-12-21T15:23:15Z

What is the size of the bin/enso native executable on your branch? Does it go down to ~200MB? E.g. has the effect of

Including opencv native libraries as resources in enso runner binary #11807

reverted back?

The current printing is not visible on CI

Akirathan · 2024-12-23T17:32:37Z

Native image build of engine-runner correctly registers native libraries even for Windows - https://github.com/enso-org/enso/actions/runs/12470087920/job/34804596736?pr=11874#step:7:1963. The Image_Tests for Windows on native image succeeds

GitHub
Native libraries are searched in lib/polyglot project dir · enso-org/enso@a9c60a5
Enso Analytics is a self-service data prep and analysis platform designed for data teams. - Native libraries are searched in lib/polyglot project dir · a9c60a5

Akirathan · 2024-12-23T17:47:01Z

What is the size of the bin/enso native executable on your branch? Does it go down to ~200MB? E.g. has the effect of
* [Including opencv native libraries as resources in enso runner binary #11807](https://github.com/enso-org/enso/pull/11807)
reverted back?

Size of enso binary on develop: 266 MB, size on this PR: 204 MB. So yes, the libopencv.so is not included in the binary.

JaroslavTulach

Ideally we should not be doing anything special in EnsoLibraryFeature.

JaroslavTulach · 2024-12-24T05:52:00Z

engine/runner/src/main/java/org/enso/runner/EnsoLibraryFeature.java

+              assert dir.exists() && dir.isDirectory();
+              nativeLibPaths.add(dir.getAbsolutePath());
+              var current = System.getProperty("java.library.path");
+              RuntimeSystemProperties.register(


I don't think this is necessary neither good idea.

Please remember that NI build runs in build time, while the .so, .dll files are about to be located in runtime. The paths are going to be different.

Can you tell me: What was failing and why you think this change is helpful? We don't need anything like this to load .so for enso_parser...

For enso_parser, we have full control over explicit search for the library in Parser$Worker$initializeLibraries - in that method, we try to explicitly load the enso_parser native library first with System.loadLibrary, and then with System.load.

For opencv, we don't have full control. Its native library opencv_java470 is loaded in the static initializer of nu.pattern.OpenCV with System.loadLibrary("opencv_java470"). OpenCV class is used by classes in our std-bits/image. In JVM mode, all classes from std-bits/image (classes in package org.enso.image) are loaded by HostClassLoader, therefore, also OpenCV class is loaded by HostClassLoader, and so, System.loadLibrary("opencv_java470") delegates to HostClassLoader.findLibrary. In native image, there is no HostClassLoader, and we have to deal with the System.loadLibrary call inside OpenCV differently.

The solution to include path to the native library inside RuntimeSystemProperties from NI build time was the only solution I could think of. It points to a path like distribution/lib/Standard/Image/0.0.0-dev/polyglot/lib/... where the native libraries are located. All the tests are currently passing, because we test NI only via cmdline, and we don't try to package the NI inside the AppImage. It is definitely not the most robust solution. I am opened to any alternatives.

TL;DR; Is there any other way how to hook into System.loadLibrary calls during runtime in NI other than setting java.library.path via RuntimeSystemProperties during NI build?

In native image, there is no HostClassLoader,

Really? How comes there is no HostClassLoader? Of course it cannot load any new classes, but the classloader can/should still be there...

This is an example of the use of HostClassLoader. Still some problems there, but this way we should be able to get HostClassLoader "into the game".

engine/runner/src/main/java/org/enso/runner/EnsoLibraryFeature.java

lib/scala/pkg/src/main/java/org/enso/pkg/NativeLibraryFinder.java

docs/polyglot/java.md

JaroslavTulach

The build time vs. runtime classpath issue has to be solved.

JaroslavTulach · 2024-12-27T05:16:47Z

docs/polyglot/java.md

+
+- Names for `<os>` are `linux`, `macos`, `windows`.
+  - Note that for simplicity we omit the versions of the operating systems.
+- Names for architectures `<arch>` are `amd64`, `x86_64`, `x86_32`.


When I buildEngineDistribution with 05197f5 I see:

Standard/Image/0.0.0-dev/polyglot/lib/amd64/windows/opencv_java470.dll Standard/Image/0.0.0-dev/polyglot/lib/amd64/linux/libopencv_java470.so Standard/Image/0.0.0-dev/polyglot/lib/amd64/macos/libopencv_java470.dylib Standard/Image/0.0.0-dev/polyglot/lib/ARMv8/macos/libopencv_java470.dylib Standard/Image/0.0.0-dev/polyglot/lib/x86_32/windows/opencv_java470.dll

I don't think the directory names are as they should be! When check my Mac OS X M2:

jshell> System.getProperty("os.name") $1 ==> "Mac OS X" jshell> System.getProperty("os.arch") $2 ==> "aarch64"

amd64 is nice (although x86_64 would be more standard), but I would prefer aarch64 as the architecture for ARMv8.

I would flatten the directories to "necessary minimum". Instead of:

Standard/Image/0.0.0-dev/polyglot/lib/amd64/windows/opencv_java470.dll
Standard/Image/0.0.0-dev/polyglot/lib/amd64/linux/libopencv_java470.so
Standard/Image/0.0.0-dev/polyglot/lib/amd64/macos/libopencv_java470.dylib
Standard/Image/0.0.0-dev/polyglot/lib/ARMv8/macos/libopencv_java470.dylib
Standard/Image/0.0.0-dev/polyglot/lib/x86_32/windows/opencv_java470.dll

I would just:

Standard/Image/0.0.0-dev/polyglot/lib/opencv_java470.dll Standard/Image/0.0.0-dev/polyglot/lib/libopencv_java470.so Standard/Image/0.0.0-dev/polyglot/lib/amd64/libopencv_java470.dylib Standard/Image/0.0.0-dev/polyglot/lib/aarch64/libopencv_java470.dylib

which matches what I see when downloading Enso IGV Plugin from latest develop build.

JaroslavTulach · 2024-12-27T07:04:06Z

engine/runner/src/main/java/org/enso/runner/EnsoLibraryFeature.java

+              assert dir.exists() && dir.isDirectory();
+              nativeLibPaths.add(dir.getAbsolutePath());
+              var current = System.getProperty("java.library.path");
+              RuntimeSystemProperties.register(


This is an example of the use of HostClassLoader. Still some problems there, but this way we should be able to get HostClassLoader "into the game".

Update java.md docs

305db07

Akirathan self-assigned this Dec 16, 2024

Akirathan added 2 commits December 16, 2024 12:44

Add docs

a67afda

Separate instance of HostClassLoader per project

c98bdaf

Akirathan added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Dec 16, 2024

Akirathan changed the title ~~Native libraries are search in lib/polyglot project dir~~ Native libraries are searched in lib/polyglot project dir Dec 17, 2024

Akirathan added 3 commits December 17, 2024 13:14

Insert default HostClassLoader for builtin lib

84be175

Revert "Insert default HostClassLoader for builtin lib"

1649ddf

This reverts commit 84be175.

Revert "Separate instance of HostClassLoader per project"

05851f5

This reverts commit c98bdaf.

Add NativeLibraryFinder

c17b62d

enso-bot bot mentioned this pull request Dec 17, 2024

Wasting 100MB by Standard.Image: Implement polyglot/lib #11483

Open

Akirathan added 18 commits December 18, 2024 10:56

OpenCV native lib is not copied to a temporary directory, but loaded …

83c43e7

…from a dll

std-image/compile does not copy opencv.jar

c51a2cc

Add JARUtils to sbt project

1d7c0bb

Add *.dylib to gitignore

98c8a61

Add error recovery to JARUtils

5f0ce0a

std-image/compile extract native libraries from OpenCV.jar

e2d7099

fmt

561bcdc

JAR extraction uses sbt cache.

f0c930e

So that no extraction is done unnecessarily.

Move extractNativeLibs task to StdBits

5f158b8

NativeLibraryFinder strips version from os.arch

c6d5963

fmt

7edc3be

NativeLibFinder simplifies os name

fc325f2

Extracted native lib dir conforms to the hierarchy naming convention

3cdad6e

Merge branch 'develop' into wip/akirathan/11483-polyglot-lib

eb555f4

Move NativeLibraryFinder to pkg and make it generic.

51cc811

Package has nativeLibraryDir method

361ab14

Fix typo in NativeLibraryFinder

218c19f

Fix path to nativeLibDir in pkg

f9b71bc

Akirathan added 2 commits December 20, 2024 17:22

Fix build

9ed04b1

Add NativeLibraryFinder.listAllNativeLibraries

3d9a377

JaroslavTulach reviewed Dec 21, 2024

View reviewed changes

Akirathan added 4 commits December 23, 2024 16:48

Gather native libraries in EnsoLibraryFeature

7bc8768

Stack of exception in EnsoLibraryFeature is printed to System.err.

5a44328

The current printing is not visible on CI

Update docs

158110c

Update docs

a9c60a5

Akirathan marked this pull request as ready for review December 23, 2024 17:22

Akirathan requested review from jdunkerley, radeusgd, GregoryTravis, AdRiley, marthasharkey, 4e6 and hubertp as code owners December 23, 2024 17:22

Akirathan added 3 commits December 23, 2024 18:54

Add unit test for searching native lib

829c0a9

Add printing system info to NativeLibraryFinderTest

5d7d748

Always replace x86_64 by amd64

a1521cd

JaroslavTulach approved these changes Dec 24, 2024

View reviewed changes

Akirathan and others added 4 commits December 24, 2024 13:55

Add list of supported names in docs

f4bec69

Update changelog

ceb3326

fmt docs

b4d3f98

Merge branch 'develop' into wip/akirathan/11483-polyglot-lib

05197f5

JaroslavTulach requested changes Dec 27, 2024

View reviewed changes

enso-bot bot mentioned this pull request Dec 28, 2024

Text.find slows down with the number of invocations #11859

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Native libraries are searched in lib/polyglot project dir #11874

Native libraries are searched in lib/polyglot project dir #11874

Akirathan commented Dec 16, 2024 •

edited

Loading

Akirathan commented Dec 17, 2024 •

edited by unfurl-links bot

Loading

JaroslavTulach left a comment

JaroslavTulach Dec 21, 2024

JaroslavTulach commented Dec 21, 2024

Akirathan commented Dec 23, 2024 •

edited by unfurl-links bot

Loading

Akirathan commented Dec 23, 2024

JaroslavTulach left a comment •

edited

Loading

JaroslavTulach Dec 24, 2024

Akirathan Dec 24, 2024

JaroslavTulach Dec 27, 2024 •

edited

Loading

JaroslavTulach Dec 27, 2024

JaroslavTulach left a comment

JaroslavTulach Dec 27, 2024

JaroslavTulach Dec 27, 2024

	## Native libraries

	Java can load native libraries using, e.g., the
	[System.loadLibrary](<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/System.html#loadLibrary(java.lang.String)>)
	or
	[ClassLoader.findLibrary](<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/ClassLoader.html#findLibrary(java.lang.String)>)
	methods. If a Java method loaded from the `polyglot/java` directory in project
	`Proj` tries to load a native library via one of the aforementioned mechanisms,
	the runtime system will look for the native library in the `polyglot/lib`
	directory within the project `Proj`. The runtime system implements this by
	overriding the
	[ClassLoader.findLibrary](<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/ClassLoader.html#findLibrary(java.lang.String)>)
	method on the `ClassLoader` used to load the Java class.

	The algorithm used to search for the native libraries within the `polyglot/lib`
	directory hierarchy conforms to the
	[NetBeans JNI specification](https://bits.netbeans.org/23/javadoc/org-openide-modules/org/openide/modules/doc-files/api.html#jni):
	Lookup of library with name `native` works roughly in these steps:

	- Add platform-specific prefix and/or suffix to the library name, e.g.,
	`libnative.so` on Linux.
	- Search for the library in the `polyglot/lib` directory.
	- Search for the library in the `polyglot/lib/<arch>` directory, where `<arch>`
	is the name of the architecture.
	- Search for the library in the `polyglot/lib/<arch>/<os>` directory, where
	`<os>` is the name of the operating system.

	if (pkg.nativeLibraryDir().exists()) {
	var nativeLibs =
	NativeLibraryFinder.listAllNativeLibraries(pkg, FileSystem$.MODULE$.defaultFs());
	for (var nativeLib : nativeLibs) {
	assert nativeLib.exists() && nativeLib.isFile();
	var dir = nativeLib.toPath().getParent().toFile();
	assert dir.exists() && dir.isDirectory();
	nativeLibPaths.add(dir.getAbsolutePath());
	var current = System.getProperty("java.library.path");
	RuntimeSystemProperties.register(
	"java.library.path", current + File.pathSeparator + dir.getAbsolutePath());
	}
	}
	}

Native libraries are searched in lib/polyglot project dir #11874

Are you sure you want to change the base?

Native libraries are searched in lib/polyglot project dir #11874

Conversation

Akirathan commented Dec 16, 2024 • edited Loading

Pull Request Description

Important Notes

Native image building modification

Checklist

Akirathan commented Dec 17, 2024 • edited by unfurl-links bot Loading

JaroslavTulach left a comment

Choose a reason for hiding this comment

JaroslavTulach Dec 21, 2024

Choose a reason for hiding this comment

JaroslavTulach commented Dec 21, 2024

Akirathan commented Dec 23, 2024 • edited by unfurl-links bot Loading

Akirathan commented Dec 23, 2024

JaroslavTulach left a comment • edited Loading

Choose a reason for hiding this comment

JaroslavTulach Dec 24, 2024

Choose a reason for hiding this comment

Akirathan Dec 24, 2024

Choose a reason for hiding this comment

JaroslavTulach Dec 27, 2024 • edited Loading

Choose a reason for hiding this comment

JaroslavTulach Dec 27, 2024

Choose a reason for hiding this comment

JaroslavTulach left a comment

Choose a reason for hiding this comment

JaroslavTulach Dec 27, 2024

Choose a reason for hiding this comment

JaroslavTulach Dec 27, 2024

Choose a reason for hiding this comment

Akirathan commented Dec 16, 2024 •

edited

Loading

Akirathan commented Dec 17, 2024 •

edited by unfurl-links bot

Loading

Akirathan commented Dec 23, 2024 •

edited by unfurl-links bot

Loading

JaroslavTulach left a comment •

edited

Loading

JaroslavTulach Dec 27, 2024 •

edited

Loading