Wasting 100MB by Standard.Image: Implement polyglot/lib

[JaroslavTulach]

I've just checked sizes of various libraries inside of Enso distribution:

enso/built-distribution/enso-engine-*/enso-*/lib/Standard$ du -s * | sort -n
164     Geo
196     Searcher
324     Examples
2616    Visualization
4052    Microsoft
4464    Test
16732   Tableau
25272   AWS
44120   Google_Api
46848   Database
48392   Table
63616   Base
72004   Snowflake
118976  Image

e.g. the Standard.Image library occupies 118MB! Why? Because its opencv.jar internally contains various ~10MB huge binaries for each supported platform.

Standard$ unzip -l Image/0.0.0-dev/polyglot/java/opencv-4.7.0-0.jar | grep nu.*opencv.*470 
 20426616  2023-03-28 02:41   nu/pattern/opencv/linux/ARMv7/libopencv_java470.so
 28689512  2023-03-28 02:41   nu/pattern/opencv/linux/ARMv8/libopencv_java470.so
 64297552  2023-03-28 02:41   nu/pattern/opencv/linux/x86_64/libopencv_java470.so
 22088596  2023-03-28 02:41   nu/pattern/opencv/osx/ARMv8/libopencv_java470.dylib
 54792848  2023-03-28 02:41   nu/pattern/opencv/osx/x86_64/libopencv_java470.dylib
 69675008  2023-03-28 02:41   nu/pattern/opencv/windows/x86_32/opencv_java470.dll
108318208  2023-03-28 02:41   nu/pattern/opencv/windows/x86_64/opencv_java470.dll

If we, remove these native libraries and keep just a single one for the platform we build the distribution for we:

• save bunch of disk space
• save CPU time while starting (wasted by extracting these libs to temporary directory)
• extract at build time rather than during runtime
• make it easier for native image to link to such extracted library

Specification

The solution is to enhance HostClassLoader and override its findLibrary method to perform native library search. We will adhere to the NetBeans specification - we are using it anyway in when building enso4igv as the libraries layout is:

$ unzip -l enso4igv*nbm | grep /lib/ | cut -d / -f3- | grep -v /$
lib/aarch64/libenso_parser.dylib
lib/enso_parser.dll
lib/libenso_parser.so
lib/x86_64/libenso_parser.dylib

You may place the native libraries (DLL or shared-object) beneath a polyglot/lib directory of an Enso Library with Polyglot Java libraries. If your native library file names for different architectures or operating systems clash, you may create subdirectories under polyglot/lib for each supported platform and nested subdirectories for each supported operating system. The directory names must match System.getProperty("os.arch") and System.getProperty("os.name").replaceAll(" .*$", "").toLowerCase(Locale.ENGLISH) respectively. The System.loadLibrary call originating from the library code will try to locate the library file in the following order of directories:

    polyglot/lib/
    polyglot/lib/<arch>/
    polyglot/lib/<arch>/<os>/

so you may place e.g. 64-bit Linux version of a foo library in a file polyglot/lib/amd64/linux/libfoo.so and 64-bit Windows version in polyglot/lib/amd64/windows/foo.dll (or also polyglot/lib/amd64/foo.dll and of course in polyglot/lib/foo.dll).

Justification

The OpenCV SharedLoader seems to be using System.loadLibrary and thus, if our HostClassLoader supports findLibrary, it should load it from polyglot/lib directory rather than extracting it to temporary location on each startup.

Tasks

Preview Give feedback

1. Update documentation to talk about polyglot/lib - copy the specification there
2. Override HostClassLoader.findLibrary method - example
3. Modify build to extract the nu/pattern/opencv/*/* libraries at appropriate place in built-distribution/**/Standard/Image/polyglot/lib
4. Modify the build to repackage opencv*.jar and remove the libraries in there
5. Modify the final packaging step to ignore libraries for other OSes - thus minimize garbage
6. Make sure MacOSX signing continues to work

[Akirathan]

Extracting anything from a jar archive during build time can be (easily) managed by setting various assembly sbt settings. I believe that I would be able to define build settings/tasks to do any kind of transformation for the jar. Once you know how should the resulting jar look like, which files should be removed from it, just let me know and I will create a patch of build.sbt that achieves that.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-12-16):

Progress: - Started with creating a separate HostClassLoader per package. It should be finished by 2024-12-25.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-12-17):

Progress: - Reverting separate HostClassLoader per project - it was a mistake - #11874 (comment).

• HostClassLoader will just search native libraries in all the projects. It should be finished by 2024-12-25.

[JaroslavTulach]

• HostClassLoader will just search native libraries in all the projects

+1, there already is work on class loader/layer per library in #10714

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-12-18):

Progress: - Convincing sbt build script to extract native libraries from opencv.jar.

• Found out the issue with InvalidSharing assertion errors in Every builtin type has a common super class #11861 (comment)
  • Changing the construction of polymorphic cache. It should be finished by 2024-12-25.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-12-20):

Progress: - Fixing problems with running Image_Tests on native image.

• Playing with EnsoLibraryFeature to include dll
  • Works with setting specific java.library.path system prop for runtime, but need to point it to built-distribution. It should be finished by 2024-12-25.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-12-23):

Progress: - sbt correctly extracts native libs from opencv.jar.

• Small DevX improvement: The task to extract native libs is not re-executed every time.
• Fixing os name and arch for MacOS - waiting for tests on CI. It should be finished by 2024-12-25.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-12-27):

Progress: - Trying other suggested alternatives to find and load opencv native library.

• How do we force HostClassLoader.findLibrary to load the opencv native lib at NI runtime?
• Initialize org.enso.image and nu.pattern during build time?
  • Does not work because of OpenCV$SharedLoader$HOLDER which is statically initialized to a different object every time.
• Force loading of all classes in EnsoLibraryFeature via HostClassLoader.
  • Having problems with fixing the NI build this way. It should be finished by 2025-01-02.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-12-31):

Progress: - Many failed attempts to include HostClassLoader.findLibrary in the NI runtime - #11874 (comment)

• Pursuing a different idea to copy native libraries along other by-default produced native libs by NI - Native libraries are searched in lib/polyglot project dir #11874 (comment) It should be finished by 2025-01-02.