"No such file or directory" when running post_install_hook.sh

[sschuberth]

Repeating a bit of background information from this Slack conversation:

I'm setting up a Kotlin / Gradle project (with a non-GraalVM JDK) in which I'd like to use TruffleRuby to run code from a Ruby Gem to be installed. My dependencies include

"org.graalvm.polyglot:polyglot:23.1.2"
"org.graalvm.polyglot:ruby:23.1.2"

and my Kotlin code looks like

Context.newBuilder().allowAllAccess(true).build().use { context ->
    context.eval("ruby", "Gem.install('licensee')")
}

However, this gives me

Caused by: org.graalvm.polyglot.PolyglotException: OpenSSL is not available. Install OpenSSL and rebuild Ruby (preferred) or use non-HTTPS sources

After an Internet search and reading through https://github.com/oracle/truffleruby/blob/master/doc/user/installing-libssl.md, I was looking for the post_install_hook.sh script on my system, which apparently gets installed at

~/.cache/org.graalvm.polyglot/ruby/ruby-home/710802544723712c59e4720b88bab3e008b5ef6c027bf4df856b9a50ac6cde761f38bc1d35dbd4a966ad7d58d8bfb3d6861b9652b03a4805428e4cb2d16d5d7f/lib/truffle/post_install_hook.sh

However, running that script (from the base directory with the long hash) yields

$ ./lib/truffle/post_install_hook.sh
Recompiling the OpenSSL C extension (against the installed libssl)
./lib/truffle/post_install_hook.sh: line 20: cd: src/main/c/openssl: No such file or directory

So apparently, the installation seems to be incomplete, or is looking for files in the wrong place.

[eregon]

Thanks for the report.

Writing down some thoughts on this:
So at least one issue here is we'd need to ship src/main/c/openssl for the embedding-via-maven/gradle use case too.
And we should make it easy/easier to run that when using the Context API, or at least document how.
But the problem is lib/truffle/post_install_hook.sh needs a truffleruby executable and we don't ship it on Maven Central currently. So we would need to include that too. I'm not sure how that would work though because there won't be a librubyvm.so nor a jvm/ under the ruby home, so that truffleruby executable won't find java/libjvm.
We'd need the caller Java process using the Context API to pass the java home or path to libjvm or so, e.g. via some env var. I don't think this information is passed currently.

We could also convert that script to a Ruby file, but that would not help, because recompiling openssl means running truffleruby extconf.rb in src/main/c/openssl and so we still need a truffleruby executable.
Also we don't ship org.graalvm.ruby.launcher/org.truffleruby.launcher/RubyLauncher (the Java code specific to the launcher) to Maven Central currently (could be done though).

I think for now it should be possible to workaround by installing a TruffleRuby standalone and using that to recompile openssl, and copy the result over in the cache (~/.cache/org.graalvm.polyglot/ruby/ruby-home/...).

Another solution would be to vendor libssl, but that is problematic for multiple reasons, notably some C extensions do not work with that when they depend on a system package which depends on system libssl, loading multiple libssl in the same process feels brittle, and security-wise it's better to use the libssl from the operating system.

As a note, on Oracle Linux 7 there should be no need to recompile the openssl extension, because we ship it compiled against the Oracle Linux 7 system libssl.

One more possibility would be to reimplement Ruby OpenSSL on top of the Java security APIs.
Unfortunately the Ruby OpenSSL API is so vast that reimplementing it on top of the Java security APIs seems a huge effort and unlikely to cover all the necessary parts (JRuby ships BouncyCastle + jruby-openssl for this). However GraalPy did this and it seems to work well (but different OpenSSL API in Python of course).

[eregon]

Something GraalPy does is shipping the Java module for the launcher code and using/generating a small Bash launcher script:
https://github.com/oracle/graalpython/blob/release/graal-vm/23.1/graalpython/lib-graalpython/modules/standalone/__main__.py#L122
That seems the easiest solution.

And then we could document running the post-install hook using the maven exec plugin or similar.

[djberg96]

Similar error, different cause. I was trying to install truffleruby 24.1.1 on a Ubuntu 24 image, and hit this:

==> Installing truffleruby-24.1.1...
-> ./lib/truffle/post_install_hook.sh

BUILD FAILED (Ubuntu 24.04 on x86_64 using ruby-build 20241105-9-g7ccb143c)

Looking at the log I saw this:

compiling ossl_x509store.c
linking shared-object openssl.so

Recompiling the Psych C extension (against the installed libyaml)
checking for yaml.h... no
yaml.h not found
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/root/.rbenv/versions/truffleruby-24.1.1/bin/truffleruby
	--with-libyaml-source-dir
	--without-libyaml-source-dir
	--with-libyaml-dir
	--without-libyaml-dir
	--with-libyaml-include
	--without-libyaml-include=${libyaml-dir}/include
	--with-libyaml-lib
	--without-libyaml-lib=${libyaml-dir}/lib

Contents of mkmf.log:
find_header: checking for yaml.h... -------------------- no

LD_LIBRARY_PATH=. "gcc -o conftest -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include/ruby/backward -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include -I.    -O3 -fno-fast-math -ggdb3 -Werror=implicit-function-declaration -Wno-int-conversion -Wno-int-to-pointer-cast -Wno-incompatible-pointer-types -Wno-format-extra-args conftest.c  -L.          -L/root/.rbenv/versions/truffleruby-24.1.1/lib/cext -Wl,-rpath,/root/.rbenv/versions/truffleruby-24.1.1/lib/cext -ltrufflerubytrampoline"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: int main(int argc, char **argv)
4: {
5:   return !!argv[argc];
6: }
/* end */

LD_LIBRARY_PATH=. "gcc -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include/ruby/backward -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include -I.    -O3 -fno-fast-math -ggdb3 -Werror=implicit-function-declaration -Wno-int-conversion -Wno-int-to-pointer-cast -Wno-incompatible-pointer-types -Wno-format-extra-args   -c conftest.c"
conftest.c:3:10: fatal error: yaml.h: No such file or directory
    3 | #include <yaml.h>
      |          ^~~~~~~~
compilation terminated.
Process failed: #<Process::Status: pid 630 exit 1>
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: #include <yaml.h>
/* end */

--------------------

external command failed with status 1

The solution was to explicitly add libyaml-dev as one of the installed packages.

[andrykonchin]

Nice work!

To be fair the dependencies (and libyaml as well) are listed in the README document and mentioned in the Installing TruffleRuby one.