Crystal Interpreter incorrect lib error: cannot find -lMagickWand-7.Q16HDRI

[jeffbdavenport]

It appears this error is only occurring with this library, however, it does not make any sense. Perhaps it has something to do with the - or the . in MagickWand-7.Q16HDRI ?

The code runs just fine using crystal test.cr but crystal i test.cr generates an error trace.

require "magickwand-crystal"

debugger

LibMagick.newMagickWand

shard.yml

name: project
version: 0.1.1

authors:
  - Jeff Davenport <[email protected]>

dependencies:
  magickwand-crystal:
    github: blocknotes/magickwand-crystal

crystal: 1.4.0-dev

license: MIT

I suspect that the linker arguments printed here are not the linker arguments that were actually used, however, I am not sure where to inspect that to verify
Error trace:

jeff@jeffs-machine project (master)$ crystal i test.cr
cannot find -lMagickWand-7.Q16HDRI
Linker arguments: -L/usr/local/bin/../lib/crystal -L/usr/local/lib -lMagickWand-7.Q16HDRI -lMagickCore-7.Q16HDRI -levent -lrt -ldl
Search path: /usr/local/lib:/usr/local/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/lib32:/usr/lib32:/usr/lib/x86_64-linux-gnu/libfakeroot:/usr/local/lib:/libx32:/usr/libx32:/usr/local/lib/i386-linux-gnu:/lib/i386-linux-gnu:/usr/lib/i386-linux-gnu:/usr/local/lib/i686-linux-gnu:/lib/i686-linux-gnu:/usr/lib/i686-linux-gnu:/lib64:/usr/lib64:/lib:/usr/lib:/usr/local/bin/../lib/crystal:/usr/local/lib (Crystal::Loader::LoadError)
  from /opt/crystal/src/compiler/crystal/loader.cr:74:31 in 'load_library'
  from /opt/crystal/src/compiler/crystal/loader.cr:42:7 in 'new'
  from /opt/crystal/src/compiler/crystal/loader/unix.cr:51:7 in 'parse'
  from /opt/crystal/src/compiler/crystal/interpreter/context.cr:366:5 in 'loader'
  from /opt/crystal/src/compiler/crystal/interpreter/context.cr:378:5 in 'c_function'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:1883:17 in 'compile_lib_call'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:1764:7 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:3000:5 in 'accept_with_wants_value'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:2990:5 in 'request_value'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:578:7 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:3000:5 in 'accept_with_wants_value'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:2990:5 in 'request_value'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:1219:5 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:566:7 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:234:5 in 'compile_def'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:1979:7 in 'create_compiled_def'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:1777:22 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:234:5 in 'compile_def'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:1979:7 in 'create_compiled_def'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:1777:22 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:447:7 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:234:5 in 'compile_def'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:1399:5 in 'get_const_index_and_compiled_def'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:2486:27 in 'initialize_const_if_needed'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:1371:17 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:3000:5 in 'accept_with_wants_value'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:2990:5 in 'request_value'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:2144:9 in 'compile_call_args'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:1773:5 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:234:5 in 'compile_def'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:1399:5 in 'get_const_index_and_compiled_def'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:680:31 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:2904:5 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:566:7 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:3000:5 in 'accept_with_wants_value'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:2994:5 in 'discard_value'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:2852:7 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:566:7 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:234:5 in 'compile_def:closure_owner'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:2952:5 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:566:7 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:234:5 in 'compile_def:closure_owner'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:2952:5 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:566:7 in 'visit'
  from /opt/crystal/src/compiler/crystal/syntax/visitor.cr:27:12 in 'accept'
  from /opt/crystal/src/compiler/crystal/interpreter/compiler.cr:165:5 in 'compile'
  from /opt/crystal/src/compiler/crystal/interpreter/interpreter.cr:207:5 in 'interpret'
  from /opt/crystal/src/compiler/crystal/interpreter/repl.cr:146:5 in 'interpret'
  from /opt/crystal/src/compiler/crystal/interpreter/repl.cr:150:5 in 'interpret_and_exit_on_error'
  from /opt/crystal/src/compiler/crystal/interpreter/repl.cr:121:5 in 'run_file'
  from /opt/crystal/src/compiler/crystal/command/repl.cr:17:7 in 'repl'
  from /opt/crystal/src/compiler/crystal/command.cr:102:7 in 'run'
  from /opt/crystal/src/compiler/crystal/command.cr:51:5 in 'run'
  from /opt/crystal/src/compiler/crystal/command.cr:50:3 in 'run'
  from /opt/crystal/src/compiler/crystal.cr:11:1 in '__crystal_main'
  from /opt/crystal/src/crystal/main.cr:115:5 in 'main_user_code'
  from /opt/crystal/src/crystal/main.cr:101:7 in 'main'
  from /opt/crystal/src/crystal/main.cr:127:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from crystal in '_start'
  from ???

my /usr/local/lib folder:

jeff@jeffs-machine lib$ pwd
/usr/local/lib
jeff@jeffs-machine lib$ ls -lah libMagickWand-7.Q16HDRI.*
-rw-r--r-- 1 root root 7.8M Sep  4 13:21 libMagickWand-7.Q16HDRI.a
-rwxr-xr-x 1 root root 1.2K Sep  4 13:21 libMagickWand-7.Q16HDRI.la
lrwxrwxrwx 1 root root   33 Sep  4 13:21 libMagickWand-7.Q16HDRI.so -> libMagickWand-7.Q16HDRI.so.10.0.0
lrwxrwxrwx 1 root root   33 Sep  4 13:21 libMagickWand-7.Q16HDRI.so.10 -> libMagickWand-7.Q16HDRI.so.10.0.0
-rwxr-xr-x 1 root root 4.1M Sep  4 13:21 libMagickWand-7.Q16HDRI.so.10.0.0

[straight-shoota]

I suspect that the linker arguments printed here are not the linker arguments that were actually used, however, I am not sure where to inspect that to verify

That's correct. crystal i doesn't use an actual linker because there's no linkable machine code generated. Instead, it loads libraries directly. The linker args shown in the error message are printed by Crystal::Loader in a similar manner as a linker would use them. You can find the source code for that here:

crystal/src/compiler/crystal/loader/unix.cr

Line 30 in a978a06

def self.parse(args : Array(String), *, search_paths : Array(String) = default_search_paths) : self

I don't think any of the characters in the library name should be a problem. The printed args suggest that the loader understood the library name correctly. To make sure, you could add a puts in Loader#open_library to log which paths are actually loaded.
Alternatively, you could try directly in a separate program if loading the library actually works: LibC.dlopen("/usr/local/lib/libMagickWand-7.Q16HDRI.so", LibC::RTLD_LAZY | LibC::RTLD_GLOBAL). Maybe LibC.dlerror could shed some light on an error. But I believe it's just empty when dlopen doesn't find the file.

If you provide some information on your system and how to install libMagickWand, I could try to reproduce the error.

[jeffbdavenport]

Interesting. This is helpful.

I tried to run what you said and this is what I see:

icr:6:0> LibC.dlopen("/usr/local/lib/libMagickWand-7.Q16HDRI.so", LibC::RTLD_LAZY | LibC::RTLD_GLOBAL)
=> Pointer(Void).null
icr:7:0> String.new(LibC.dlerror)
=> "/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block"

But if I use export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgomp.so.1 before running, I just get a null pointer

If I add in the puts where you said,
I get this output supplemented by the same stack trace, which the path appears to be fine.

jeff@jeffs-machine project (master)$ crystal i test.cr 
/usr/local/lib/libMagickWand-7.Q16HDRI.so
/usr/local/lib/libMagickWand-7.Q16HDRI.so
cannot find -lMagickWand-7.Q16HDRI
Linker arguments: -L/usr/local/bin/../lib/crystal -L/usr/local/lib -lMagickWand-7.Q16HDRI -lMagickCore-7.Q16HDRI -levent -lrt -ldl

jeff@jeffs-machine lib$ ll /usr/local/lib/libMagickWand-7.Q16HDRI.so
-rwxr-xr-x 1 root root 4255024 Feb 28 10:45 /usr/local/lib/libMagickWand-7.Q16HDRI.so*

I am just running on Ubuntu 21.10, I compiled the ImageMagick package using this repository https://github.com/ImageMagick/ImageMagick.git

LibC.dlopen("/usr/lib/x86_64-linux-gnu/libMagickWand-6.Q16HDRI.so.6", LibC::RTLD_LAZY | LibC::RTLD_GLOBAL)

If I do the above, It loads the library just fine. It seems to be something with the compiled version, after doing sudo make uninstall on that library the original program seems to run just fine

[straight-shoota]

All right. /lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block should be the relevant information here. The rest probably doesn't help anything.

I don't know what exactly that error means or what you can do to fix it. Maybe tweak the build configuration of the library?
At least this doesn't seem to be a problem with the loader itself, it's an issue with dlopen and the specific library. So I'm closing this issue because there's nothing actionable for us. That does not mean to cut off discussion, though. I'm happy to continue that here and try to understand the cause.

It might however add an item to our problems-with-libdl list.

[jeffbdavenport]

Well, personally I am satisfied with uninstalling the compiled version here to resolve my issue. But I disagree that there is nothing actionable. It would have been much nicer if the error that was returned was an appropriate error message and not "cannot find -lMagickWand-7.Q16HDRI". That was leading me down the path of spending a long time trying to install the library, troubleshooting the filesystem, and the linker load path, which all were not the issue.

[straight-shoota]

Yes, that's totally right of course. Sorry.
I had actually already fixed a patch for that, I just got hold up publishing it. Here it is: #11860