Symbol not found: _dlacpy_64_ when calling svdvals on a DistribuedArray

[JBlaschke]

Hi,

I tried the example here: https://github.com/JuliaParallel/Elemental.jl#svd-example-with-distributedarrays-on-4-processors on my macOS 11.6.1 laptop and I get:

dyld: lazy symbol binding failed: Symbol not found: _dlacpy_64_

I'm wondering if this is a macOS problem (do I need to set a path maybe?), and if anyone here has any ideas.

[andreasnoack]

It looks like you are fetching the wrong BLAS. How did you install the package as well as Julia?

[JBlaschke]

I installed Julia using homebrew, and Elemental using the Pkg CLI.

When I look for _dlacpy_ in libEl, I see that it needs _dlacpy_64_

nm ~/.julia/artifacts/05b4bcf6b87541795cebdcb2ec42f2617af03590/lib/libEl.0.dylib | grep _dlacpy_
                 U _dlacpy_64_

But when I look for _dlacpy_ on the entire system, I don't find _dlacpy_64_ ... only _dlacpy_ and variants like: _LAPACKE_dlacpy_work. I looked on the entire laptop using:

for lib in (find / -name \*.dylib 2>/dev/null)
    echo $lib
    nm $lib | grep _dlacpy_ | grep -v " U "
end >> find_dlacpy

I assume that I need to install a specific version of BLAS.

[JBlaschke]

Some more info: I did a system-wide search for all ELF references to _dlacpy_ (i.e. not excluding the U entries from nm -- as I did above). The only references to _dlacpy_64_ that I could find are in libEL. This leads me to believe that it's built against a different BLAS version than what's available on a mac.

Can you advise what I should install?

[JBlaschke]

BTW @andreasnoack I have openblas installed (also using Homebrew), but that doesn't define the 64 version, just the standard _dlycpy_.

[JBlaschke]

This is the way openBLAS is build in Homebrew: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/openblas.rb

I wonder if the problem is that the default is not built with INTERFACE64=1 ...
JuliaLang/julia#1527

[andreasnoack]

The problem is the Homebrew julia. Please try the version from julialang.org. I believe it should fix the issue here.

[JBlaschke]

I stumbled across the same solution. Thanks! However I prefer to have package managers manage my software for me. This made me try the Julia cask: brew install --cask julia. This also works, but duplicates a lot of system libraries (not a problem when your laptop has 1TB of SSD ... but a bit of an issue when you don't 😉 ). So I also built this Homebrew tap: https://github.com/JBlaschke/homebrew-blas64 which has a version of Julia which builds and links against the 64-bit BLAS (using SYMBOLSUFFIX=64_), but re-uses the remaining system libraries.

Also note that Elemental.jl could check the BLAS vendor:

1. Using regular brew install julia:

julia> using LinearAlgebra; LinearAlgebra.BLAS.vendor()
:openblas

2. Using brew install --cask julia, or brew tap JBlaschke/homebrew-blas64; brew install julia64:

julia> using LinearAlgebra; LinearAlgebra.BLAS.vendor()
:openblas64

Let me know if you want me to add this check -- it could warn the user that they are using the wrong version of Julia. Alternatively, I could investigate at https://github.com/JuliaLinearAlgebra/libblastrampoline

[andreasnoack]

The linking here is a bit complicated but I think that Elemental is actually searching the wrong library here since it's supposed to use the version of OpenBLAS declared as a dependency in https://github.com/JuliaBinaryWrappers/Elemental_jll.jl/blob/b72537ed6bde7b1f52bc9cefc26153c6fef2aff3/Project.toml#L8. I'm not sure why it is not using that version but instead the one that Julia is linked against. Maybe it's something in the homebrew configuration of the julia tap or maybe it's the Elemental binary which isn't correctly configured. Let's ask @staticfloat since he probably knows.

[staticfloat]

As long as OpenBLAS ships with Julia, OpenBLAS_jll will always refer to that same BLAS library. Because OpenBLAS is "special", it gets loaded at startup, and declaring a dependency on OpenBLAS_jll won't let you change which version of OpenBLAS you're loading; you're always going to get the one that Julia ships with (at least, for now!).

The issue here is that we make assumptions about what Julia ships with, but then having Homebrew use system-provided libraries breaks that assumption. This is true for OpenBLAS, but it can be true for many other things as well, so while you're welcome to use this kind of Julia @JBlaschke, please note that it's not unusual for things to break in unexpected ways due to using different versions of libpcre, libcurl, libopenblas, etc... than Julia is expecting. I know it sacrifices a couple of hundred MBs of disk space, but we think it's more than worth it to have a standardized environment that we can support across many different kinds of machines.