Julia dependencies installation fails due to libunwind

[valeriupredoi]

GA test fail and Circle test fail symptomatic due to Julia not being built with libunwind=1.6

See these:

• Change pinning to include minor version? conda-forge/libunwind-feedstock#19
• Failing to install julia-1.6.4: Segmentation fault conda-forge/julia-feedstock#132
• Update pin for libunwind conda-forge/conda-forge-pinning-feedstock#2260

[zklaus]

I did some investigation. To me, it seems to be due to problems of the upstream Julia repositories.

[valeriupredoi]

cheers, Klaus! I am getting this with julia 1.6.4:

(esmvaltool-stock) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool$ julia /home/valeriu/ESMValTool/esmvaltool/install/Julia/setup.jl --verbose
[ Info: Installing Julia dependencies
  Activating environment at `~/ESMValTool/esmvaltool/install/Julia/Project.toml`
  Installing known registries into `~/.julia`
Segmentation fault (core dumped)

I'll try 1.6.3 that worked before

[valeriupredoi]

no luck, same situation with julia=1.6.3 - what do you think, Klaus, wait see if they sort their issues out?

[zklaus]

Hmm. For me, it worked even with 1.6.4. Have you tried nuking your ~/.julia directory?

[valeriupredoi]

indeed I did, still no show. What OS you using? Could it be Ubuntu-related?

[valeriupredoi]

OK here's what I did: I used a slightly older environment (from about a week or so ago), and installed the Julia deps all fine (with esmvaltool install Julia), then I looked up the julia version in that env: julia=1.6.3=h2a39549_1 from conda-forge, and fixed the exact version number and build hash in the environment file to create a new env, did that, and checked again the installation of Julia packages - this time around it fails with segfault - it is a dependency of Julia that is creating that problem, something that gets updated in the newly built env bit stays at a working version in the older env. We can probably identify that dependency/library, but is it worth the effort?

[valeriupredoi]

BINGO! I found the offending dependency (got lucky it was only one that differed from the two envs): libunwind, the working one is

# Name                    Version                   Build  Channel
libunwind                 1.5.0                h9c3ff4c_1    conda-forge

and the one that's killing Julia is 1.6

[zklaus]

Thanks for the excellent detective work. Let's not go down the pinning rabbit hole just yet. What this needs is a rebuild of Julia upstream. I am discussing with the conda-forge crew how to progress. Feel free to jump in in the gitter room https://app.element.io/#/room/#conda-forge:matrix.org

[valeriupredoi]

awesome, I will, in a second, I created a PR with the pin of libunwind just to be 100% sure that's the culprit (on a proper test platform), so I'll link that in the discussion, we don't have to merge it, as long as Julia rebuild happens soon enough 👍

[zklaus]

So... As @valeriupredoi figured out, there was an update of libunwind. This broke abi compatibility in Julia, which means that the Julia package on conda-forge should be rebuilt. The question is why this didn't happen automatically and what needs to be changed in the conda-forge bowels. After brief discussions with the conda-forge crowd, we came down on relying on the abi-laboratory for an assessment, which means that we have to wait until tomorrow.

[valeriupredoi]

yer, and Klaus led me to what appears to be the most lively conversation about dependencies I've ever seen, I must have transcended to the highest level of geek space 🤣

[mkitti]

I'm the julia-feedstock maintainer of the recent releases. I'm watching here and on gitter.

[valeriupredoi]

@mkitti welcome to our land! For now it's best to wait (until Tuesday, or latest Wed), since the ABI Lab will get rebuilt with the latest libunwind=1.6 every Tuesday and Friday - that is currently causing Julia not to work properly, @zklaus can tell you what that ABI Lab is since I got no idea 😆

[valeriupredoi]

@zklaus our CI tests are still failing even though it's Wed now (feels a bit like Tokyo Drift 😁 ) - has the ABI been updated?

[mkitti]

https://abi-laboratory.pro/index.php?view=timeline&l=libunwind ABI tests ran yesterday and did not show a problem with compatibility.

[valeriupredoi]

cheers @mkitti - unfortunately our CI is still stumbling - could it be that now the environment gets done with yet another newer libunwind=1.6.2?

[valeriupredoi]

OK since conda-forge/libunwind-feedstock#19 and another related conda-forge/julia-feedstock#132 are still open I am guessing the compatibility with libunwind>=1.6 is not ready yet 👍
Sorry, also of actual interest conda-forge/conda-forge-pinning-feedstock#2260

[zklaus]

I agree that things are still in motion and we need a bit more time. Perhaps we should deactivate the julia bit in our CI for the time being?

@mkitti, I think the timeline of the ABI tests does show compatibility problems. Specifically, the 1.6.0 version shows only 97% backwards compatibility. Drilling down (clicking on the percentage), we find that struct unw_dyn_info got a new member variable added somewhere in the middle, shifting all following members in the memory layout. That means that code that was compiled against the earlier version will dereference (sort of) random memory as pointers, which is exactly the kind of thing that produces a segfault.

[mkitti]

I see it now. https://abi-laboratory.pro/index.php?view=compat_report&l=libunwind&v1=1.5.0&v2=1.6.0&obj=8ee57&kind=abi

[valeriupredoi]

julia=1.6.4 works well with libunwind=1.5.0 see here - I will close this since this configuration now works well for us, cheers 🍺