Too many old Julia versions, 1.5 is not packaged, and all versions fail in tests

[doronbehar]

Julia 1.5 is out, see: https://docs.julialang.org/en/v1/NEWS/ and we don't have it packaged. More over, see how all-packages.nix defines our other Julias:

nixpkgs/pkgs/top-level/all-packages.nix

Lines 8935 to 8956 in fdfa609

	julia_07 = callPackage ../development/compilers/julia/0.7.nix {
	gmp = gmp6;
	inherit (darwin.apple_sdk.frameworks) CoreServices ApplicationServices;
	};
	julia_10 = callPackage ../development/compilers/julia/1.0.nix {
	gmp = gmp6;
	inherit (darwin.apple_sdk.frameworks) CoreServices ApplicationServices;
	};
	julia_11 = callPackage ../development/compilers/julia/1.1.nix {
	gmp = gmp6;
	inherit (darwin.apple_sdk.frameworks) CoreServices ApplicationServices;
	};
	julia_13 = callPackage ../development/compilers/julia/1.3.nix {
	gmp = gmp6;
	inherit (darwin.apple_sdk.frameworks) CoreServices ApplicationServices;
	};
	julia_1 = julia_10;
	julia = julia_1;

Several issues: julia is pointing to a TLS version which is old IMO. Certainly there are breaking changes seen in their NEWS pages, but since Nixpkgs is a somewhat rolling release distro, are we going to keep these older versions and default to them for ever? I've once tried to reach for the community to hear if there's a strict policy regarding such scenarios, but I think the answer is no

Anyway, I'd like to propose a change that will keep only the LTS version and the latest - currently 1.4, and perhaps make the attribute julia, point to the latest.

Any thoughts from maintainers or users?

cc @raskin @rob @garrison

[doronbehar]

Also, now I noticed julia_13 all of our Julias fail to build, e.g: https://hydra.nixos.org/build/123240981/nixlog/1

[garrison]

Anyway, I'd like to propose a change that will keep only the LTS version and the latest - currently 1.4, and perhaps make the attribute julia, point to the latest.

I agree. And the 1.5 release candidates are out, so this will become the latest version in the near future.

Last time I looked, the package built the same deps for all versions, including 0.7. In versions 1.0 and later, there are actually fewer dependencies (e.g. arpack and fftw have been dropped), so the package should be cleaned up to not include these unnecessary libraries in the build.

I also think the default version should be the latest version (i.e., unless somebody explicitly installs julia-1-lts, they should get the latest).

[doronbehar]

I also think the default version should be the latest version (i.e., unless somebody explicitly installs julia-1-lts, they should get the latest).

OK I'm glad I have your support. I'll let this issue stay open for a while and hopefully I'll be able to take care of this a month or so from now.

[veprbl]

cc @7c6f434c

[7c6f434c]

Back when Julia tests were not so messed up, I think there was often 1 previous version.

I guess with 1.3 needing overriding to no tests, it is fine to just have julia-1-lts and Julia 1.4 as julia once Julia 1.4 gets built fine with some tests.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-can-we-build-a-julia-derivation-with-tests-disabled/8052/2

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-can-we-build-a-julia-derivation-with-tests-disabled/8052/3

[rbvermaa]

Haven't had a lot of time to work on julia in nixpkgs, so just posting this as a FYI.

We used multiple versions of julia at work, but we use a custom derivation, similar to this: https://gist.github.com/rbvermaa/08f916a63ba4f3f6123f97478309dfa0

We also do not run the tests, but in general it has been working great.

Previously we also used the binary builds patched with patchelf, as there were some differences in the structure of the results of the vanilla binary build provided by julia and what we got when building from source. We gave up on using 'system' libraries, and build the -full.tar.gz releases that julia provides, as julia is very specific in the versions for its dependencies (LLVM patches mostly).

[doronbehar]

@rbvermaa I tried to use that -full.tar.gz tarball in order to write a derivation from scratch. Here's my attempt:

https://github.com/doronbehar/nixpkgs/blob/julia-latest/pkgs/development/compilers/julia/default.nix

It fails with:

Could not find working curl, wget, or fetch.
You need to install one of these to download dependencies.
make[1]: *** [/build/julia-1.4.2/deps/suitesparse.mk:144: /build/julia-1.4.2/deps/srccache/SuiteSparse.v5.4.0-2.x86_64-linux-gnu.tar.gz] Error 1
make: *** [Makefile:60: julia-deps] Error 2

Naturally I didn't further attempt to add curl or wget to the inputs, as that would fail in a sandbox as well. I've opened an issue upstream: JuliaLang/julia#36604 complaining that that -full.tar.gz tarball doesn't really contain all dependencies.

[rbvermaa]

@doronbehar I ran with sandbox enabled, and the expression I linked with works for me. So I think it is fine to just add curl to the deps, as with the full tarball, it isn't actually used.

[rht]

There is another approach to avoid running tests: using precompiled binary. I tried downloading the 1.4.2 binary from https://julialang.org/downloads/ and then afterward ran patchelf on the binary, then I got a running Julia interpreter just like that.

[rbvermaa]

We did the same for a while using code similar to this:

  patchJuliaBinaryLinux = args:
    stdenv.mkDerivation ({
      installPhase = ''
        cp -R . $out
        if [[ ! -e $out/lib/julia/libz.so.1 ]]; then
          ln -s ${zlib}/lib/libz.so.1 $out/lib/julia/libz.so.1
          ln -s ${zlib}/lib/libz.so $out/lib/julia/libz.so
        fi
        for i in $out/bin/*; do
          patchelf --set-interpreter $(cat $NIX_CC/nix-support/dynamic-linker) \
                   $i
          done
        '';
        dontStrip = true;
      } // args);

....

     julia_14_binary =
       patchJuliaBinaryLinux rec {
         version = "1.4.2";
         name = "julia-${version}";
         src = builtins.fetchTarball "https://julialang-s3.julialang.org/bin/linux/x64/1.4/julia-${version}-linux-x86_64.tar.gz";
       };

Unfortunately, we actually sometimes need to patch Julia, so we switched to the julia expressions from source, that I posted earlier.

[doronbehar]

Thanks @rbvermaa that expression works well thanks to "USE_BINARYBUILDER=0". I'm testing it now.

[doronbehar]

OK so after a long build, I got a working Julia derivation with these references:

/nix/store/y8n2b9nwjrgfx3kvi3vywvfib2cw5xa6-libunistring-0.9.10
/nix/store/fhg84pzckx2igmcsvg92x1wpvl1dmybf-libidn2-2.3.0
/nix/store/bqbg6hb2jsl3kvf6jgmgfdqy06fpjrrn-glibc-2.30
/nix/store/858a98xy9lxm322wjgsw150clmi8zf2h-gfortran-9.3.0-lib

And this closure size:

/nix/store/d13gkgdl809rm2xdwscqjqhvhwpy47w2-julia-1.4.2	 298.1M

Which I must say it is satisfying. The question is, is this how we'd want it to be built? Using all of their dependencies and practically non of ours? Some of their dependencies are required to be patched, according to their docs and they also specify that using the OS' libraries is not officially supported? Anyway I'd happy to get help on deciding this.

[7c6f434c]

@doronbehar does this lead to Julia tests passing?

I think for every dependency they patch, it is OK to just use their patched version. I guess it could be nice to use a separate build unpatched (but using the exact same version sounds like a likely restriction), but whatever dependency does not work right away can be switched to bundled.

[doronbehar]

@doronbehar does this lead to Julia tests passing?

I think it did, I've set doCheck = true; and the build passed but I haven't inspected the log much, but here it is:

julia.log

Currently the expression at my branch produces this log but there's no PR yet as I think there are many makeFlags that should probably be set as well. If anyone feels confident to open a PR with this feel free to ping me for a review.

I think for every dependency they patch, it is OK to just use their patched version.

I agree with the idea, but what's bothering me is that it's not absolutely clear what versions are patched and which are not - the docs vs what's actually found in their patches directory is not totally inline. Also, it seems that some of the patches are old, and were suggested upstream (to the libraries) but they weren't merged there for some reason. I'm still investigating, See JuliaLang/julia#31215 (comment)

My current target is to decide what libraries to use our version vs their version, with the preference to use our version, in order to hopefully reduce closure size due to sharing shared libraries. The idea is to decide this by checking whether the libraries have included the patch Julia suggests, and then whether the library's maintainer has released a new version that includes Julia's maintainers' patch, and whether we ship that new version.

[7c6f434c]

> @doronbehar does this lead to Julia tests passing? I think it did, I've set `doCheck = true;` and the build passed but I haven't inspected the log much, but here it is: https://gist.github.com/f11b21fb53984d481d5d5480ec0dc179

I think this log is cut short before the interesting part…

[doronbehar]

I've uploaded it here:

julia.log

[7c6f434c]

Hmmm. Reading the log does not convince me the tests were run… Pity.

[rht]

Since 1.5 was just released, the title should probably be updated to be 1.5.

[tbenst]

Downside with the patching approach, is users lose the ability to compile against MKL, which has HUGE performance increase (last benchmark I saw showed that MKL was ~4x faster at matrix multiplication!!). Would need to also package JuliaPro to get MKL. Of course, anything working > nothing working, although would be a shame to lose compilation options.

[garrison]

lose the ability to compile against MKL

Says https://github.com/JuliaLang/julia/blob/master/doc/build/build.md:

If you are building Julia for the sole purpose of incorporating Intel MKL, it may be beneficial to first try MKL.jl. This package will automatically download MKL and rebuild Julia's system image against it, sidestepping the need to set up a working build environment just to add MKL functionality.

[tbenst]

Thanks garrison, I saw that, but I have a feeling it won't play nicely with Nix...although if we go patching route, perhaps could do that during buildPhase

[doronbehar]

I have no interest wasting time working around bugs that authors tried to spare me by specifying precise versions.

I see. You are talking about "authors" of Nixpkgs' expressions? I'm still a bit perplexed, as I earlier said, and correct me if I'm wrong: python setup.py should fail if == is used in setup.py, and the exact version is not available. Sometimes (e.g), we do hardcode such version requirement changes in our expressions, but hopefully that's done only when it's justified.

It seems a sort of cognitive dissonance in our next community that we all care so deeply about reproducible builds, but for convenience often disregard the officially supported dependencies.

I agree (maybe a bit related). Generally I think, every dep's version related issue should be handled by someone like you, who's familiar with python packages you use. I'd at least (open PRs) to fix the issues I encounter :).

Pkg.add is mostly declarative. Of course, Julia is constantly re-compiling code, and until there is true support for precompilation, I don’t see how it’s possible to use nix store.

I see. I guess the situation is more complex than I thought, but I don't think I'd be able to form a strong opinion until I'll get a Julia interpreter from an expression I'm satisfied with.

hub.packtpub.com/julia-co-creator-jeff-bezanson-on-whats-wrong-with-julialang-and-how-to-tackle-issues-like-modularity-and-extension

Thanks for the link :).

[tbenst]

Sometimes (e.g), we do hardcode such version requirement changes in our expressions, but hopefully that's done only when it's justified.

Unfortunately, this pattern is done constantly (including in my own contributions!). The problem is that, except an extremely rare cases like tensorflow, nixpkgs does not allow multiple versions of same python package to co-exist. Every time we update a package, we remove the previous version from nixpkgs.

Confounding this issue, we often disable tests because sometimes the effort is just too high when developers bake in assumptions that aren’t compatible with the Nix ethos (for some reason this seems to happen more with tests then source code...). The most frequent cause of failure is the code needs to download something to run the test. (I’ve tried to lobby in the past that the test phase should be permitted to have Internet access as long as it doesn’t modify /nix/store, but I digress).

In any case, the end result is that using a python package on Nix typically means you aren’t using the exact dependencies that upstream declared, and you’re more likely to have a package that despite having changes in dependencies has never had the test run.

Thus, amazingly, I’ve had more luck with Julia packages working on NixOS than Python when all other binaries are present in environment.

[doronbehar]

except an extremely rare cases like tensorflow, nixpkgs does not allow multiple versions of same python package to co-exist.

@tbenst I don't think there's a strong policy in our Python ecosystem against multiple versions of packages - if this is what upstream recommends. Anyway, the whole topic is out of scope for this issue. Feel free to open a new issue and cc me and @jonringer who's working hard to ship a 20.09 release.

[tbenst]

Yes, sorry getting a little off topic here. I mainly mention it to suggest that our efforts in Julia that start from scratch Can learn from these issues. I’m not too familiar with the new binary builder, but BinDeps.jl was quite nice in that it would first search for libraries before attempting to install. See this issue for some context: JuliaPackaging/BinDeps.jl#115. Note that a core Julia dev gave a pretty great endorsement on Nix (albeit back in 2014): JuliaPackaging/BinDeps.jl#115 (comment).

I think it might be most productive to do this development outside of nixpkgs, and do it in Julia to hook in with nixpkgs as needed. That way users could seamlessly use Pkg in Julia. I’d love to ]add Flux and have it just work instead of the current hack that I wrote that a lot of people seem to be using (#70536 (comment))

[garrison]

Short summary of the Discourse thread: for LLVM and libuv it is essential to use the patches/forks. For other packages it is probably unnecessary to use the exact version specified by upstream.

[doronbehar]

Thanks @garrison for being there with me :). I think I'm gonna work on this once we are through with #93333, to avoid merge conflicts.

[doronbehar]

TIL Guix manages to package Julia 1.4.1:

https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/julia.scm?id=master

What's interesting, is that they USE_SYSTEM_LLVM but they build it with Julia's LLVM patches fetched from the internet - they don't rely on Julia's build system to build it's own LLVM. For similar dependencies which the real system's version can't be used, they do the same.

I think we should try to copy this approach.

[nalimilan]

FWIW I did that in Fedora originally, but I stopped. Since LLVM makes breaking releases very quickly, you'll probably want to move to a new version that Julia doesn't support quite soon.

[doronbehar]

you'll probably want to move to a new version that Julia doesn't support quite soon.

Maybe I don't understand, but why should I move to an LLVM version that julia doesn't support? Also, aren't the llvm_7 and llvm_8 versions don't introduce breaking changes between minor releases? We could stick to llvm_8 since that's what Guix does and I think Julia uses this version as a base too.

[nalimilan]

I don't really know how NixOS works, hence the "FWIW". If you used versioned LLVM packages and keep older major versions then it's probably OK.

[cmcaine]

Just offering a user perspective:

Julia provides a package manager that provides mostly reproducible (in a practical/scientific reproducibility sense rather than a bit-identical sense) environments easily from manifests. Increasingly, these environments are isolated from the host OS by binary packages for dependencies being served by Julia.

I don't want Nix to manage dependencies that Julia/Pkg.jl can handle for me: Julia already handles those excellently and conveniently. I just want Nix to handle the (usually few) system dependencies.

What I would like as a NixOS user is for Nix to provide me with a working Julia 1.5 derivation that provides: julia, a FHS wrapper or something providing compatible versions of the few things that Julia's binary packages need (just libc, libstdc++, libgfortran; I think. see BinaryBuilder.jl docs).

When I say "compatible", I mean that these libraries should work with the BinaryBuilder.jl binaries and with the julia binary provided by Nix.

Not all Julia packages are BinaryBuilder-ised yet, so I'd also want an easy way to specify that I want compatible versions of other packages installed (e.g. Qt, cairo, etc). For packages that need python, I don't know if I'd like Julia or Nix to handle installing and managing that. The simplest approach would be to tell Julia to do it (there is pretty good support in Julia for distributing isolated environments of python + packages + dependencies; better than in pip, imo).

I don't know how doable any of that is in Nix, just describing what a good outcome would look like to me.

[tbenst]

@cmcaine I think this is quite doable. I would propose that we add a new package, julia-env or similar, that is a "batteries included" build that contains system libraries for all common Julia packages. Would this approach work for BinaryBuilder? I haven't used this yet. If not, we could fall back to a FHSuserenv, but I'm reluctant to do so if at all avoidable as there are outstanding nix bugs (ie this) that create seg faults in some heavy data processing workflows.

[cmcaine]

@tbenst I think that is a good proposal and I think it will work with BinaryBuilder packages (e.g. SDL2_jll). In any case, some approach that works with BinaryBuilder packages should be found because those are a critical part of the Julia environment now :)

Some important packages (e.g. GR) still want to build and install their own software dynamically, and that doesn't work well in Nix, but most of those can plausibly move to the BinaryBuilder system (PR for GR here), so I think letting those be broken would be tolerable (and a big improvement over the status quo of Julia on nix).

Perhaps we can get something working quickly by just patchelf'ing the official Julia builds for now? (I don't understand how all of this stuff works!)

[samuela]

Is there anything presently blocking just getting julia 1.5 into nixpkgs? I don't care about BinaryBuilder.jl/the Pkg.jl ecosystem working with nixpkgs, just a good ol' fashioned julia install.

[tbenst]

@samuela yes, see #98043

[cmcaine]

If you just want Julia and don't mind about binary packages breaking you can just download Julia from the official site and patchelf it.

…

On Sun, 4 Oct 2020, 09:32 Tyler Benster, ***@***.***> wrote: @samuela <https://github.com/samuela> yes, see #98043 <#98043> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#91930 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNZA6MPOUOVG3V5VJJOSFDSJAXJRANCNFSM4ONJG3IA> .

[cstich]

@cmcaine I think this is quite doable. I would propose that we add a new package, julia-env or similar, that is a "batteries included" build that contains system libraries for all common Julia packages. Would this approach work for BinaryBuilder? I haven't used this yet. If not, we could fall back to a FHSuserenv, but I'm reluctant to do so if at all avoidable as there are outstanding nix bugs (ie this) that create seg faults in some heavy data processing workflows.

On the julia discourse there was also the idea floated to patch Artifact.jl to patchelf dynamically loaded binary dependencies.

[solna86]

Guix has a Julia package and it works quite well. Perhaps some ideas can be borrowed from there: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/julia.scm

It'd be really useful to get Julia working on Nix. It's the only big outstanding thing that doesn't work and prevents me from migrating some servers in my organization.

[doronbehar]

Guix has a Julia package and it works quite well. Perhaps some ideas can be borrowed from there: git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/julia.scm

I tried to copy their expressions in #98043 and it doesn't work yet, but there's also another seemingly working attempt by @cstich at #101933 .

[doronbehar]

So, do we want anything else now that 1.5.3 is packaged (:green_heart: @cstich) ? I think we:

1. Don't need julia_13
2. Can clean up the expressions a bit - not use the shared.nix file if it's not shared.

I did (2) along with the julia_10 fix in #103783. As for (1) I will need a bit of feedback.

[7c6f434c]

I guess some people like me still use julia_13 via test-disabling overrides, so a bit of time after 1.5 reaches channels for smooth transition could make sense?

[tbenst]

Maybe backport 1.5 to 20.09?

[doronbehar]

OK so I disabled the tests for julia_13 in #104166 and made julia_15 the default julia there as well. All merged pull requests referenced here should be backported.