Move LinearAlgebra to its own repo

[ararslan]

Note that SuiteSparse is only the set of solvers for sparse arrays, not the entire sparse array functionality, though a case could be made to remove that from Base as well.

• SuiteSparse
• SparseArrays
• LinearAlgebra

EDIT by @ViralBShah: I am updating this issue to be about separating the linear algebra capabilities into separate repos rather than being excised, which is a doable first step for the 1.x releases.

[ararslan]

A challenge in doing this is setting up the resulting package to fetch the library. For most things we can just use BinDeps to get binaries from a system package manager if they're available, otherwise build from source. However, system package managers vary widely in their offering of SuiteSparse, and older versions can't build as a shared library by default, if I'm not mistaken. Further, WinRPM doesn't have a package for SuiteSparse, and typically the requisite C compiler for libraries like this isn't available by default on Windows. Thus I'm not sure what the best solution for this would be. Any suggestions would be welcome.

[jebej]

Wouldn't it be possible to compile with gcc from WinRPM?

[ararslan]

Could be, I was thinking about that as well.

[jebej]

Some packages have started manually calling gcc.exe to do that (eg Clipper.jl). It would be nice to integrate gcc into eg BinDeps to automatically handle compilation on Windows. At that point we will then be able to rely on there being a compiler on Windows.

[ararslan]

Though it may work in some cases, that doesn't really seem like a good general purpose solution. There are also licensing concerns with distributing GCC, since it's GPL.

It's likely that for SuiteSparse I'll end up using Travis and/or AppVeyor deployment to GitHub releases so that no local compilation will be necessary; the appropriate binaries will be installed along with the package.

[jebej]

Is that the reason for the thumbs down?

[ararslan]

¯\_(ツ)_/¯ It's hard to know others' concerns if no one voices them. As such, I can't speak for the others, distributing GCC with BinDeps (or some similar approach) just doesn't seem like the right general purpose solution to me.

[jebej]

I didn't mean to suggest to distribute gcc, but to use WinRPM gcc, which is what the aforementioned Clipper.jl is doing.

[jebej]

It seemed like a good solution since I see sometimes see people mentioning how Windows not having a compiler installed by default is limiting.

[ararslan]

I'm not a BinDeps expert so I can't really speak to all of he pros and cons of various approaches, but it's not a bad suggestion in my opinion, and I do think it warrants further discussion. I recommend opening a discussion on the matter on Discourse so that hopefully others can weigh in without this issue getting too off topic. 🙂

[lobingera]

@jebej I voted with 'down' in relation to some story some time ago when it was decided that 'julia is not a distribution system for git'. The same would apply for 'BinDeps is not a distribution system for a c-compiler'. BinDeps has exactly the one job to support binary dependencies mostly in the form of shared libraries that can be accessed by ccall. I can see the need of a general BuildSharedLibrary.jl that can build from sources a library - maybe included into package that are closely connected to a compiler (like Clang.jl and Cxx.jl). So i voted against including a build system into BinDeps, but not against providing a build system.

[tkelman]

WinRPM can give you a compiler executable but it can't give you an environment that can run configure scripts or makefiles.

My personal plan is to split and replace BinDeps with two pieces, a BinaryBuilder.jl that makes it easy for package authors / maintainers to automatically build portable binaries across the major platforms (then once up and running, make rebuilds and version updates simple to automate on CI and upload to github releases), and a simple-as-possible BinaryDownloader.jl for users at install time that just downloads and extracts these tarballs. I'm working on writing up the design and plan in a Julep and putting together the complicated pieces. It should look somewhat like a decentralized version of conda-forge (without the conda). This obviously isn't ready yet though, so for now it's best to get the shared library build going from source on Linux, and prepare a dll download for Windows. Those dll binaries can be cross-compiled by the package maintainer if you don't have access to Windows.

[lobingera]

@tkelman

I'm working on writing up the design and plan in a Julep

Is this visible already, i didn't find it?

[tkelman]

Not yet. A bunch of notes and thoughts and a few scripts, very disorganized still.

[jebej]

Thanks for the explanations. It seems we all were suggesting pretty much the same thing (BuildSharedLibrary.jl or BinaryBuilder.jl). I had proposed using WinRPM's gcc in BinDeps.jl since its purpose reads "Tool for building binary dependencies for Julia modules".

[ararslan]

@ViralBShah moved this to the stdlib, so this is effectively done. We can move it out of the repo entirely later.

[tkelman]

That should be tracked, given it's the last dynamically loaded GPL dependency without a linking exception

[ViralBShah]

Given that all of stdlib will need to be moved out anyways, I think this will happen. Needs BinDeps2. I guess we could leave it open but move it off milestone.

[ViralBShah]

A serious effort went into JuliaLang/julia#27638. The main reason to hold back was that SuiteSparse by itself was too deeply tied to SparseArrays. So, both would need to be moved out together. Even moving both out is likely to have an issue since @KristofferC mentioned somewhere that there are type piracy issues that will need resolving.

[ViralBShah]

I am reopening this issue to reopen the discussion of moving SparseArrays and SuiteSparse out into their own packages. I suspect this is good to get done early in the 1.7 release cycle.

[oscardssmith]

One thing to think about if we want to do this is whether we should have banded matrices (and or other structured matrices) in the same repo.

[StefanKarpinski]

Once it's not a stdlib anymore, folks can do what they want, but it seems like a mistake to me if they can technically be kept separate, they should be.

[ViralBShah]

Based on some recent discussions, a first step is to move LinearAlgebra, SparseArrays, and SuiteSparse to their own respective repos. They will still continue to be stdlibs, but in separate repos. That will greatly simplify things in the julia repo in terms of issue/PR traffic, reduce CI time, and also allow us to start testing the usage of MKL and improving the robustness of our linear algebra stack.

@KristofferC pointed out that there is some type piracy in the sparse packages that makes this a bit more involved. But it would be great to get this project underway.

Eventually perhaps in Julia 2.0, we can actually move more stdlibs out into packages - but the first step is to move them into separate repos.

[ViralBShah]

SuiteSparse moved in JuliaLang/julia#40770

[jlapeyre]

I needed to generalize SparseArrays a bit for a project. I copied the code from the Julia repo to a separate repo: https://github.com/jlapeyre/SparseArraysN.jl

There is a note in the README on type piracy. I flagged and commented out methods from this separate-repo version that are probably in the sysimage, causing warnings to be printed when SparseArraysN is compiled.

These flags (The word DISABLED in comments) might save a few minutes for someone trying to fix the type-piracy issue.

Moving SparseArrays to a separate repo, and in general disentangling it, will facilitate these experiments. See also JuliaSparse/SparseArrays.jl#72

[ViralBShah]

Any volunteers for moving SparseArrays.jl into a separate repo?

[jlapeyre]

Any volunteers for moving SparseArrays.jl into a separate repo?

@ViralBShah I moved SparseArrays.jl from stdlib to a separate repo: SparseArrays.jl

I disabled a small number of tests using @test_broken. As far as I can see, the problems with inadvertent type piracy have been fixed (in the stdlib version) since I last commented above.

I migrated files from the master branch today. As you can see in the README.md, the migration required almost no work.

[ViralBShah]

With JuliaLang/julia#43127 done, we should be able to move SparseArrays.jl out of the julia repo into its own separate repo. It will still continue to be in they system image.

[ViralBShah]

@DilumAluthge Adding you to this discussion, since you may be interested.

[DilumAluthge]

I'm always a fan of moving stdlibs to their own repos!

[ViralBShah]

Thanks @KristofferC. Now we are left with LinearAlgebra! I suppose we let the dust settle on sparse and then see if it's worth doing for LinearAlgebra - and perhaps that only happens in 1.9.

[waldyrious]

Cross-referencing JuliaLang/julia#43813 ("move SparseArrays to the external repo https://github.com/JuliaLang/SparseArrays.jl").

[vtjnash]

Doesn't look like this requires an open issue to be done

[ViralBShah]

Related: JuliaLang/julia#51904