Move FFTW to a package

[tkelman]

In response to a question from @jongwook at #17000 (comment) (I'd really like to keep that issue focused specifically on its title, of FFTW threading - if this issue gets resolved before that one does, then we'll move issue #17000 to the package)

is it related to the GPL of FFTW

Yes. The goal is to have a default Julia distribution that doesn't contain any GPL components (aside from libgit2 and gcc runtimes which have linking exceptions, and possibly a few "mere aggregation" programs that aren't linked into the same address space), as soon as 0.6 if we can come up with a minimally-disruptive way of moving FFTW and SuiteSparse out to packages.

in favor of other FFT implementation like MKL

Hard to say. I believe MKL has a mostly-FFTW-compatible FFT API, but it's not exactly equivalent so might not be a complete drop-in replacement in all respects. Either we make an FFTW.jl package that is capable of handling either the GPL FFTW library or the MKL version based on user choice (or whether the version of Julia being used is detected to already be using MKL for BLAS), or we separate the bindings for the GPL FFTW and the MKL FFT API into independent packages.

A pure-Julia MIT-licensed FFT was implemented in #6193 but never merged. I personally think that code should have been published as a Julia package from the start, so it could have been used for the past 2 years rather than sitting as a closed PR. I believe @yuyichao occasionally rebases a branch up to recent master but I'm not sure when he last did so.

[oxinabox]

Are you saying that there should be no FFT in Base?
The other "technical computering" languages have FFT in their standard libraries,
(Being Matlab/Octave, R, Python-with-numpy, Mathematica).

Removing FFT from Base, would be "Batteries Not Included".
Which is arguably a bad thing.

On the other hand it would be a move towards "Small Standard Library, Big Ecosystem"
which is arguably a good thing.

[tknopp]

The most simple thing would be moving FFTW into a package and integration of #6193 into Base. The question about the standard library is a little bit independent. Of course #6193 would fit very well into a package of the standard library but as long as there is not infrastructure for that Base is the correct location.

The issue that I see are:

• no rfft support in WIP: new DFT api #6193
• no dct/dst support

If these are critical is of course up to discussion.

[tkelman]

IMO we shouldn't have an fft in base any more. fft's should not be a dependency of all other code written in the language. We're building out the infrastructure for shipping distributions of Julia that include default packages, which this could be if you don't mind the license (or a pure Julia FFT could be if you do).

[timholy]

@oxinabox, Julia is evolving from "technical computing language" to "general purpose language which has awesome support for technical computing." In this long run, this will considerably broaden Julia's appeal. Moving this out follows the Python model---numpy is not part of Python, either. I agree with @tkelman, I don't see any reason that fft should be defined in Base at all.

[tknopp]

I am in principle with @tkelman here but we would seriously need a way that newcomers know how to get the fft. A workaround could be JuliaExtras package that would bring all the batteries in that we could then remove from Base. The user then would just need to remember to add JuliaExtra. The package would only be a metapackage that reexports the actual package (e.g. FFTW.jl)

[nalimilan]

Before repeating a discussion regarding moving feature to separate packages which has happened elsewhere, see #5155. Let's keep this issue focused on the case of FFTW.

[tknopp]

Ok, sorry that I pointed to the more general issue. Then I might ask more directly: How should the deprecation policy should work for the FFTW functions in base? Deprecate in 0.6, remove in 0.7?

[oxinabox]

@tkelman I agree. (I also agree with the reverse, see my posting both.).
I just wanted to be clear as to if the goal of this issue was the get rid of the the GPL Liscensed FFTW,
or if the goal was to get rid of FFT all together from base.
Thanks for clarifying.

[stevengj]

This may be a bit premature; I think we need to implement a robust way to ship and test "batteries included" Julia distros before we start pushing a lot of things out of Base.

[tkelman]

The license issue needs fixing now, it's a major impediment to Julia usage in a number of areas. "Batteries included" Julia distributions exist already, that work is not happening in this repository yet but it's well underway.

[tkelman]

Other pieces of code have been moved out of base to packages with a "this function has been moved to package xyz" error. This will be very much in line with that, although the package will be a bit larger and more complicated than past examples.

[gabrielgellner]

I really hope this movement to make tones of tiny packages for, what I consider, basic science stuff doesn't fracture the package ecosystem. If we have lots of little fft packages, with lets say different licenses, api, etc, and then some packages begin to build on it is going to make using julia as an end user suck. If this ships before we can just do something like using SciBase that grabs all of these basic parts that have been removed, and that has the blessing of the major powers that be I think it will be really bad for julia.

[stevengj]

(At least for FFT functions, the dft.jl interface in Base is set up so that lots of different backends can be provided, with the same fft(x) and plan_fft(x) interface.)

[ChrisRackauckas]

See this discussion: https://groups.google.com/forum/#!topic/julia-users/ug5Jh6y5biA.

I think a good way to go forward is to start making some metapackages held by the Julia orgs. For example, FFT could move out of Base into JuliaMath, and there can be a standard Math.jl which simply re-exports the "Julia standard math library" using Reexport.jl. This would have agreed on packages (with version requirements?) for a specific domain. JuliaMath could have Math.jl metapackage, JuliaStats could have a Stats.jl metapackage, etc.

I think this is better than just lumping them all together into a SciBase because "Science" is huge: the ML people will want the machine learning stack in there, there's a whole statistics stack which replicates base R, if you want to replicate MATLAB/SciPy then you'd want some differential equations solvers which are large packages themselves, arguably PyCall and everything associated is "standard Julia". I think using Math, Stats is both succinct and informative to others if you share your script, and move the discussion of "standards" to a more domain specific level.

Does anyone see any design issues that may arise due to using Reexport here (other than the fact that it's a personal repo)? Or does anyone else have a good design?

[tknopp]

@ChrisRackauckas: this is actually the purpose of #5155. One just has to find a clever grouping scheme which isn't that easy. I would directly say that Math is a pretty bad name since it is much to general. Stats or Statistics is much better. FFT would IMHO fit good into a SignalProcessing metapackage, which would also hold the entire Images packages.

[ararslan]

I get the feeling this isn't going to happen in the 0.6 release time frame?

[ViralBShah]

I don't think this is going to be as difficult as Rmath, but someone needs to own it.

[nsmith5]

I think support for basic numerical routines is important in Base. Is there anything wrong with a "python + numpy" model for Julia? While I can understand (but personally disagree) with moving GPL code out of Base, I don't see why we shouldn't have pure Julia implementations for the numerical basics (linear algebra, special functions, fft etc).

It seems a little bizarre to have the kind of linear algebra and special function support we have in Base without fft no? Or is there a plan to push these out of Base as well?

[yuyichao]

Or is there a plan to push these out of Base as well?

Yes, as long as we can. We'd like to put them into package but make sure they can be loaded easily/loaded by default so it's still easy to use.

[nsmith5]

Ah ok, fair enough.

[StefanKarpinski]

We've also already moved special functions out of Base and would also like to move much of the linear algebra code out.