RFC: adds conditional modules (#6195)

[vtjnash]

conditional modules (described in julep #6195) are executed when a dependency module becomes
available and/or replaced. it has the following syntax:

module A requires B, C.D
  using B
  import C.D
  # do stuff with B, C.D
  # module A will be rerun whenever
  # B or C.D are (re)defined
end

TODO:

• sort & order (re)loading to minimize repeated work and standardize execution order
• handle recursive requires more gracefully
• add to documentation
• add to tests
• add better error message for accessing A.B.* when the dependencies for B are not loaded / add a placeholder module for A.B to reserve the name
• decide whether using A.B should autoload the declared dependencies for B

Possible changes:

• decide whether to allow a literal tuple (spliced in from a macro) for the requirements list, or only
  tuple expressions
• decide on module ... requires ... or module ... when ...
• decide whether ... should be automatically imported inside the module, to avoid repetition

[StefanKarpinski]

What's the advantage of this approach over adding general load hooks that trigger when a named module is loaded? If we end up going with this approach, I think that it really ought to be module A when B, C.D not requires.

[vtjnash]

tl;dr; It's trivial to simulate one approach with the other, so it is not a question of which approach is more general. However, I think that module-based load-hooks are actually more sane & more general.

They both start executing code, so the difference is in how they are intended to be used. I expect that the function approach will require the use eval. Without the eval call, the function cannot barely even access the new module and cannot define functionality based on the new module. Therefore, I expect that the function version would only be used to simulate the module version, as shown in my expected-simplistic-use-case example below:

module A
export myf
myf() = 1
LOAD_HOOKS["B"] = function()
  @eval begin
     using B
     myf(::B) = 2
   end
end
end

vs.

module A
export myf
myf() = 1
module AB when B
     importall ..A
     using B
     myf(::B) = 2
end
end
end

Theoretically, the module version would also be more amenable to caching partial results, resulting in much better optimization (e.g. faster load times) for the module approach over the function approach.

Finally, the namespace segregation of the module version may be helpful to the user, since it makes it harder for the library to provide a constant or function that is only available after the user imports a different module. This is simply speculation, but my expectation is that the module approach forces the user to clearly define what is part of the default interface of the module, and what is only possible when another module is defined. For example, in the LOAD_HOOKS version, calling export my_new_function would have undefined behavior.

Finally, the module method could be smart enough to take a call to import A.AB, and automatically require("B") first

[StefanKarpinski]

Solid arguments. I wanted to do the least invasive thing that would work, but it may well be that we want much deeper integration for this kind of feature.

[vtjnash]

@JeffBezanson can I get your comments? I feel like this is something worth merging before 0.3 (rather than shortly after), since it instantly breaks backwards-compatibility.

[tknopp]

I think we really need the functionality provided here. One question is whether in the example

module A
    export myf
    myf() = 1
    module AB when B
         importall ..A
         using B
         myf(::B) = 2
    end
end

the module AB and the importall ..A is boilerplate or really necessary. But I have to admit that the current version is clear enough when one gets used to it that it is probably not worth adding magic parser things.

[nalimilan]

Sounds a nice solution.

I wonder whether it would be possible to unload modules if one wants to. This is not high priority, but how would it fit in the design if one wanted to allow this in the future? In R in happens quite often that you want to unload/detach a package, e.g. because it conflicts with another one.

[tknopp]

probably this can be done the same way as variable unloading #2385 ;-)

[StefanKarpinski]

I agree that we really need this and it should happen before 0.3 because it will be a major compatibility watershed. It's a good point that the function hook approach can actually be made to require less boilerplate (especially that importall).

Since modules are generally constant, I think the only way to unload things would be to reload everything except for the thing you're unloading.

[mlubin]

We're using Pkg.installed in JuliaOpt for lack of any known better alternatives. This call alone is adding 2.5 seconds to our start up time, so 👍 for any improvements here. Is there any better approach that we can use right now?

In MathProgBase, we need to choose a default package to solve each problem class (linear programming, integer programming, etc.). See https://github.com/JuliaOpt/MathProgBase.jl/blob/master/src/defaultsolvers.jl. How can we use this syntax for that?

[mlubin]

CC @IainNZ @joehuchette

[toivoh]

Would it be enough that a module that is not referenced anymore can be
garbage collected (and perhaps finalized?).

[nalimilan]

IMHO the main point is that exported symbols are hidden. Freeing memory would of course be good, but I'm not sure it would make such a big difference.

[vtjnash]

I envisioned that you would use isdefined at the point of usage (or in __init__ ?)

Using Pkg.installed (or Pkg.dir) seems brittle to me, since the user/sysadmin may install the module elsewhere

Unloading is very hard, since modules often add methods to external functions (and affecting type inference). I expect it is (mostly) possible with this solution (and not possible with the function hook approach). But very time consuming to implement, so unlikely to happen anytime soon.

[mlubin]

In our case we want to try to automatically load one (but not all) of the default packages. That is, if A, B, and C are "installed", then we just want to load A. Loading all of them will kill the startup time.

[StefanKarpinski]

In our case we want to try to automatically load one (but not all) of the default packages. That is, if A, B, and C are "installed", then we just want to load A. Loading all of them will kill the startup time.

This kind of complex requirement is actually an argument for the hook approach. You can add hooks for A, B and C – and each hook unhooks the other hooks. There's no way to have enough syntax to express complex things like that declaratively. But I'm not sure I understand the situation correctly. This is for code that you always want to have loaded when both this and some other module are loaded, whereas what @mlubin is describing sounds like something else.

[mlubin]

Yes, in our case we want to try to load a certain set of packages (with nontrivial precedence rules) ourselves instead of forcing the user to explicitly import them.

[vtjnash]

I'll post some possible example code for your use case later tonight (when I'm at a computer)

If the user explicitly loaded a package, would you always use that, or always default to the highest precedence? Or could the user load multiple and select between them, but defaulting to one of them?

[mlubin]

The way it currently works, we default to the highest precedence. I think this makes more sense so that the behavior doesn't change based on the status of the user's session (which packages might have been loaded before). Users can override the default choice by loading their own packages and explicitly setting a solver to use.

For example:

linprog([-1,0],[2 1],'<',1.5)

uses the default linear programming solver, and

using Clp
linprog([-1,0],[2 1],'<',1.5, ClpSolver())

uses Clp.

At a higher level in JuMP, we have

m = Model()

uses the default solver, and

using Gurobi
m = Model(solver=GurobiSolver())

overrides it to use Gurobi.

Note that "solvers" aren't just the modules themselves, they are instances of a type that implements a certain interface. These instances can also carry parameters for the optimization algorithm, like GurobiSolver(TimeLimit=100), ClpSolver(PrimalTolerance=1e-6), etc.

[vtjnash]

There's lots of ways to structure this, to achieve various results. I've decided to use Winston as an example, since it is pretty similar, but a lot simpler. The goal here is to seamlessly provide multiple backends (Gtk and Tk), and use whichever is available. Unlike the solver interface, on OS X, Tk is observed to segfault if another toolkit (such as Gtk) is loaded in the same process, so it isn't as much of a concern to have the Julia interfaces for them play together nicely.

module Winston
module TkW requires Tk
  import ..Winston
  window(...) = ...
   __init__() = Winston.set_available(TkW)
end
module GtkW requires Gtk
  import ..Winston
  window(...) = ...
   __init__() = Winston.set_available(GtkW)
end
end
function set_available(toolkit)
  global window = toolkit.window
end
function __init__()
  if !isdefined(:TkW) && !isdefined(:GtkW)
    # try to initialize a GUI
    try
      require("Gtk")
    catch
      try
        require("Tk")
      catch
        warn("Could not load the Gtk or Tk GUI toolkit. Interactive plotting will be unavailable.")
      end
    end
  end
end
end

One of the key observations is that, unlike the function hook approach (which needs eval heavily), I never have a need to use eval. This also means that, unlike the function hook approach, Julia can safely cache the independent results of each module loading.

[tknopp]

Seeing this in action is great Jameson. I think you have found a really nice solution for a problem that arises in various packages. +1 to get something like this into 0.3 if its mature (have not tested yet).

[timholy]

Wow, really nice @vtjnash.

[tknopp]

Ping @SimonDanisch as there are also lots of package interdependencies in the OpenGL package family he is working on. I see this for providing optional OpenGL image export without the need to have a hard dependency on Cairo.jl

[MikeInnes]

I'm happy with the extra module syntax if it can cover all cases, but let me play devil's advocate:

It might be useful – and powerful – to have a function hook system with a macro to cover the common case (and hide the boilerplate). For example, say you have an @require macro which creates a function hook to evaluate the given code in the current module:

module A
f(::Int) = 2
@require B f(::B.T) = 4

@require B begin
   # Multiple definitions, submodules etc.
   f(::B.S) = 5
end

# Resolve arbitrarily complex dependencies
foo_loaded = false
bar_loaded = false
@require Foo begin
  foo_loaded = true
  set_default_implementation()
end
@require Bar begin
   bar_loaded = true
   set_default_implementation()
end

# Submodules
@require (Gtk, Winston) module DisplayA
  # Something...
end

end

Some advantages:

• Extra definitions don't have to live in submodules – @require can make sure the code is evaluated directly in module A so that it both has access to all definitions and can define new ones
• But, @require B module A can support anything module A when B does, including method caching, smart loading etc.
• Built on the language rather than into it – modules stay simple, and new syntax doesn't have to be supported. @require is simply part of the Pkg library
• Nothing opaque – it's easy to drop down to more basic load hook functions if you need to get more advanced

Arguably, you get the best of both worlds that way.

[vtjnash]

hiding the eval call in a macro in base doesn't suddenly invalidate my arguments against it

we already need to write an optimization for loading modules. if syntax is part of submodule usage, we get that speedup for free here too. that speedup isn't possible, in general, if the content of a module depends on the presence of external module -- it is already going to be hard enough to detect if any of the functions have changed and reload as needed, without adding syntax that makes it easier to hide these dependencies. if instead, function hooks are directly altering a module based on when another module is or becomes available, I'm not sure I can still do that optimization.

I have no problem with having an LOAD_HOOKS["B"] = function() end functionality, but like any good function, it shouldn't use eval (except for eval :(module end) end), and for consistency it shouldn't be evaluated until the module has finished and __init__ called (which reminds me: I need to fix that order in my pull request). However, in fact, your implementation already would work fine (if you added global in front of your variables) in a local scope, so perhaps there is a use case.

[MikeInnes]

The thing is, if defining functions in a module via eval precludes optimisations, that's already a problem outside of this issue. eval is already used extensively in base, and even if it wasn't I can still write eval(Foo, :(foo() = true)) (and I sincerely hope this isn't going to start breaking everything in future because I depend on it).

What I'm thinking is that you have a process that goes like

1. Load the module
2. Cache the module's definitions
3. Execute the module's load hooks (this might happen more than once as modules are loaded)

That way you solve the problem of a module's methods being dependent on available modules (because during step 1 they aren't).

The only issue with this is that the dependent code itself isn't cached. I suspect that won't be a huge problem, since it will mostly be small amounts of glue code, but if you really want to cache as much as possible you can then use the inner module technique you've suggested.

Dependent inner modules is a neat solution to this, so I'm not trying to put that down – I'm just trying to point out that using load hooks and eval alongside them, when absolutely necessary, wouldn't cause any problems that don't already exist.

[vtjnash]

The thing is, if defining functions in a module via eval precludes optimisations

right, i meant to say /after/ the end statement

[MikeInnes]

Right, which is exactly what I was talking about – as I said, I can (and do) call eval(Module, :(foo() = true)) /after/ the end statement just fine. You're talking about making optimisations at the same time as implying that they are impossible.

You've said yourself that it's trivial to simulate one approach with the other – it must follow that any problems introduced by one are also introduced by the other. If load hooks preclude optimisations, and this PR enables load hooks, this PR precludes optimisations, surely?

[StefanKarpinski]

I really think it's important to get this settled for 0.3 and had tagged it as such, but someone seems to have removed the tag.

[JeffBezanson]

@vtjnash and I did a bunch of thinking about this yesterday, in particular looking for some kind of prior art. In short, it's hard to find a package system that does anything like this, and many people seem to think it's a bad idea. For example see

npm/npm#930
composer/composer#4168

Somebody in the second link is quite scathing on this (perhaps too much).

From reading npm threads, my main takeaway is that while they have similar issues, they mostly assume that "plugins" or "bridge" packages will be separate packages, and the debates are around how to express the dependencies and how to detect which plugins are available. In contrast, we've tried to minimize the number of packages and usings you must do. On the whole, I come away mostly against this feature, for a few reasons:

• It significantly complicates the mental model of dependencies. Instead of just packages and their dependencies, we'd have a predicate thrown in: if X is installed, then Y depends on version z of it. I think this is going to increasingly leak into version resolution in an unpleasant way. If we give the version conflict error eagerly, then you wonder why you're not allowed to use X and Y without the bridge code you might not need. If we don't give an error, then how a package works subtly changes because of versions you have installed. Neither option sounds good to me.
• Currently, code is only loaded if somebody asked for it (or, of course, if code that you asked for asked for it, and so on). Conditional modules add a "phantom requirement" where code is loaded not because somebody asked for it, but because some condition was met (i.e. packages A and B were both installed). The analogy I came up with is you buy a car, and the dealer looks in your closet to see if you own skis, and if you do they install a roof rack and bill you for it without asking. Maybe some people find that convenient, but many people would not approve.
• This is a poor substitute for designing interfaces (as mentioned briefly e.g. in package interactions #2025). People might not like interface-definition packages like StatsBase, but I think they are better than writing O(n^2) bits of glue code. Protocol types might help a lot with this.

I'm not convinced it's so bad to tell users to write an extra using or two. In the long run the extra control and transparency are worth it.

npm has "optional dependencies" (which simply means that failing to build/install the dependency is not fatal) and advocates the equivalent of try; using Foo; end. Examples like #6884 (comment) strike me as using module as an if statement. I think we're better off using top-level if statements like we can now: the first time you load some code is effectively its build process, and then we cache the result. I think simply running through the logic once unless you clear the cache is a simpler model, instead of having load hooks firing at different times. However this should be discouraged in favor of anything else that could possibly work.

[vtjnash]

I think the following analogy is more accurate: I own a car and recently bought a bike rack. Cars do not have a common interface (the trunks are similar, but not identical), nor do bike racks come in unique models for each car. Instead, it came with an instruction booklet that lists cars by make and model and explains how to adapt the bike rack for the trunk configuration of my car. Conditional modules are that instruction booklet.

For prior art, I feel that package managers have historically been disconnected from the language design, and thus lag the language in functionality. I think the closest prior art would be the helper apps for the kernel that make it responsive to environment changes / user action: udev, dbus, etc. (also perhaps subsets of zeroconf / bonjour / mdns; and launchd / systemd behaviors). whereas I think autoconf / cmake are fairly representative of the traditional static solution to this (and lead to one the most problematic elements for package systems: variants) which have traditionally gone with the build-time detection approach (and failing badly at being a robust solution, due to the lack of language support -- and exacerbated by their need to be language agnostic)

I'm against any design that results in the necessity for the user to clear the cache (or conversely, for clearing the cache to cause a behavioral change). If the cache is not able to resolve dependency links correctly, it is failing badly at its primary purpose (which is to transparently accelerate repeated operations). That doesn't mean we need conditional modules, just that we need to implement this dependency link.

[StefanKarpinski]

@vtjnash: so what's your position on this issue then?

[JeffBezanson]

Ok, I agree on manual cache-clearing. That would not be good. We could potentially have if isavailable("Images"), and record which packages a given package asked about to track dependencies... but I'm not sure.

In addition to whether cache-clearing is manual, there's the issue of when it happens. Currently you need to restart and reload a package for us to look at whether a recompile is needed. That might be preferable to packages constantly re-configuring themselves at runtime based on which other packages you load.

There are multiple kinds of use cases here. A major distinction I think is based on the amount of code involved. If a piece of "adapter" code is a large module, it should be a separate package installed manually. But many uses of Requires.jl seem to be adding very small bits of code, e.g. a single extra method definition. Since it's one extra method of a function normally part of the enclosing package, that strikes me as better expressed by an if statement around the definition.

But I really think most of these cases are better solved by dependency injection, as here: JuliaIO/JSON.jl#129, or with interfaces. Consider this tiny method definition: https://github.com/dcjones/Gadfly.jl/blob/51e245da67aa97e8a3cc3ae5731ac1e8e6d8bc95/src/Gadfly.jl#L47 It seems weird to me that we'd need conditional loading hacks and the involvement of a package or module system just to express what seems to be a small fact about Plots.

[StefanKarpinski]

But I really think most of these cases are better solved by dependency injection, as here: JuliaIO/JSON.jl#129, or with interfaces.

Dependency injection works well enough for return types but I worry about type instability and performance implications. Especially for dependencies injected via keyword arguments.

Consider this tiny method definition: https://github.com/dcjones/Gadfly.jl/blob/51e245da67aa97e8a3cc3ae5731ac1e8e6d8bc95/src/Gadfly.jl#L47 It seems weird to me that we'd need conditional loading hacks and the involvement of a package or module system just to express what seems to be a small fact about Plots.

In a sense, the issue here is that the resolution of Reactive.Signal is eager – if it were possible to define this without needing Reactive to be loaded yet, this could just be a normal method definition, which would get resolved when and if Reactive was loaded.

[MikeInnes]

Since it's one extra method of a function normally part of the enclosing package, that strikes me as better expressed by an if statement around the definition.

In most cases the depended-on package is going to be loaded, if at all, after the parent package has loaded, so an if statement doesn't work. e.g. Juno boots and Gadfly is not loaded yet; user loads Gadfly; we define display(::Juno, ::Gadfly) = ....

The current alternatives to this seem to be to:

1. put the definition in Gadfly (via an interface package, making Juno a dependency, or whatever) -- but then package authors have the burden of maintaining support for all Julia frontends, new frontend authors are at a huge disadvantage, and display is generally more mediocre.
2. make Gadfly a dependency of Juno -- fine for one or two packages, less fine as the number of Julia packages we want to support grows.
3. make the user execute using DisplayGadflyInJuno by hand. Same as above.

(1) could work if we had really good ways of expressing display in a frontend-agnostic way, but that's a hard (/intractable) problem in itself. Base's current display system, at least, is a long way from that ideal and probably always will be. (2) and (3) don't remotely scale. I really need an approach that solves this problem, even if that approach is "allow Requires.jl to exist in peace".

[JeffBezanson]

Isn't the design already unscalable if new code needs to be written for every combination of graphic generator and graphic displayer? I guess the reason it might not be is that the new code needed is very small, like I see here:

@require Gadfly begin
  displayinline(p::Gadfly.Plot) = DOM.div(p, style = "background: white")
end

I find it hard to believe that there is no way to define an interface that allows this one line to move into Gadfly.

[StefanKarpinski]

This is another example where evaluating Gadfly.Plot lazily would solve the problem.

[MikeInnes]

There's a middle ground between complete coupling of generators/displayers and complete decoupling. Right now we aim for things to be generic where possible and pave over the gaps where necessary. If it's not necessary too often then it scales well enough.

Again, I'm not arguing that this stuff is technically impossible without conditional loading, just that the large amount of extra coordination needed between package authors will slow things down a lot, among other disadvantages.

Stefan's right that lazily-loading types / methods would cover this use case, if it's possible to implement that.

[mauro3]

Lazy type evaluations were touched upon on julia-dev two days ago: https://groups.google.com/d/msg/julia-dev/U1HxsML0w4Q/jcOpk1eGAwAJ

The context was somewhat similar with slimming of Base but keeping the un-used bits around.

[JeffBezanson]

This is intriguing, because you'd really rather have those definitions always be part of Jewel.jl, but only get called if an object of one of those types arises. I imagine in other dynamic languages you might write code like if string(typeof(x))=="Gadfly.plot", which is the same except using a string as the lazy version of the type, and using an if statement instead of dispatch.

We would need to figure out the evaluation rule. When we see Gadfly.plot, we don't know why Gadfly isn't defined: is it because we're waiting for a package to be loaded, or because of a typo? Using a separate evaluation rule for method types seems pretty fraught.

Something like header files comes to mind. What we need to insert a method is just the top line of a type definition: its name, parameters, and supertype. Of course we could also use something like this for mutually-referencing type definitions. However I don't yet see a natural place to put such things when working across packages.

[MikeInnes]

I like this train of thought a lot. It would make a ton of sense to write something like

extern DiskData.BigVector{T} <: AbstractVector{T}

foo(::BigVector) = ...

Something like this could also address some of the issues with sharing generic functions across packages when they're not defined in Base.

[JeffBezanson]

I think that could work. Of course it's annoying to need to duplicate the declaration, but it doesn't increase the coupling too much since in these cases you already have to know about the other package's type.

[iamed2]

+1 for extern, I've had to do some organizational gymnastics to get this sort of thing to work even within one package with multiple modules.

[StefanKarpinski]

extern DiskData.BigVector{T} <: AbstractVector{T}

Does an extern declaration even need to provide that much information?

[JeffBezanson]

With that amount of information we can handle method definitions as normal. Without it, we would need to do something much more complicated along the lines of deferring the method insert until the type is fully defined.

[shashi]

I think extern is a nice and simple solution to the main problem here. 👍

[StefanKarpinski]

Some combination of extern and dependency injection seem to address all needs that have come up.

[shashi]

I suspect

extern X.foo
X.foo(::MyType) = ...

Would just work in light of jb/functions..?

[mauro3]

Are lazy types still on the table? If so, this would also resolve being able to declare mutually referential types?

type A
   b::B
end
type B
   a::A
end

Presumably, using extern and having the two types in different modules would work.

[JeffBezanson]

@mauro3 Yes! I think this approach is viable. Types already exist in a partially-constructed state very briefly, to allow field types to depend on their enclosing type. We just need to allow this state to persist longer.

We can also introduce provisional modules, which can only contain extern (or forward) declarations and cannot have code evaluated in them. When an actual module with the same name is defined, the provisional module becomes actual.

[StefanKarpinski]

Possible syntax for provisional modules:

extern module Foo
    # type declarations and function signatures go here
end

[StefanKarpinski]

Closing in favor of #15705.