separate packages for MOI wrappers?

[mlubin]

Since we're about to write a bunch of MOI wrappers, now would be a good time to reconsider the current structure of the solver wrappers. I propose that we move all of the MOI wrappers into separate packages, like GurobiMOI MathOptInterfaceGurobi, CPLEXMOI MathOptInterfaceCPLEX, CbcMOI MathOptInterfaceCbc, etc. The reasons are:

• We would like solver companies and developers to become more engaged with the Julia interfaces, and it will be easier to do so if we don't require them to maintain MOI wrappers as well.
• There is potential for contention when non-JuliaOpt developers want to include something in a solver wrapper (e.g., WIP: adding cplex annotations to be used for bender decomposition CPLEX.jl#125). With MOI wrappers in a separate package, the logical separation of what should go where will be more clear.
• It will be easier to write the new wrappers starting from fresh empty packages.
• We'll finally have consistency with GLPK.

It's a bit uglier to say using MathOptInterfaceGurobi versus using Gurobi. JuMP can get around this by providing a @usingsolver macro that translates @usingsolver Gurobi, Mosek into using MathOptInterfaceGurobi, MathOptInterfaceMosek.

CC's for those not watching this repo: @ulfworsoe @tkelman @odow

[chriscoey]

I'm all for modularity.

For the names, I'd vote for prefixing MOI rather than suffixing - easier to read because it distinguishes Gurobi and MOIGurobi immediately rather than at the end, and sounds more like "math opt interface for gurobi"

[mlubin]

We'll have to see what we can get past the METADATA guards

[tkelman]

Don't love the acronym naming, explaining to people they need to Pkg.add("GLPKMathProgInterface") to use GLPK with JuMP never made any sense and adding more acronyms to that makes it more confusing. Not going to stop you from doing this if you're convinced it's a good idea, but it sounds like more bookkeeping work and a less obvious system for end users to navigate for not much benefit. (Non-solver) Maintenance takes effort when Julia or the solver-independent interface layer changes. The former will hopefully slow down after 1.0. For the latter, it might make sense to do things this way for interim development purposes to keep them decoupled as the new API's evolve, until they're all ready to release and switch over. Doesn't feel right to me long-term though.

[mlubin]

We can make it more explicit with MathOptInterfaceGurobi. I don't think the concept of a separate package having the "JuMP" interface to a solver is inherently hard to navigate or confusing as long as we're consistent. On the JuMP side we can have a special macro for loading packages, as I mentioned, and perhaps JuMP.addsolver("Gurobi") which just translates to Pkg.add("MathOptInterfaceGurobi").

[joaquimg]

I vote for MOI.addsolver("Gurobi") and @loadsolvers Gurobi to be exported by MOI

[mlubin]

Question: what if we have a pure-julia solver whose natural interface is MOI? Do we still need a second package? Does Pajarito also need MathOptInterfacePajarito?

[joaquimg]

Without that name how would addsolver and @loadsolvers work?
But since MOI is a natural interface for Pajarito, MathOptInterfacePajarito does not look good for me...

[mlubin]

Julia doesn't have a nice way to deal with plug-ins that register themselves on installation, afaik. We could maintain an explicit list of solver names and package names in MOI.

[mlubin]

We've gotten pretty far without ever having a list of solvers, but it's not that unreasonable to do. We could also use this list to store the capabilities of each solver so that we could make recommendations based on the problem class.

[joaquimg]

Why would we need a plugin that registers itself?
I don't like the list, because if people start creating a lot of experimental solvers it would add to MOI extra information that is not well maintained or it will add a barrier for these experimantal solvers

[mlubin]

To make @usingsolver work we'd need a mapping of solver names (and capabilities) to packages. Either we have a centralized list in MOI.jl or we let each solver register itself when it's installed. There's no barrier for experimental solvers, you'll be free to manually load the package and use XXXSolver().

[ccoffrin]

Assuming more hybrid solvers like Pajarito are developed, there could be significant maintenance pain for each new solver to make a PR to MOI. Right now this is basically handled by the Julia Pkg manager. But if there is no easy alternative, this is fine by me.

[mlubin]

If we're at the point where we can't keep up with PRs to add new solvers to the list in MOI, then either we've been very successful or we've abandoned MOI :)

[ccoffrin]

I had the very successful outcome in mind 😄

[mlubin]

So I'll summarize the conclusion as follows:

We let the MOI interfaces be implemented in separate packages as appropriate. We should probably do this commercial solvers like Cplex and Gurobi for the reasons I listed above. For Cbc, Clp, ECOS, etc maybe there's not much of a reason to split off the MOI wrapper yet. Pajarito shouldn't need a separate wrapper package.

MOI has a dictionary of solver names to solver packages and implements @usingsolver and MOI.addsolver() so that people can add solvers with the short names. In the JuMP documentation, we recommend these methods over using and Pkg.add() for consistency.

Anything else?

[joehuchette]

I'm in agreement with @tkelman, I don't see how the seemingly marginal or hypothetical gains from this proposal justify writing our own mini package manager in JuMP.

[odow]

What about having JuMP/MathOptInterface "claim" the short versions of the names. Move the C interfaces to GurobiBase, ClpBase etc. Then JuMP users still go "using Gurobi".

[mlubin]

Gurobi can essentially claim Gurobi.jl whenever they want by trademark rights. If we're going to do restructuring, it should be in the direction of making this less disruptive when it happens, not more disruptive.

[mlubin]

I don't see how the seemingly marginal or hypothetical gains from this proposal justify writing our own mini package manager in JuMP.

Calling it a mini package manager is a bit of an exaggeration. I'm imagining:

solverdict = Dict(:Gurobi => :MathOptInterfaceGurobi,
                  :CPLEX => :MathOptInterfaceCPLEX,
                  :Pajarito => :Pajarito)
macro usingsolver(args...)
    # using solverdict[args[1]], solverdict[args[2]], ...
end
function addsolver(s)
    Pkg.add(solverdict[s])
end

[IssamT]

After reading a couple of time this thread and after the explanation provided by Miles, I think that this is a good idea.

Indeed we are still facing an issue with BlockDecomposition.jl dependency towards CPLEX.jl despite the current - not very clean - state of CPLEX.jl which defines two functions that are wrappers to the C interface but that are not used to implement any function in MathProgBase.

Having the C Wrappers separated from the MOI interface will be a clean way to solve our problem.

That said I agree with @tkelman : Pkg.add("GLPKMathProgInterface") to use GLPK with JuMP is not intuitive for the users. Why don't we call the MathOptInterface simply JuMPInterface.jl (it is really a julia math programming interface). And even if someone wants to create his own modeling language, no one denies him from using JuMPInterface.jl as a sublayer without using JuMP.jl

Solver packages like Gurobi.jl could be split then to the C wrapper Gurobi.jl and the JuMPInterface implementer named something like JuMPGurobi.jl.

The user would be less confused writing

Pkg.add("JuMP")
Pkg.add("JuMPGurobi") 

About pure julia solvers like Pajarito, I am not sure about the answer since there is no C wrapper but it would be nice to keep the consistency.

I remember that when I first used JuMP and liked it, I did a quick search on github "JuMP jl" to see all packages and Prajito wasn't listed among the results. I only discovered it much later. It is still the case today https://github.com/search?p=1&q=JuMP+jl&type=Repositories&utf8=%E2%9C%93
Had it been named JuMPPajarito, I would have found it faster.

Since we are talking about naming, it would be nice to have a clear convention for all stable packages that are part of juliaopt, including JuMP upper layers (that should be different from the sublayer convention). I've seen a couple of questions about it in the past (e.g. jump-dev/JuMP.jl#739)

[blegat]

I find it cleaner to prefix it with MOI rather than JuMP. Other packages than JuMP are using MathProgBase and will use MOI and prefixing it with JuMP wouldn't send the right message. MOI is a thin layer that enables you to write solver independent code while JuMP is much more than that.
I agree that it would be nice to have a clear convention but having either SolverName.jl or SolverName.jl and MOISolverName.jl seems easy to grasp.
@tkelman I understand that acronym should be avoided in general but in this case isn't reasonable to use MOI instead of the long MathOptInterface ? Anyone starting to use JuliaOpt will know the acronym immediately.

[tkelman]

Anyone starting to use JuliaOpt will know the acronym immediately.

That's true for people writing JuliaOpt packages, not for people using them. To most users MathProgBase or any replacements are just words that happen to scroll by when they first install JuMP (or extensions, or Convex.jl I suppose) and they never see or think about them ever again.

Module names tab complete, having a name that's more self explanatory and discoverable is worth the few extra keystrokes.

[IssamT]

I think that most people would accept the acronyme in HTTP.jl because HTTP is known for people other than those writing this package. unfortunately, we can not use such argument: while OR people use MIP often for mixed integer programming, MP isn't a known acronyme for math programming.

However, it is already accepted for JuMP. I think both names should be close somehow, and JuMP with MathOptInterface In my opinion aren't.

@blegat I see your issue with JuMP prefix. how about JuMPInterface suffix?

JuMP users will write:

using "JuMP", "GurobiJuMPInterface"

SweetModeling users will write:

using "SweetModeling", "GurobiJuMPInterface"

EDIT:
I am kind of saying that if the acronyme MP is not common, JuMP is well-known by OR users of julia. Come on! I knew Julia thanks to JuMP :)

[mlubin]

I can see the marketing argument for calling this package JuMPInterface instead of MathOptInterface. Just because it's a simple and standalone interface doesn't mean that it needs a separate and not easily recognized name. Needs some more thought though.

[mlubin]

I remember that when I first used JuMP and liked it, I did a quick search on github "JuMP jl" to see all packages and Prajito wasn't listed among the results

We have package listings at juliaopt.org and http://www.juliaopt.org/JuMP.jl/0.17/installation.html#getting-solvers, so I'm not really convinced by the argument that anything related to JuMP should have JuMP in the package name. Pajarito wasn't listed under juliaopt.org for a while because it was not ready for users.

[chriscoey]

Convex.jl (and future packages besides JuMP) uses MOI. Calling it JuMPInterface and calling the solver wrapper packages JuMPInterfaceXXXX may briefly make things seem simpler for first-time users, but ultimately it obscures the structure of the ecosystem. Users should go and see the diagram on the front page of JuliaOpt, and the naming conventions should make sense in the context of that fundamental structure. Perhaps that diagram should appear in the readme of every package it mentions. If a user understands the ecosystem, they are less likely to ask basic questions, and more likely to become a developer.

[chriscoey]

If we really wanted to make things simple for first-time or one-time users, we would have a JuMP function that sets up the desired solver, as Miles proposed. I think it's a little melodramatic to call it a package manager.

[IssamT]

We have package listings at juliaopt.org and http://www.juliaopt.org/JuMP.jl/0.17/installation.html#getting-solvers, so I'm not really convinced by the argument that anything related to JuMP should have JuMP in the package name. Pajarito wasn't listed under juliaopt.org for a while because it was not ready for users.

I agree that it is a weak argument and shouldn't be one of the main arguments for choosing the package names. I m just saying that sometimes when you are ready to face and fix bugs rather than rewriting a new package from scratch, you may accept to use an existing code even if it is not ready to be listed yet. And for that, you can be happy with what github is offering :)

[IssamT]

Thumbs up for http://www.juliaopt.org/ and http://www.juliaopt.org/packages/ listing all the nice JuMP solvers and extensions !

[mlubin]

@chriscoey is right that renaming MOI to JuMPInterface obscures the structure of the ecosystem. Our most recent design decisions in JuMP and especially MOI have favored clarity over superficial ease-of-use that eventually leads to confusion, and I don't see a reason to break that trend. So that's a no from me on JuMPInterface.

Following the clarity argument and the Julia package naming conventions, we shouldn't have the abbreviation MOI in package names.

[IssamT]

Thanks for the clarifications @mlubin. MathOptInterface is then, in my opinion, a good name for this package.

We agree nevertheless that this name is a generic name and not related to any other package within the Juliaopt organization, nor it is related to the name of the organization. So this is a claim that for now, MathOptInterface.jl is defining the standard way under julia for interfacing the existing solvers.

In my opinion, I guess this won't be an issue for anyone, since all solver interfaces that have already been used for a while under julia, are part of Juliaopt organization and use all the unique interface.

If another organization (eg ILOG) wants to make their own interface, they will be the ones required to add the ILOG prefix in the package name to avoid confusion.

Assuming that we accept this, then I don't see a reason to add MathOptInterface to SolverName package. If someone ever claims SolverName.jl a solution, in my opinion, that would make sense is to prefix it with the name of the organization juliaopt or with the name of a known package within that organization.

Therefore I upvote for the suggestion of @odow for having SolverName.jl be the SolverName implementer of MathOptInterface and SolverNameBase.jl be the pure C wrapper.

[joaquimg]

As @mlubin said

Gurobi can essentially claim Gurobi.jl whenever they want by trademark rights.

The same holds for CPLEX

[mlubin]

How about we don't do any unnecessary package reorganization until we see Pkg3. I've been asking for package namespaces for years now... If the name of the package were JuliaOpt/Gurobi.jl instead of Gurobi.jl it would be much less objectionable to keep the MOI wrapper there. At the moment I don't think we have the right tools to properly address this issue.

[chriscoey]

Probably reasonable... Is there an ETA on Pkg3?

[mlubin]

Julia 1.0?

[joehuchette]

Namespacing seems like the correct solution here.

[mlubin]

I consider #27 as a resolution of this issue for the time being.