[MOI] Set Gurobi-specific attributes

[odow]

As raised in this Discourse post, we should implement Gurobi-specific MOI attributes to allow the getting and setting of attributes from JuMP.

A sketch of the idea looks like:

struct ConstraintAttribute{T} <: MOI.AbstractConstraintAttribute
    name::String
end

function MOI.set(
    model::Optimizer, attr::ConstraintAttribute{Int}, 
    con::MOI.ConstraintIndex{MOI.ScalarAffineFunction{Float64}, T}, val::Int
    ) where {T}
    row = model[con]
    set_intattrelement!(model.inner, attr.name, row, val)    
    return
end

function MOI.get(
    model, attr::ConstraintAttribute{Int}, 
    con::MOI.ConstraintIndex{MOI.ScalarAffineFunction{Float64}, T}) where {T}
    return get_intattrelement(model.inner, attr.name, model[con])
end

MOI.set(model, ConstraintAttribute{Int}("Lazy"), con, 3)
MOI.get(model, ConstraintAttribute{Int}("Lazy"), con) # --> 3

Here is the list of Gurobi attributes
http://www.gurobi.com/documentation/8.1/refman/attributes.html

Here is the list of Gurobi.jl attribute getters and setters
https://github.com/JuliaOpt/Gurobi.jl/blob/master/src/grb_attrs.jl
Note that different functions need to be called based on the type of the attribute (e.g., get_intattr, get_dblattr, get_strattr).

Here is the list of abstract MOI attributes
https://github.com/JuliaOpt/MathOptInterface.jl/blob/06216dc6907b5ec76390c8380e4655065c83ea22/src/attributes.jl#L16-L35

We should repeat the Julia code above for the other types of attributes listed in the Gurobi documentation, defining new structs as needed.

We probably need to check that the string attribute names match the right int/dbl/str type.

Any code also needs tests and documentation.

For anyone working on this, make some small changes, and then open a pull-request. It doesn't need to be complete. We can workshop the changes before you spend too much time trying to implement everything.

Also, if you have a better idea for how to implement this, say so.

[prashantc371]

Hi,
I would like to know the equivalent steps for the Lazy constraints generation that is described in the "Lazy Constraints" section of http://www.juliaopt.org/JuMP.jl/0.18/callbacks.html#lazy-constraints? I am using a Gurobi Solver. But I would like to see the place where I can

Any pointers will be highly appreciated.
Thanks.

[odow]

Please provide a link if you post in multiple places: https://discourse.julialang.org/t/setting-constraints-as-lazy-with-jump-0-19-and-gurobi/24516/7?u=odow. I have replied on Discourse.

[fa-bien]

2.5 months later I finally have time to work on this! I am trying something based on the suggestion above but it is not working, at least not the way I am trying. My goal is to get this to work after a model is created using model = Model(with_optimizer(Gurobi.Optimizer)) as suggested in the discourse thread. It already works when creating the model in direct-mode (as mentioned in the discourse thread) but the whole point here is to not do that.

Gurobi.set_intattrelement!() needs a Gurobi.Model, and an index for the constraint in the corresponding Gurobi.Optimizer. I can obtain one with model.moi_backend.optimizer.model, but this Gurobi.Optimizer has 0 variable and 0 constraint, therefore it is impossible to access the constraint that I want to set as lazy. Here is a small example:

model = Model(with_optimizer(Gurobi.Optimizer))
@variable(model, x >= 0)
@constraint(model, c, 2x >= 1)
grb_model = model.moi_backend.optimizer.model
grb_model[index(c)]

The output is a KeyError, so I assume that I am not looking in the right place. Any idea where I can access the correct Gurobi.Model?

[odow]

We can forget about JuMP for now. It is already possible to set MOI attributes. See, e.g.,

using JuMP, Gurobi
model = Model(with_optimizer(Gurobi.Optimizer))
@variable(model, x >= 0)
@constraint(model, c, 2x >= 1)
MOI.set(model, MOI.ConstraintName(), c, "my_c")
@assert name(c) == "my_c"

What we need to do is define the equivalent of MOI.ConstraintName for other things. Then this would work

MOI.set(model, Gurobi.ConstraintAttribute("Lazy"), c, 1)

The rough steps are as follows. (All the code should go in src/MOI_wrapper.jl.)

Define ConstraintAttribute, VariableAttribute, and ModelAttribute:

struct ConstraintAttribute <: MOI.AbstractConstraintAttribute
    name::String
end

Here is the documentation for the MOI super-types.

Then, we need a MOI.set function like so:

function MOI.set(
    model::Optimizer, key::ConstraintAttribute, 
    c::ConstraintIndex{MOI.ScalarAffineFunction{Float64}, <:Any}, value::Int
)
    set_intattr!(model.inner, key.name, _info(model, c).row, value)
    return
end

Somehow, we need to deal with the different types of attributes (int, dbl, char, etc), and also variable and model attributes.

Does this make sense? You might want to hold off making changes until #216 is merged.

[fa-bien]

Thank you, it does make sense, I will wait until #216 is merged then do as you suggest.

[fa-bien]

Just a quick note, I cannot have both Gurobi 0.7.0 and JuMP 0.19 installed at the same time. If I install Gurobi#master then install JuMP, it installs JuMP 0.18.6. If I install JuMP#master then Gurobi, it installs Gurobi 0.6.0. If I try to install both with #master, there is an unsatisfiable requirement for MathOptInterface. Is this normal or is there something wrong with my setup? I think I need both Gurobi 0.7 and JuMP 0.19 to be able to add the MOI feature described above.

[blegat]

You need to use this branch of JuMP for it to work with MOI v0.9: jump-dev/JuMP.jl#2003

[fa-bien]

After tinkering a bit I am stuck, since Gurobi.set_intattrelement!() requires a Gurobi.Model but all I have when creating my model using with_optimizer() is a Model. Is there any way to obtain a Gurobi.Model from my JuMP Model? The only thing I can find is model.moi_backend.optimizer.model.inner which is of the right type but appears to be empty, i.e. it does not contain variables or constraints previously added to the model.

[blegat]

You need to do MOIU.attach_optimizer(model) to fill it in. Note that this is automatically called at optimize!(model)

[fa-bien]

Thank you! Is this call expensive? If yes, is there any way to check if the model is already "attached"?

[blegat]

Yes but if you call attach and it's already attached it's a no-op so it shouldn't harm. Anyway, you can check that state with MOIU.state(backend(model)), see https://www.juliaopt.org/MathOptInterface.jl/stable/apireference/#Caching-optimizer-1

[mlubin]

Note that touching model.moi_backend.optimizer.model.inner is generally a bad idea because the indices between the JuMP model and this Gurobi model may not map 1:1. direct_model was designed for the use case where you want to work with a JuMP model and the Gurobi model at the same time.

[fa-bien]

Thanks again, I somehow did not realise that this documentation existed, that's on me. On my system the first call to MOIU.attach_optimizer(model) is indeed very expensive:

using JuMP, Gurobi, CPUTime

model = Model(with_optimizer(Gurobi.Optimizer))
@variable(model, x >= 0)
@constraint(model, c, 2x >= 1)
@objective(model, Min, x)

@CPUtime MOIU.attach_optimizer(model)

elapsed CPU time: 370.165058 seconds

This is using JuMP#master and Gurobi 0.7.0.

[fa-bien]

Note that touching model.moi_backend.optimizer.model.inner is generally a bad idea because the indices between the JuMP model and this Gurobi model may not map 1:1. direct_model was designed for the use case where you want to work with a JuMP model and the Gurobi model at the same time.

Thanks, I had no idea, I am discovering the whole code (as made obvious from my questions above). The reason why I am trying to do this is to be able to use MOI.set() with a model created using with_optimizer(), as it was suggested. Now I understand that this is a bad idea to begin with (or there is something else I am missing). Is there some kind of consensus on what to do here? Should I just stick to direct_model?

[odow]

Are you on Julia 1.2? 370s is way too long. How long does optimize! take?

[fa-bien]

Yes, this is Julia 1.2. optimize!() takes long as well, but all the time is spent in the initial MOIU.attach_optimizer() call.

[odow]

You've probably hit JuliaLang/julia#32167. I suggest that you revert to Julia 1.1 until we tag a new version of MOI (and JuMP) with a fix.

[fa-bien]

EDIT: this is in fact solved by calling update_model!(opzr.inner). I do not believe it needs to be fixed. However it is not clear whether MOI.get() should automatically update the model or not.

Following all the recent updates I am using Julia 1.2 with JuMP 0.20.0, MOI 0.9.1 and Gurobi 0.7.2. I have a piece of code that should work but somehow does not, I believe there is an issue with Gurobi.jl so I made a MWE:

using JuMP, Gurobi

model = JuMP.direct_model(Gurobi.Optimizer())
@variable(model, x >= 0)
@constraint(model, c, 2x >= 1)
@objective(model, Min, x)

opzr = backend(model)

Gurobi.set_intattrelement!(opzr.inner, "Lazy", Gurobi._info(opzr, index(c)).row, 2)
Gurobi.get_intattrelement(opzr.inner, "Lazy", Gurobi._info(opzr, index(c)).row)

The call to get_intattrelement() returns 0. A quick experiment on a large model suggests that using set_intattrelement!() to set many constraints as lazy does have an effect, so I am guessing that get_intattrelement() does not function as intended (or that I do not understand the intent).

[odow]

After setting or modifying any attribute that requires an updated, you should call _require_update(model). Then, before querying any attribute that might have required an update, you should call _update_if_necessary(model). See
https://github.com/JuliaOpt/Gurobi.jl/blob/db91e1e9fabc3f62f30265780e6a577016ebba4e/src/MOI_wrapper.jl#L187-L208

For example
https://github.com/JuliaOpt/Gurobi.jl/blob/db91e1e9fabc3f62f30265780e6a577016ebba4e/src/MOI_wrapper.jl#L1933-L1942

[fa-bien]

Thanks, it's great that these functions exist. Is there any reason why they are not systematically called in Gurobi setters and getters? Formulated differently: is there any attribute that does not require an update following modification?

[odow]

Is there any reason why they are not systematically called in Gurobi setters and getters?

We attempted to keep the low-level API similar to the C API, so at that level it's the user's job to call update. This is not the case in the MOI level, so we do it using the functions outlined above.

is there any attribute that does not require an update following modification?

No.

The only exception is that Gurobi will update the model on a call to update, before a call to optimize, or a call to write. See: https://www.gurobi.com/documentation/8.1/refman/updatemode.html

[fa-bien]

First pull request: #243

I only included constraint attributes so far. If people are satisfied with what I did, I can add variable and model attributes too.

[fa-bien]

More pull requests submitted, we now have variable attributes and soon model attributes too. I would also like to have methods that are more convenient to call, I think this needs to be added to JuMP, any idea where I should add this? I'm thinking about something like this:

using JuMP, Gurobi

model = JuMP.direct_model(Gurobi.Optimizer())
@variable(model, x >= 0)
@constraint(model, c, 2x >= 1)
@objective(model, Min, x)

MOI.set(c, "Lazy", 2)

Or if that is not possible, something like

MOI.set(model, c, "Lazy", 2)

[odow]

This should work:

using JuMP, Gurobi

model = JuMP.direct_model(Gurobi.Optimizer())
@variable(model, x >= 0)
@constraint(model, c, 2x >= 1)
@objective(model, Min, x)
MOI.set(model, Gurobi.ConstraintAttribute("Lazy"), c, 2)
optimize!(model)

It's pretty convenient. It just needs to be documented: jump-dev/JuMP.jl#2057

[odow]

From the Gurobi side, I think this issue can be considered closed by #243, #245, and #247.

[rdpbecker]

It should be noted here that the constraints need to be set as lazy individually. If you have a set of constraints defined together, you can't set them as lazy all together. For example, if you have @constraint(model, c[i in 1:2], 2x>=i), you can't just write

MOI.set(model, Gurobi.ConstraintAttribute("Lazy"), c, 2)

Instead, you need

for i in 1:2
    MOI.set(model, Gurobi.ConstraintAttribute("Lazy"), c[i], 2)
end