WIP: use add_constraints in macros #2748

odow · 2021-10-12T02:14:59Z

I took a look at #1939.

It's actually fairly trivial to implement this for constraints. It's harder for variables. The key is that we should just do it for broadcasted constraints, since these are much more likely to be of the same form. And we can limit ourselves to scalar constraints.

However...

I can't find a solver with a speed-up. It seems like we've solved much of the performance problems through the ecosystem.

Here's a benchmark which gets at the difference:

using JuMP, Gurobi
const ENV = Gurobi.Env()

function main(N)
    GC.gc()
    model = direct_model(Gurobi.Optimizer(ENV))
    @variable(model, x[1:N])
    A = rand(N, N)
    f = @expression(model, A * x)
    s = MOI.GreaterThan(0.0)
    cons = ScalarConstraint.(f, s)
    @time add_constraints(model, cons, "")
end

function main2(N)
    GC.gc()
    model = direct_model(Gurobi.Optimizer(ENV))
    @variable(model, x[1:N])
    A = rand(N, N)
    f = @expression(model, A * x)
    s = MOI.GreaterThan(0.0)
    cons = ScalarConstraint.(f, s)
    @time add_constraint.(model, cons, "")
end

Which yields

julia> @time main(2000);
  0.164610 seconds (16.03 k allocations: 168.573 MiB)
  1.143480 seconds (90.12 k allocations: 432.391 MiB, 14.63% gc time)

julia> @time main(2000);
  0.161069 seconds (16.03 k allocations: 168.573 MiB)
  1.120660 seconds (90.12 k allocations: 432.391 MiB, 13.44% gc time)

julia> @time main2(2000);
  0.110007 seconds (18.02 k allocations: 107.786 MiB)
  1.073805 seconds (92.11 k allocations: 371.605 MiB, 14.25% gc time)

julia> @time main2(2000);
  0.108142 seconds (18.02 k allocations: 107.786 MiB)
  1.117617 seconds (92.11 k allocations: 371.605 MiB, 17.60% gc time)

So fewer allocations by count, but the constructions of moi_function.(cons) and moi_set.(cons) lead to more allocations my memory size which slow things down.

We've also jigged things so that the various copy_to are now much faster than they were when #1939 was opened.

We can consider adding this, but I would vote to close this and #1939 and leave as-is for now.

codecov · 2021-10-13T01:11:06Z

Codecov Report

Merging #2748 (6113b9c) into master (ce402d6) will increase coverage by 0.14%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2748      +/-   ##
==========================================
+ Coverage   94.03%   94.18%   +0.14%     
==========================================
  Files          43       43              
  Lines        5519     5523       +4     
==========================================
+ Hits         5190     5202      +12     
+ Misses        329      321       -8

Impacted Files	Coverage Δ
src/constraints.jl	`95.89% <100.00%> (+0.32%)`	⬆️
src/macros.jl	`95.76% <100.00%> (+0.58%)`	⬆️
src/Containers/nested_iterator.jl	`100.00% <0.00%> (ø)`
src/Containers/vectorized_product_iterator.jl	`95.23% <0.00%> (ø)`
src/aff_expr.jl	`96.00% <0.00%> (+0.09%)`	⬆️
src/Containers/macro.jl	`100.00% <0.00%> (+3.33%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce402d6...6113b9c. Read the comment docs.

rafabench · 2021-10-21T15:53:03Z

I have tested with other solvers and the results looks almost the same for 2000 and slightly better for 10000

using JuMP, Xpress, GLPK, CPLEX, Gurobi
const GRB_ENV = Gurobi.Env()

function main(optimizer,N)
    GC.gc()
    model = direct_model(backend(Model(optimizer)))
    @variable(model, x[1:N])
    A = rand(N, N)
    f = @expression(model, A * x)
    s = MOI.GreaterThan(0.0)
    cons = ScalarConstraint.(f, s)
    @time add_constraints(model, cons, "")
end

function main2(optimizer,N)
    GC.gc()
    model = direct_model(backend(Model(optimizer)))
    @variable(model, x[1:N])
    A = rand(N, N)
    f = @expression(model, A * x)
    s = MOI.GreaterThan(0.0)
    cons = ScalarConstraint.(f, s)
    @time add_constraint.(model, cons, "")
end

function test_solvers(optimizers, names, N)
    for (i,optimizer) in enumerate(optimizers)
        @info "Testing $(names[i])"
        oldstd = stdout
        redirect_stdout(devnull)
        main(optimizer,N);
        main2(optimizer,N);
        redirect_stdout(oldstd)
        println("Main")
        @time main(optimizer,N);
        println("Main2")
        @time main2(optimizer,N);
    end
end

names = ["Xpress", "GLPK", "CPLEX", "Gurobi"]
optimizers = [() -> Xpress.Optimizer(),() ->  GLPK.Optimizer(),() ->  CPLEX.Optimizer(),() ->  Gurobi.Optimizer(GRB_ENV)]

N = 2000
test_solvers(optimizers, names, N)

N = 10000
test_solvers(optimizers, names, N)

And the results are:
N = 2000

┌ Info: Testing Xpress
└ @ Main In[5]:3
Main
  0.092146 seconds (16.02 k allocations: 122.699 MiB, 9.35% gc time)
  1.444067 seconds (21.91 M allocations: 902.583 MiB, 19.99% gc time)
Main2
  0.065943 seconds (16.02 k allocations: 122.653 MiB, 12.15% gc time)
  1.422197 seconds (21.91 M allocations: 902.538 MiB, 19.38% gc time)
┌ Info: Testing GLPK
└ @ Main In[5]:3
Main
  0.097314 seconds (16.02 k allocations: 122.699 MiB, 30.29% gc time)
  1.445950 seconds (21.94 M allocations: 902.948 MiB, 21.63% gc time)
Main2
  0.072134 seconds (16.02 k allocations: 122.653 MiB, 4.92% gc time)
  1.512831 seconds (21.94 M allocations: 902.903 MiB, 24.07% gc time)
┌ Info: Testing CPLEX
└ @ Main In[5]:3
Main
  0.090583 seconds (16.02 k allocations: 122.699 MiB, 8.60% gc time)
  1.477325 seconds (21.97 M allocations: 903.468 MiB, 19.70% gc time)
Main2
  0.070718 seconds (16.02 k allocations: 122.653 MiB, 12.44% gc time)
  1.428765 seconds (21.97 M allocations: 903.423 MiB, 19.35% gc time)
┌ Info: Testing Gurobi
└ @ Main In[5]:3
Main
  0.077367 seconds (16.02 k allocations: 122.699 MiB, 11.07% gc time)
  1.420848 seconds (21.92 M allocations: 902.644 MiB, 19.70% gc time)
Main2
  0.066729 seconds (16.02 k allocations: 122.653 MiB, 11.84% gc time)
  1.403598 seconds (21.92 M allocations: 902.599 MiB, 19.60% gc time)

N = 10000

┌ Info: Testing Xpress
└ @ Main In[5]:3
Main
  4.793383 seconds (80.03 k allocations: 2.983 GiB, 27.57% gc time)
 57.871738 seconds (583.52 M allocations: 25.048 GiB, 30.77% gc time)
Main2
  5.668366 seconds (80.02 k allocations: 2.983 GiB, 41.02% gc time)
 60.560627 seconds (583.52 M allocations: 25.048 GiB, 34.79% gc time)
┌ Info: Testing GLPK
└ @ Main In[5]:3
Main
  4.779884 seconds (80.03 k allocations: 2.983 GiB, 27.53% gc time)
 58.396432 seconds (583.41 M allocations: 25.047 GiB, 30.08% gc time)
Main2
  5.641114 seconds (80.02 k allocations: 2.983 GiB, 41.80% gc time)
 60.696609 seconds (583.41 M allocations: 25.046 GiB, 34.34% gc time)
┌ Info: Testing CPLEX
└ @ Main In[5]:3
Main
  4.792076 seconds (80.03 k allocations: 2.983 GiB, 27.89% gc time)
 57.103322 seconds (583.86 M allocations: 25.053 GiB, 30.97% gc time)
Main2
  5.704172 seconds (80.02 k allocations: 2.983 GiB, 41.12% gc time)
 60.088743 seconds (583.86 M allocations: 25.053 GiB, 34.51% gc time)
┌ Info: Testing Gurobi
└ @ Main In[5]:3
Main
  4.719514 seconds (80.03 k allocations: 2.983 GiB, 27.68% gc time)
 56.955054 seconds (583.28 M allocations: 25.045 GiB, 30.46% gc time)
Main2
  5.662543 seconds (80.02 k allocations: 2.983 GiB, 41.62% gc time)
 60.543900 seconds (583.28 M allocations: 25.045 GiB, 34.50% gc time)

odow · 2021-10-21T21:35:44Z

Did you run the benchmarks on this branch? The allocations are the same, but I'd expect main to be slightly higher.

rafabench · 2021-10-21T23:41:18Z

Yes, my Pkg status looks like this:

[a076750e] CPLEX v0.8.0 `https://github.com/jump-dev/CPLEX.jl.git#master`
[60bf3e95] GLPK v0.15.0
[2e9cd046] Gurobi v0.10.0 `https://github.com/jump-dev/Gurobi.jl.git#master`
[4076af6c] JuMP v0.22.0 `https://github.com/jump-dev/JuMP.jl.git#od/add_constraints`
[295af30f] Revise v3.1.20
[9e70acf3] Xpress v0.14.0 `https://github.com/jump-dev/Xpress.jl.git#moi_v0.10`

I will run again to check if the allocations stay the same

odow · 2021-10-22T00:07:58Z

Hmm. I think this errs on the side of might be better, might be worse. But it's not significantly so. I think that just means we've improved things since the original issues were opened.

For the same reason as #1905 (comment), I'd propose we close this PR and linked issue in favor of more general performance issues such as #2735 and #42 (a 2-digit issue!) to track on-going performance. Ideally, we'll set up something that makes it easy to benchmark JuMP, and then keep attacking that.

WIP: use add_constraints in macros

1ad1c16

odow added Status: Needs review Type: Performance Status: Needs developer call This should be discussed on a monthly developer call labels Oct 12, 2021

odow mentioned this pull request Oct 12, 2021

Macros should use add_variables & add_constraints rather than singular methods #1939

Closed

odow added 2 commits October 13, 2021 11:06

Update constraints.jl

e3e722b

Update constraints.jl

6113b9c

odow closed this Oct 25, 2021

odow deleted the od/add_constraints branch October 25, 2021 02:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: use add_constraints in macros #2748

WIP: use add_constraints in macros #2748

odow commented Oct 12, 2021

codecov bot commented Oct 13, 2021 •

edited

Loading

rafabench commented Oct 21, 2021

odow commented Oct 21, 2021 •

edited

Loading

rafabench commented Oct 21, 2021

odow commented Oct 22, 2021

WIP: use add_constraints in macros #2748

WIP: use add_constraints in macros #2748

Conversation

odow commented Oct 12, 2021

codecov bot commented Oct 13, 2021 • edited Loading

Codecov Report

rafabench commented Oct 21, 2021

odow commented Oct 21, 2021 • edited Loading

rafabench commented Oct 21, 2021

odow commented Oct 22, 2021

codecov bot commented Oct 13, 2021 •

edited

Loading

odow commented Oct 21, 2021 •

edited

Loading