Implementation of column generation stages #525

rrsadykov · 2021-05-10T16:39:00Z

Reimplements pull request #519 according to the discussion there

…d exact stage)

codecov · 2021-05-10T16:45:22Z

Codecov Report

Merging #525 (63ee7ff) into release-0.4.0 (b5795cb) will decrease coverage by 0.13%.
The diff coverage is 90.22%.

@@                Coverage Diff                @@
##           release-0.4.0     #525      +/-   ##
=================================================
- Coverage          83.95%   83.82%   -0.14%     
=================================================
  Files                 47       49       +2     
  Lines               4606     4648      +42     
=================================================
+ Hits                3867     3896      +29     
- Misses               739      752      +13

Impacted Files	Coverage Δ
src/Algorithm/basic/solveipform.jl	`90.78% <60.00%> (-1.99%)`	⬇️
src/Algorithm/treesearch.jl	`94.57% <83.33%> (-0.46%)`	⬇️
src/MathProg/formulation.jl	`73.87% <83.33%> (+0.08%)`	⬆️
src/Algorithm/basic/pricingcallback.jl	`85.71% <85.71%> (ø)`
src/Algorithm/pricing.jl	`90.90% <90.90%> (ø)`
src/MathProg/MOIinterface.jl	`90.90% <91.66%> (-0.83%)`	⬇️
src/Algorithm/conquer.jl	`82.35% <92.59%> (-0.38%)`	⬇️
src/Algorithm/basic/solvelpform.jl	`94.23% <100.00%> (ø)`
src/Algorithm/colgen.jl	`95.72% <100.00%> (-0.76%)`	⬇️
src/Algorithm/colgenstabilization.jl	`99.24% <100.00%> (+0.01%)`	⬆️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b5795cb...63ee7ff. Read the comment docs.

src/Algorithm/colgen.jl

src/Algorithm/conquer.jl

src/Algorithm/pricing.jl

src/MathProg/formulation.jl

src/MathProg/optimizerwrappers.jl

rrsadykov · 2021-05-12T09:22:34Z

I have pushed the changes requested by Guillaume.

In the future, the user should provide himself the primal and dual bounds in the callback. I will create an issue for that.

rrsadykov · 2021-05-12T11:08:58Z

I have simplified ColCutGenConquer algorithm. In fact, it does not need to know which stage is heuristic or not. This information is passed by the column generation algorithm through bounds.

guimarqu

I'm ok for the new algorithm that call either PricingCallback or the MOI optimzer. However, I'm afraid that the MOI optimizer is always provided even if the formulation is not entered by the user.

guimarqu · 2021-05-12T14:14:29Z

src/Algorithm/pricing.jl

@@ -0,0 +1,25 @@
+@with_kw struct DefaultPricing <: AbstractOptimizationAlgorithm


This algorithm should be renamed because it can be used to solve a Formulation outside the column generation algorithm. When we don't know if the optimizer of a formulation is a ~~pricing callback~~ User optimizer or a MOI optimizer, we'll have to call this method to optimize the formulation (that's why SolveIpForm was supporting both the ~~pricing callback~~user optimizer and the MOI optimizer). The name of the algorithm should state that it's going to solve the formulation.

By the way, docstring is missing.

EDIT : wording

I do not completely understand why this algorithm can be called outside column generation. It uses pricing callback. What is the other use case?

It is more than a pricing callback, it's the solver of the subproblem. Pricing callback is a use case of this solver.

BD.specify!.(subproblems, lower_multiplicity = 0, solver = my_pricing_callback)

PS: I edited my first comment.

So, which name do you suggest?

src/Algorithm/conquer.jl

guimarqu · 2021-05-12T14:29:07Z

src/decomposition.jl

@@ -471,7 +475,12 @@ function buildformulations!(
    assign_orig_vars_constrs!(spform, origform, env, annotations, ann)
    create_side_vars_constrs!(spform, origform, env, annotations)
    closefillmode!(getcoefmatrix(spform))
-    initialize_optimizer!(spform, getoptbuilder(prob, ann))
+    initialize_moioptimizer!(spform, getmoioptbuilder(prob, ann))


If the user can use both the pricing callback and the MOI optimizer, what happens if the user didn't specify the constraints of the subproblems ? Are we going to loose time updating the solver buffer that we will never flush ?

I did not think about this. Is it a big overhead if we have MOI optimizer for the subproblem we never use?

I don't know but we should avoid updating the MOI optimizer if the user does not provide the whole formulation.

I do not know how to do it.

src/Algorithm/pricing.jl

guimarqu · 2021-05-12T14:35:41Z

src/MathProg/formulation.jl

@@ -3,7 +3,8 @@ mutable struct Formulation{Duty <: AbstractFormDuty}  <: AbstractFormulation
    var_counter::Int
    constr_counter::Int
    parent_formulation::Union{AbstractFormulation, Nothing} # master for sp, reformulation for master
-    optimizer::AbstractOptimizer
+    moioptimizer::AbstractOptimizer


Suggested change

moioptimizer::AbstractOptimizer

moioptimizer::Union{Nothing, MoiOptimizer}

to make sure you receive a MoiOptimizer

I cannot do that because MoiOptimizer is defined later in optimizerwrappers.jl. And I cannot move optimizerwrappers.jl above formulation.jl, as Formulation is used there.

Ok I see. That's a problem.

guimarqu · 2021-05-12T14:36:08Z

src/MathProg/formulation.jl

@@ -3,7 +3,8 @@ mutable struct Formulation{Duty <: AbstractFormDuty}  <: AbstractFormulation
    var_counter::Int
    constr_counter::Int
    parent_formulation::Union{AbstractFormulation, Nothing} # master for sp, reformulation for master
-    optimizer::AbstractOptimizer
+    moioptimizer::AbstractOptimizer
+    useroptimizer::AbstractOptimizer 


Suggested change

useroptimizer::AbstractOptimizer

useroptimizer::Union{Nothing, UserOptimizer}

to make sure you receive a UserOptimizer

src/decomposition.jl

rrsadykov · 2021-05-13T11:37:34Z

Guillaume, may be you implement yourself the changes you want to do? There should remain few things to do.

artalvpes · 2021-05-14T00:41:47Z

Hi @rrsadykov ,

I just started looking at this version. The interface is really much more intuitive! Congratulations! However, there were two features in the original version that are missing here, which I consider very important: showing the current pricing stage in the log file, and counting the number of of calls to the exact stage in the test. The former helps to check whether the performance of the solver is as expected when solving larger instances, and the latter is to make sure that the heuristic pricing is effective to save some exact pricing calls. For example, the heuristic pricing might never be called or might never produce useful columns or such columns might never be inserted in the master and the test would pass (relying only in the exact phase to solve the relaxations).

guimarqu · 2021-05-15T00:47:07Z

I suggest to keep the name SolveIpForm for the optimizer of the formulation with integer variables.
We keep the algorithm split MoiSolver / UserSolver. I like it, it's cleaner.
SolveIpForm automatically calls the appropriate algorithm MoiSolver / UserSolver depending on the type of the solver.

We can change the specify! method in BlockDecomposition to allow the user to pass a vector of optimizers :

BD.specify!.(subproblems, lower_multiplicity = 0, solver = [MOI.Gurobi, my_callback_stage2, my_callback_stage3])

and we change the dispatch argument into solver :

stages = [ 
    ClA.ColumnGeneration(pricing_prob_solve_alg = ClA.SolveIpForm(solver = 1)),
    ClA.ColumnGeneration(pricing_prob_solve_alg = ClA.SolveIpForm(solver = 2)),
    ClA.ColumnGeneration(pricing_prob_solve_alg = ClA.SolveIpForm(solver = 3)),
]

So in that case, we do the exact phase with MOI.Gurobi and we use a callback for stages 2 & 3. The user will have to write a callback for each phase.

I don't see any better solution at the moment to handle #525 (comment).

Moreover, if we don't fix this, we'll have a design issue with customized optimizers. I would like to have these optimizers connected to Coluna through MOI. So to avoid buffering changes several times, I suggest to identify the solvers that have the same MOI optimizers and set the parameters just before optimizing the formulation. For instance, if you define solvers of your subproblem like this,

solvers = [
     with_attributes(MOI.RCSP, mode = :exact), 
     with_attributes(MOI.RCSP, mode = :heuristic),
     with_attributes(MOI.RCSP, mode = :speed_heuristic)
]

Coluna will instanciate MOI.RCSP only once for the subproblem, and will set the parameters just before optimizing the subproblem. We'll have to find a way to set parameters only when needed (because changing parameters may need some calculations ?).

What do you think ? (cc @vitornesello )

rrsadykov · 2021-05-15T08:20:35Z

So in that case, we do the exact phase with MOI.Gurobi and we use a callback for stages 2 & 3. The user will have to write a callback for each phase.

Should the user also write a callback for the MOI.Gurobi stage?

Is it possible here to give attributes like solver=JuMP.optimizer_with_attributes(GLPK.Optimizer, "tm_lim" => 60 * 2_200)?

guimarqu · 2021-05-15T08:46:58Z

So in that case, we do the exact phase with MOI.Gurobi and we use a callback for stages 2 & 3. The user will have to write a callback for each phase.

Should the user also write a callback for the MOI.Gurobi stage?

Is it possible here to give attributes like solver=JuMP.optimizer_with_attributes(GLPK.Optimizer, "tm_lim" => 60 * 2_200)?

No the user doesn't write a callback when he uses a MOI optimizer. He can pass an optimizer with attributes (already working).

rrsadykov · 2021-05-15T08:51:40Z

If the solver in SolveIpForm is fixed, it is not worth anymore for me to define child algorithms MoiSolver and UserSolver. Instead we may have SolveIpFormWithMOI and SolveIpFormWithUserSolver. So we may have something like

BD.specify!.(subproblems, lower_multiplicity = 0, user_solvers = [my_callback_stage2, my_callback_stage3])

and then

stages = [ClA.ColumnGeneration(pricing_prob_solve_alg = ClA.SolveIpFormWithMOI(moi=MOI.Gurobi)),
          ClA.ColumnGeneration(pricing_prob_solve_alg = ClA.SolveIpFormWithUserSolver(solver = 1)),
          ClA.ColumnGeneration(pricing_prob_solve_alg = ClA.SolveIpFormWithUserSolver(solver = 2))
          ]

@guimarqu, what do you think?

rrsadykov · 2021-05-15T08:55:36Z

Also, I do not understand how this would allow us to avoid the problem raised in #525 (comment). How Coluna knows whether to create moioptimizer or not?

guimarqu · 2021-05-15T08:59:26Z

We can keep the split even if the user won't call these two algorithms.

Your solution won't work if one wants to optimize subproblems with different solvers.

If the solver builder stored in the annotation of the subproblem returns a MOI optimizer, Coluna will create a MoiOptimizer. Otherwise it's a UserOptimizer.

rrsadykov · 2021-05-15T09:02:16Z

I would like to have customized optimizers connected to Coluna through MOI.

I do not think this is good idea. First, custom optimizer is not necessarily a mathematical optimizaton algorithm to be connected through MOI (Mathematical Optimization Interface). For example, RCSP solver is not a mathematical optimization solver. Second, it does not suffice to write solvers = [with_attributes(MOI.RCSP, mode = :exact)] to use RCSP solver for the pricing. You need also to define your graph, resources, etc. Moreover, you need to connect also additional algorithms, like reduced cost fixing, enumeration, also you need to define additional storage units.

We really like the user interface we have in VrpSolver (through mapping). For me, Coluna should be able to allow dependent packages to use any user interface they want to define subproblems (through custom annotations?).

guimarqu · 2021-05-15T09:08:47Z

I would like to have customized optimizers connected to Coluna through MOI.

I do not think this is good idea. First, custom optimizer is not necessarily a mathematical optimizaton algorithm to be connected through MOI (Mathematical Optimization Interface). For example, RCSP solver is not a mathematical optimization solver. Second, it does not suffice to write solvers = [with_attributes(MOI.RCSP, mode = :exact)] to use RCSP solver for the pricing. You need also to define your graph, resources, etc. Moreover, you need to connect also additional algorithms, like reduced cost fixing, enumeration, also you need to define additional storage units.

We really like the user interface we have in VrpSolver (through mapping). For me, Coluna should be able to allow dependent packages to use any user interface they want to define subproblems (through custom annotations?).

Yes I agree. I was speaking only about the communication between Coluna and the solver. For instance, when Coluna changes the cost of the subproblem variables to give the reduced costs to the subsolver or also the bounds of the variables.

rrsadykov · 2021-05-15T09:23:31Z

Yes I agree. I was speaking only about the communication between Coluna and the solver. For instance, when Coluna changes the cost of the subproblem variables to give the reduced costs to the subsolver or also the bounds of the variables.

I think Coluna just needs to calculate the reduced costs (and bounds, etc) of the representative variables in the master (MasterRepPricingVar). This is sufficient. Then pricing solver will use this information to run (+ dual values of non-robust cuts), and then it returns the solution which is defined on these representative variables (+ custom info if needed).

I think that BD.specify!() is not needed for custom pricing solvers. BlockDecomposition should only provide interface for 1) adding custom models as subproblems and 2) declaring which variables defined by the user are sp. representative variables and which variables are standard (pure master). No variables need to be defined in custom subproblems.

rrsadykov · 2021-05-15T09:48:46Z

However, there were two features in the original version that are missing here, which I consider very important: showing the current pricing stage in the log file, and counting the number of of calls to the exact stage in the test.

Current pricing stage is shown in run!(algo::ColCutGenConquer,...) before running this stage:

            if length(algo.stages) > 1 
                @logmsg LogLevel(0) "Column generation stage $stage is started"
            end

Counting the number of calls can easily be counted in the callback.

guimarqu · 2021-05-15T12:06:36Z

Yes I agree. I was speaking only about the communication between Coluna and the solver. For instance, when Coluna changes the cost of the subproblem variables to give the reduced costs to the subsolver or also the bounds of the variables.

I think Coluna just needs to calculate the reduced costs (and bounds, etc) of the representative variables in the master (MasterRepPricingVar). This is sufficient. Then pricing solver will use this information to run (+ dual values of non-robust cuts), and then it returns the solution which is defined on these representative variables (+ custom info if needed).

I think that BD.specify!() is not needed for custom pricing solvers. BlockDecomposition should only provide interface for 1) adding custom models as subproblems and 2) declaring which variables defined by the user are sp. representative variables and which variables are standard (pure master). No variables need to be defined in custom subproblems.

BD.specify! is needed for the custom pricing solvers. That's how you pass the information to Coluna. 1) the custom model will be given together with the solver. 2) No variable will be defined in the custom subproblems but when the user writes the subproblems models, he will have to map the subproblem variables to the decisions taken in the subproblem. This is specific to the custom solver.

If you agree with this future change #525 (comment), I think it's ready for merge.

rrsadykov · 2021-05-15T12:33:08Z

BD.specify! is needed for the custom pricing solvers. That's how you pass the information to Coluna. 1) the custom model will be given together with the solver. 2) No variable will be defined in the custom subproblems but when the user writes the subproblems models, he will have to map the subproblem variables to the decisions taken in the subproblem. This is specific to the custom solver.

Ok, custom models can be given with BD.specify!. The pricing solver can be given by directly setting pricing_prob_solve_alg in ColumnGeneration. Yes, mapping is the responsibility of the custom subproblem. It just needs to have access to subproblem representative variables in the master.

rrsadykov · 2021-05-15T12:34:01Z

If you agree with this future change #525 (comment), I think it's ready for merge.

Ok for me.

* dev branch to prepare release of 0.4.0 * Move storage & records in ColunaBase (#507) * Renaming in storage (#509) * RecordContainer -> RecordWrapper * state -> record * rename lot of things, remove getters of Storage not used by Algorithms * Storage -> StorageUnitWrapper * Deletion of `AbstractData` (#510) * RecordContainer -> RecordWrapper * state -> record * rename lot of things, remove getters of Storage not used by Algorithms * start removing AbstractData structs * tests ok * getunit -> getstorageunit; StorageDict -> Storage * fix docstring * Bijection StorageUnit -> Record (#518) * Bijection StorageUnit -> Record * address Ruslan's comments * Implementation of column generation stages (#525) * Implementation of column generation stages (for example, heuristic and exact stage) * Update after conversation with Guillaume + stabilization correction * Simplification for ColCutGenConquer * Some more modifs due to Guillaume comments * Counting the number of exact calls when testing the pricing stages (#530) * Add bound callback tests (#532) * add bound callback tests * include bound callback in runtests * fix test * Apply suggestions from code review Co-authored-by: Guillaume Marques <[email protected]> * add comment * Apply suggestions from code review Co-authored-by: Guillaume Marques <[email protected]> Co-authored-by: Guillaume Marques <[email protected]> * Vector of optimizers in `Formulation` (#534) * vector of optimizers in formulation * solver_id -> optimizer_id * add Manifest * update Manifest * remove Manifest because does not work * changes * rm files * address Ruslan's comment * Update src/MathProg/optimizerwrappers.jl Co-authored-by: Vitor Nesello <[email protected]> * add Manifest * change ci * remove ci change * rm Manifest Co-authored-by: Vitor Nesello <[email protected]> * UnitsUsageDict -> UnitsUsage (#522) * UnitsUsageDict -> UnitsAccess * wip * improve * tests ok * Custom data for variables and constraints (#495) * draft for support of customer data * custom data in solution * custom data for cut callback * computecoeff * store custom data of solutions in manager * add Manifest * rm Manifest * Support to custom cuts over custom data assigned to columns with new test * tests ok Co-authored-by: Artur Alves Pessoa <[email protected]> * Prototyping custom model/optimizer (#535) * Start example * wip draft * continue * add map * works with caching optimizer * varids in Env * wrong result * multiply costs by -1 * fix scaling * Apply suggestions from code review * Update test/interfaces/model.jl Co-authored-by: Lara Pontes <[email protected]> * Follow up of "custom data" (#538) * add AbstractCustomData and set/get inc_val * fix bugs * remove duplicate methods * delete unnecessary prefixes and fix some bugs * revert some changes and update docstring * custom information for dw sp (#542) * docstring for restricted master heuristic (#543) * docstring for restricted master heuristic * Update src/Algorithm/conquer.jl * Refactoring `ObjValues` & `OptimizationState` (#544) * clean + doc * move doc from objvalues to optstate; tests of objvalues; update ci * update * update branching priorioty deprecated method * tests ok * tests ok * tests ok * delete duplicated tests Co-authored-by: Ruslan Sadykov <[email protected]> Co-authored-by: Artur Pessoa <[email protected]> Co-authored-by: Lara di Cavalcanti Pontes <[email protected]> Co-authored-by: Vitor Nesello <[email protected]> Co-authored-by: Lara Pontes <[email protected]>

Implementation of column generation stages (for example, heuristic an…

6ba2d87

…d exact stage)

rrsadykov requested review from guimarqu and artalvpes May 10, 2021 16:39

rrsadykov mentioned this pull request May 10, 2021

Added support to pricing phases for use of heuristic pricing #519

Closed

guimarqu reviewed May 10, 2021

View reviewed changes

Update after conversation with Guillaume + stabilization correction

7544a2d

rrsadykov requested a review from guimarqu May 12, 2021 09:20

Simplification for ColCutGenConquer

1351829

guimarqu reviewed May 12, 2021

View reviewed changes

Some more modifs due to Guillaume comments

63ee7ff

guimarqu approved these changes May 15, 2021

View reviewed changes

guimarqu mentioned this pull request May 15, 2021

Multiple solvers for subproblems atoptima/BlockDecomposition.jl#54

Closed

guimarqu merged commit 54bb3e7 into release-0.4.0 May 15, 2021

guimarqu deleted the colgen_stages branch May 15, 2021 12:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of column generation stages #525

Implementation of column generation stages #525

rrsadykov commented May 10, 2021

codecov bot commented May 10, 2021 •

edited

Loading

rrsadykov commented May 12, 2021 •

edited

Loading

rrsadykov commented May 12, 2021

guimarqu left a comment

guimarqu May 12, 2021 •

edited

Loading

rrsadykov May 12, 2021

guimarqu May 12, 2021 •

edited

Loading

rrsadykov May 13, 2021

guimarqu May 12, 2021

rrsadykov May 12, 2021

guimarqu May 12, 2021

rrsadykov May 13, 2021

guimarqu May 12, 2021

rrsadykov May 12, 2021 •

edited

Loading

guimarqu May 12, 2021

guimarqu May 12, 2021

rrsadykov May 12, 2021

rrsadykov commented May 13, 2021

artalvpes commented May 14, 2021 •

edited

Loading

guimarqu commented May 15, 2021

rrsadykov commented May 15, 2021 •

edited

Loading

guimarqu commented May 15, 2021

rrsadykov commented May 15, 2021

rrsadykov commented May 15, 2021

guimarqu commented May 15, 2021 •

edited

Loading

rrsadykov commented May 15, 2021

guimarqu commented May 15, 2021

rrsadykov commented May 15, 2021

rrsadykov commented May 15, 2021

guimarqu commented May 15, 2021

rrsadykov commented May 15, 2021

rrsadykov commented May 15, 2021

		@@ -0,0 +1,25 @@
		@with_kw struct DefaultPricing <: AbstractOptimizationAlgorithm

	moioptimizer::AbstractOptimizer
	moioptimizer::Union{Nothing, MoiOptimizer}

	useroptimizer::AbstractOptimizer
	useroptimizer::Union{Nothing, UserOptimizer}

Implementation of column generation stages #525

Implementation of column generation stages #525

Conversation

rrsadykov commented May 10, 2021

codecov bot commented May 10, 2021 • edited Loading

Codecov Report

rrsadykov commented May 12, 2021 • edited Loading

rrsadykov commented May 12, 2021

guimarqu left a comment

Choose a reason for hiding this comment

guimarqu May 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guimarqu May 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rrsadykov May 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rrsadykov commented May 13, 2021

artalvpes commented May 14, 2021 • edited Loading

guimarqu commented May 15, 2021

rrsadykov commented May 15, 2021 • edited Loading

guimarqu commented May 15, 2021

rrsadykov commented May 15, 2021

rrsadykov commented May 15, 2021

guimarqu commented May 15, 2021 • edited Loading

rrsadykov commented May 15, 2021

guimarqu commented May 15, 2021

rrsadykov commented May 15, 2021

rrsadykov commented May 15, 2021

guimarqu commented May 15, 2021

rrsadykov commented May 15, 2021

rrsadykov commented May 15, 2021

codecov bot commented May 10, 2021 •

edited

Loading

rrsadykov commented May 12, 2021 •

edited

Loading

guimarqu May 12, 2021 •

edited

Loading

guimarqu May 12, 2021 •

edited

Loading

rrsadykov May 12, 2021 •

edited

Loading

artalvpes commented May 14, 2021 •

edited

Loading

rrsadykov commented May 15, 2021 •

edited

Loading

guimarqu commented May 15, 2021 •

edited

Loading