ColGen Iteration & Phases API

[guimarqu]

I'll update the description from time to time to take into account the feedback

Defining interfaces make algorithms easier to test, document, and customize.

It should lead to:

• a main definition of your algorithm which is a sequence of calls to the functions of the interface + generic operations + error handling
• default implementation of the interface

The main definition of the algorithm must be independent of MathProg.
It allows us to easily test the main definition of the algorithm with stub & fake methods of the interface.

The default implementation of the interface has dependencies with Coluna MathProg/Base…
It’s not easy to test because it usually requires the creation of a formulation/reformulation.

Column generation

I'm preparing the API for column generation. I suggest three generic methods for this algorithm:

1.

A first one that "iterates" on column generation phases (phase 1, phase 2, phase 3)

# ignore arguments at the moment
function run!()
    phase = initial_phase(context)
    while !isnothing(phase)
        setup_reformulation(context, phase, reform)
        run_colgen_phase!(context, phase, reform)
        phase = next_phase(context, phase, reform)
    end
    return
end

I think this one is ok.

2.

A second is the main column generation algorithm

function run_colgen_phase!(context, phase, reform)
    while !stop_colgen_phase(context, phase, reform)
        # cleanup
        before_colgen_iteration(context) # does nothing by default (will be useful to print formulation)
        run_colgen_iteration!(context, phase, reform)
        after_colgen_iteration(context) # does nothing by default (same as after)
        if separate_cuts()
            before_cut_separation!() # does nothing by default
            run_cut_separation!()
            after_cut_separation!() # does nothing by default
        end
    end
end

3.

A third one is an iteration of the column generation

# ignore arguments at the moment
function run_colgen_iteration!()
    mast_result = optimize_master_problem!(context, phase, reform)
    stop = check_master_result(mast_result)

    pb = compute_primal_bound!(context, phase, reform)
    red_costs = compute_sp_vars_red_costs(context, phase, reform, mast_dual_sol)

    update_sp_vars_red_costs!(context, sp, red_costs)
    update_master_constrs_dual_vals!(context, phase, reform, mast_dual_sol)

    for sp in get_dw_subprobs(reform)
        pricing_result = optimize_pricing_problem!(context, sp)
        stop = check_pricing_result(pricing_result)
    end

    db = compute_dual_bound!(context, phase, reform)

    return
end

WIP (stabilization, cuts, error handling)

These three methods should only depend on the ColGen interface. They should not call any other methods from Coluna that manipulated complicated objects. Otherwise, it will be a pain to test them.

Sequence of phases

This is the first suggestion for the implementation of next_phase:

stateDiagram-v2
    [*] --> Phase3
    Phase3 --> Phase1 : Converged but LP solution has artifical vars \n Ph3 # iterations limit but last LP solution has artifical vars
    Phase3 --> [*]: Converged \n Infeasible subproblem \n Time limit \n Ph3 # iterations limit
    Phase1 --> [*]: Infeasible \n Ph1+2 # iterations limit (consider as infeasible) \n Time limit
    Phase1 --> Phase2: Converged with no artificial vars
    Phase2 --> [*]: Converged \n Ph1+2 # iterations limit \n Time limit

Loading

Edits:

• March 3: remove loops for essential cut separation on phase state diagram (comment of Ruslan).
• March 3: cut separation in run_colgen_iteration (comments of fv & Ruslan)

[guimarqu]

ping @rrsadykov @artalvpes

[codecov]

Codecov Report

Patch coverage: 46.66% and project coverage change: +17.43 🎉

Comparison is base (ce96370) 1.91% compared to head (31d62b4) 19.34%.

❗ Current head 31d62b4 differs from pull request most recent head 3d7b183. Consider uploading reports for the commit 3d7b183 to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #759       +/-   ##
===========================================
+ Coverage    1.91%   19.34%   +17.43%     
===========================================
  Files          58       62        +4     
  Lines        5337     5501      +164     
===========================================
+ Hits          102     1064      +962     
+ Misses       5235     4437      -798     

Impacted Files	Coverage Δ
src/Algorithm/colgen.jl	0.00% <ø> (ø)
src/Algorithm/colgen/utils.jl	0.00% <0.00%> (ø)
src/Algorithm/treesearch/interface.jl	0.00% <ø> (ø)
src/Coluna.jl	100.00% <ø> (ø)
src/Algorithm/colgen/default.jl	59.15% <59.15%> (ø)
src/ColGen/interface.jl	63.63% <63.63%> (ø)

... and 21 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[rrsadykov]

Guillaume @guimarqu,

You say that "It allows us to easily test the main definition of the algorithm". Can you give example(s) of the tests you can do without creating (re)formulations?

The API for the column generation you prepare will be used only for two implementations (fake one for the tests and the normal one for standard col.gen.) or there will other implementations?

For the API itself, I find it ok. The only thing: why you have loop arcs for essential cut separation? I do not think that it is necessary to restart a phase when we add a violated cut, we can just resolve the master and continue to generate columns.

Ruslan

[guimarqu]

Example for the tree search interface:
https://github.com/atoptima/Coluna.jl/blob/master/test/unit/Algorithm/treesearch_interface.jl
This is a simple example of using the tree search API without MathProg formulation but this is not enough for tests.

We need to test the boundaries carefully. In colgen: what should happen if the master optimization fails (define behavior for each termination status), same for subproblems. What should be the return status if the time limit is reached, and max nb of iterations is reached? Make sure all the transitions work.
So we may have an implementation to test each boundary + a default implementation. One can also decide to change the implementation of some functions to try subgradient for instance.

We can do that quite easily only if we stop testing the column generation as a black box.

[rrsadykov]

Thanks, Guillaume,

I will study the interface for the tree search, I forgot it already.

Normally, for an API there should be at least two uses. We can say here that it will be used for tests for the default implementation. Thus it is ok for me.