More aircond

doc/src/examples.rst

@@ -392,3 +392,88 @@ previous methods:
 
 We see that, for this toy example, the L-shaped method has converged to the
 optimal solution within just 10 iterations.
+
+
+aircond
+-------
+
+This is fairly complicated example because it is multi-stage and the
+model itself offers a lot of flexibility.  The aircond example is
+unusual in that the model file, ``aircond.py``, lives in
+``mpisppy.tests.examples`` directory. Scripts and bash files that use
+it live in ``examples.aircond``.  A good place to start is the
+``aircond_cylinders.py`` file that starts with some functions that
+support the main program. The main program makes use of the command
+line parsers provided with the library supplemented by arguments
+provided by the aircond reference model using the line
+
+::
+
+  parser = aircond.inparsers_adder(parser)
+
+
+ The ``args`` obtained by the parser are passed directly to the vanilla hub
+ and spoke creator which knows how to use the arguments from the ``baseparsers``.
+ The arguments unique to aircond are processed by the ``create_kwargs`` function
+ in the reference model file.
+
+ A simple example that uses a few of the options is shown in ``aircond_zhat.bash``, which
+ also calls the ``xhat4xhat`` program to estimate confidence intervals for the solution
+ obtained.
+
+
+ hydro
+ -----
+
+Hydro is a three stage example that was originally coded in PySP and we make extensive use
+of the PySP files. Unlike farmer and aircond where the scenario data are created from distributions,
+for this problem the scenario data are provided in files.
+
+Using PySPModel
+^^^^^^^^^^^^^^^
+In the file ``hydro_cylinders_pysp.py`` the lines
+
+::
+
+   from mpisppy.utils.pysp_model import PySPModel
+   ...
+   hydro = PySPModel("./PySP/models/", "./PySP/nodedata/")
+
+cause an object called ``hydro`` to be created that has the methods needed by vanilla and the hub and
+spoke creators as can be seen in the ``main`` function of ``hydro_cylinders_pysp.py``.
+
+
+Not using PySPModel
+^^^^^^^^^^^^^^^^^^^
+
+In the file ``hydro_cylinders.py`` the file ``hydro.py`` is imported because it provides the functions
+needed by vanilla hub and spoke creators.
+
+
+netdes
+------
+
+This is a very challenging network design problem, which has many instances each defined by a data file.
+For this problem, cross scenario cuts are helpful
+so the use of that spoke is illustrated in ``netdes_cylinders.py``.  
+
+sslp
+----
+
+This is a classic problem from Ntaimo and Sen with data in PySP format
+so the driver code (e.g., ``sslp_cylinders.py`` that makes use of ``sslp.py``) is somewhat similar to the
+hydro example except sslp is simpler because it is just two stages.
+
+UC
+--
+
+This example uses the ``egret`` package for the underlying unit commitment model
+and reads PySP format data using the ``pyomo`` dataportal. Data files for a variety
+of numbers of scenarios are provided.
+
+sizes
+-----
+
+The sizes example (Jorjani et al, IJPR, 1999) is a two-stage problem with general integers in each stage. The file
+``sizes_cylinders.py`` is the usual cylinders driver. There are other examples in the directory, such
+as ``sizes_demo.py``, which provides an example of serial execution (no cylinders).

doc/src/hubs.rst

@@ -56,3 +56,26 @@ APH
 
 The implementation of Asynchronous Projective Hedging is described in a
 forthcoming paper.
+
+Hub Convergers
+--------------
+
+Execution of mpi-sppy programs can be terminated based on a variety of criteria.
+The simplest is hub iteration count and the most common is probably relative
+gap between upper and lower bounds. It is also possible to terminate
+based on convergence within the hub; furthermore, convergence metrics within
+the hub can be helpful for tuning algorithms.
+
+Unfortunately, the word "converger" is also used to describe spokes that return bounds
+for the purpose of measuring overall convergence (as opposed to convergence within the hub
+algorithm.)
+
+..
+   The scenario decomposition methods (PH and APH) allow for user written
+   convergence metrics as plug-ins. A pattern that can be followed is shown
+   in the farmer example. The ``farmer_cylinders.py`` file can have::
+
+   from mpisppy.convergers.norm_rho_converger import NormRhoConverger
+   ...
+   xxxxxx
+

doc/src/spokes.rst

@@ -72,8 +72,8 @@ solutions can be tried before receiving new x values from the hub.
 This spoke also supports multistage problems. It does not try every subproblem, but
 shuffles the scenarios and loops over the shuffled list.
 At each step, it takes the first-stage solution specified by a scenario, 
- and then uses the scenarios that follows in the shuffled loop to get the 
- values of the non-first-stage variables that were not fixed before.
+and then uses the scenarios that follows in the shuffled loop to get the 
+values of the non-first-stage variables that were not fixed before.
 
 slam_heuristic
 ^^^^^^^^^^^^^^
@@ -97,7 +97,7 @@ spoke_sleep_time
 
 This is an advanced topic and rarely encountered.
 In some settings, particularly with small sub-problems, it is possible for
-ranks within spokes to become ``of of sync.''  The most common manifestation of this
+ranks within spokes to become of of sync.  The most common manifestation of this
 is that some ranks do not see the kill signal and sit in a busy-wait I/O loop
 until something external kills them; but it can also be the case that Lagrangian
 bound spokes start operating on data from different hub iterations; they should notice

examples/aircond/aircond_cylinders.py

@@ -152,8 +152,8 @@ def main():
 
     ScenCount = np.prod(BFs)
     #ScenCount = _get_num_leaves(BFs)
-    scenario_creator_kwargs = {"branching_factors": BFs,
-                               "start_ups":args.start_ups}
+    sc_options = {"args": args}
+    scenario_creator_kwargs = aircond.kw_creator(sc_options)
 
     all_scenario_names = [f"scen{i}" for i in range(ScenCount)] #Scens are 0-based
     # print(all_scenario_names)
@@ -203,16 +203,18 @@ def main():
     wheel = WheelSpinner(hub_dict, list_of_spoke_dict)
     wheel.spin()
 
+    fname = 'aircond_cyl_nonants.npy'
     if wheel.global_rank == 0:
         print("BestInnerBound={} and BestOuterBound={}".\
               format(wheel.BestInnerBound, wheel.BestOuterBound))
-
-
+        if write_solution:
+            print(f"Writing first stage solution to {fname}")
+    # all ranks need to participate because only the winner will write
     if write_solution:
         wheel.write_first_stage_solution('aircond_first_stage.csv')
         wheel.write_tree_solution('aircond_full_solution')
-        wheel.write_first_stage_solution('aircond_cyl_nonants.npy',
-                first_stage_solution_writer=first_stage_nonant_npy_serializer)
+        wheel.write_first_stage_solution(fname,
+                    first_stage_solution_writer=first_stage_nonant_npy_serializer)
 
 if __name__ == "__main__":
     main()

examples/aircond/aircond_slow.bash

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+SOLVERNAME="cplex"
+
+# get an xhat 
+# xhat output file name is hardwired
+mpiexec --oversubscribe -np 3 python -m mpi4py aircond_cylinders.py --bundles-per-rank=0 --max-iterations=100 --default-rho=1 --solver-name=${SOLVERNAME} --branching-factors 4 3 2 --Capacity 200 --QuadShortCoeff 0.3 --start-ups --BeginInventory 50 --rel-gap 0.01
+

examples/aircond/aircond_zhat.bash

@@ -1,10 +1,11 @@
 #!/bin/bash
 
-SOLVERNAME="gurobi_persistent"
+#SOLVERNAME="gurobi_persistent"
+SOLVERNAME="cplex"
 
 # get an xhat
 # xhat output file name is hardwired
-mpiexec --oversubscribe -np 9 python -m mpi4py aircond_cylinders.py  --branching-factors 5 4 3 --bundles-per-rank=0 --max-iterations=50 --default-rho=1.0 --solver-name=${SOLVERNAME} --rel-gap=0.001 --abs-gap=2 --start-ups
+mpiexec --oversubscribe -np 9 python -m mpi4py aircond_cylinders.py  --branching-factors 5 4 3 --bundles-per-rank=0 --max-iterations=50 --default-rho=1.0 --solver-name=${SOLVERNAME} --rel-gap=0.001 --abs-gap=2
 
-#echo "starting zhat4xhat"
-#python -m mpisppy.confidence_intervals.zhat4xhat mpisppy.tests.examples.aircond aircond_cyl_nonants.npy --solver-name ${SOLVERNAME} --branching-factors 4 3 2 --num-samples 5 --confidence-level 0.95
+echo "starting zhat4xhat"
+python -m mpisppy.confidence_intervals.zhat4xhat mpisppy.tests.examples.aircond aircond_cyl_nonants.npy --solver-name ${SOLVERNAME} --branching-factors 4 3 2 --num-samples 10 --confidence-level 0.95

examples/aircond/aircond_zhatII.bash

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+SOLVERNAME="cplex"
+SAMPLES=3
+
+# get an xhat 
+# xhat output file name is hardwired
+mpiexec --oversubscribe -np 3 python3 -m mpi4py aircond_cylinders.py --bundles-per-rank=0 --max-iterations=50 --default-rho=1 --solver-name=${SOLVERNAME} --branching-factors 4 3 2 --Capacity 200 --QuadShortCoeff 0.3 --start-ups --BeginInventory 50
+
+echo "starting zhat4xhat with ${SAMPLES} samples"
+python -m mpisppy.confidence_intervals.zhat4xhat mpisppy.tests.examples.aircond aircond_cyl_nonants.npy --solver-name ${SOLVERNAME} --branching-factors 4 3 2 --num-samples ${SAMPLES} --confidence-level 0.95 --Capacity 200 --QuadShortCoeff 0.3 --start-ups --BeginInventory 50

examples/sizes/sizes_demo.py

@@ -181,14 +181,15 @@
         scenario_creator,
         scenario_denouement,
         scenario_creator_kwargs={"scenario_count": ScenCount},
+        extensions=MinMaxAvg,
+        PH_converger=None,
+        rho_setter=None,
     )
     ph.options["PHIterLimit"] = 3
 
     from mpisppy.extensions.avgminmaxer import MinMaxAvg
     options["avgminmax_name"] =  "FirstStageCost"
-    conv, obj, bnd = ph.ph_main(extensions=MinMaxAvg,
-                                PH_converger=None,
-                                rho_setter=None)
+    conv, obj, bnd = ph.ph_main()
 
 
 

mpisppy/confidence_intervals/multi_seqsampling.py

@@ -178,6 +178,8 @@ def _gap_estimators_with_independent_scenarios(self, xhat_k, nk,
                  (TBD: drop this arg and just use the function in refmodel)
         Returns:
             Gk, Sk (float): mean and standard devation of the gap estimate
+        Note:
+            Seed management is mainly in the form of updates to SeedCount
 
         """
         ama_options = self.options.copy()
@@ -187,13 +189,14 @@ def _gap_estimators_with_independent_scenarios(self, xhat_k, nk,
             ama_options['EF_solver_options']= self.solver_options
         ama_options['num_scens'] = nk
         ama_options['_mpisppy_probability'] = 1/nk #Probably not used
+        ama_options['start_seed'] = self.SeedCount
 
         pseudo_branching_factors = [nk]+[1]*(self.numstages-2)
         ama_options['branching_factors'] = pseudo_branching_factors
         ama = amalgomator.Amalgomator(options=ama_options, 
                                       scenario_names=estimator_scenario_names,
                                       scenario_creator=self.refmodel.scenario_creator,
-                                      kw_creator=self._kw_creator_without_seed,
+                                      kw_creator=self.refmodel.kw_creator,
                                       scenario_denouement=scenario_denouement)
         ama.run()
         #Optimal solution of the approximate problem

mpisppy/confidence_intervals/sample_tree.py

@@ -122,7 +122,13 @@ def _create_amalgomator(self):
         if self.solver_options is not None:
             ama_options['EF_solver_options']= self.solver_options
         ama_options['num_scens'] = self.numscens
+        if "start_seed" not in ama_options:
+            ama_options["start_seed"] = self.seed
         ama_options['_mpisppy_probability'] = 1/self.numscens #Probably not used
+        # TBD: better options flow (Dec 2021)
+        if "branching_factors" not in ama_options:
+            if "kwargs" in ama_options:
+                ama_options["branching_factors"] = ama_options["kwargs"]["branching_factors"]
 
         scen_names = self.refmodel.scenario_names_creator(self.numscens,
                                                           start=self.seed)