Incomplete optimisation data with CPLEX solver

[ghost]

Hi,

I've been trying out Pyomo for the past few days and am very pleased with it. Today, however, I noticed I am unable to retrieve all the optimisation data that I would normally be able to obtain from CPLEX (via the command line, matlab, python bindings, etc).

In particular, I'm referring to the best bound, final relative MIP gap, deterministic computation time, and the feasible solution pool.

I suppose I could get this information by writing the problem to a file and doing what I need to do externally (without using Pyomo for those steps). The problem with this approach is that it would require me to develop two closely-related scripts (one in Pyomo and one using some other technology) to know which variable is which, therefore limiting Pyomo's usefulness.

Alternatively, I could get that information from the cplex object (used in the official cplex python scripts) since I'm using cplex_direct, but I can't seem to find that object.

This issue has also been identified in: https://groups.google.com/g/pyomo-forum/c/61VcjzUrzds

Is there a simpler way to access this information? If not, are there plans to complement the information provided through the solve method?

Any help is much appreciated.

[michaelbynum]

I believe the best bound and wall clock time are loaded into the results object with the cplex_direct interface. If you want to access the underlying cplex object, you will have to use the cplex_persistent interface. We currently only have one solver interface that supports solution pools (the gurobi_persistent interface), but it should be easy to add to the cplex_persistent interface.

[ghost]

Thank you michaelbynum.

I haven't tried using the cplex_persistent interface but I will soon.

Are the 'Lower bound' and 'Higher bound' fields in the solution object the best bound and incumbent solution values (respectively, and vice-versa, depending on the optimisation sense)? They are always the same but I've only tested very simple problems for which CPLEX and GLPK always find an optimal solution (i.e., there would be no difference anyway). I plan on testing more complicated problems in the coming days to check this.

On a more general note, where can I find detailed information about the solution objects other than by going through Pyomo's source code? They seem to be different depending on the solver, which makes sense to some extent, but I can't seem to find more information about them. It would be good to have this documented for those of us who are not Python experts, if it isn't already.

[michaelbynum]

Yes, results.problem.lower_bound and results.problem.upper_bound provide the best bound and best incumbent, depending on objective sense.

We are currently working on cleaning up the solver interfaces and results objects, but it is taking some time.

[jsiirola]

@michaelbynum, I was recently digging into the CPLEX.py interface, and it appears that the parser is actually setting the lower_bound == upper_bound in the results object any time that CPLEX returns "optimal":

pyomo/pyomo/solvers/plugins/solvers/CPLEX.py

Lines 810 to 818 in 13cd4f8

	elif tokens[0].startswith("solutionStatusString"):
	solution_status = ((" ".join(tokens).split('=')[1]).strip()).lstrip("\"").rstrip("\"")
	if solution_status in ["optimal", "integer optimal solution", "integer optimal, tolerance"]:
	soln.status = SolutionStatus.optimal
	soln.gap = 0.0
	results.problem.lower_bound = soln.objective['__default_objective__']['Value']
	results.problem.upper_bound = soln.objective['__default_objective__']['Value']
	if "integer" in solution_status:
	mip_problem=True

This strikes me as a bug in the LP (and maybe NL) interface.

[michaelbynum]

I agree that this is a bug. I believe the direct and persistent interfaces handle this correctly, but better testing of this is definitely warranted.

[bknueven]

I agree with @michaelbynum that the cplex direct/persistent interfaces are doing the right thing with regards to bounds. See: https://www.ibm.com/support/knowledgecenter/SSSA5P_12.8.0/ilog.odms.cplex.help/refpythoncplex/html/cplex._internal._subinterfaces.MIPSolutionInterface-class.html

pyomo/pyomo/solvers/plugins/solvers/cplex_direct.py

Lines 695 to 705 in 40187a8

	if cpxprob.solution.get_solution_type() != cpxprob.solution.type.none:
	if (cpxprob.variables.get_num_binary() + cpxprob.variables.get_num_integer()) == 0:
	self.results.problem.upper_bound = cpxprob.solution.get_objective_value()
	self.results.problem.lower_bound = cpxprob.solution.get_objective_value()
	elif cpxprob.objective.get_sense() == cpxprob.objective.sense.minimize:
	self.results.problem.upper_bound = cpxprob.solution.get_objective_value()
	self.results.problem.lower_bound = cpxprob.solution.MIP.get_best_objective()
	else:
	assert cpxprob.objective.get_sense() == cpxprob.objective.sense.maximize
	self.results.problem.upper_bound = cpxprob.solution.MIP.get_best_objective()
	self.results.problem.lower_bound = cpxprob.solution.get_objective_value()