Ranks within hub falling out of sync if display_timing is on

[tvalenciaz]

If display_timing is on and one of the ranks>0 has more local scenarios than rank 0, the gather functions will fall out of sync. Rank 0 will interpret the additional scenarios' instance creation time as its optimization sense and raise a spurious warning saying not all scenarios have the same optimization sense (maximize or minimize).
Pull Request fixing this forthcoming.

mpi-sppy/mpisppy/spbase.py

Lines 270 to 290 in 06fbefe

	for sname in self.local_scenario_names:
	instance_creation_start_time = time.time()
	s = self.scenario_creator(sname, **scenario_creator_kwargs)
	self.local_scenarios[sname] = s
	if self.multistage:
	#Checking that the scenario can have an associated leaf node in all_nodenames
	stmax = np.argmax([nd.stage for nd in s._mpisppy_node_list])
	if(s._mpisppy_node_list[stmax].name)+'_0' not in self.all_nodenames:
	raise RuntimeError("The leaf node associated with this scenario is not on all_nodenames"
	f"Its last non-leaf node {s._mpisppy_node_list[stmax].name} has no first child {s._mpisppy_node_list[stmax].name+'_0'}")
	if self.options.get("display_timing", False):
	instance_creation_time = time.time() - instance_creation_start_time
	all_instance_creation_times = self.mpicomm.gather(
	instance_creation_time, root=0
	)
	if self.cylinder_rank == 0:
	aict = all_instance_creation_times
	print("Scenario instance creation times:")
	print(f"\tmin={np.min(aict):4.2f} mean={np.mean(aict):4.2f} max={np.max(aict):4.2f}")
	self.scenarios_constructed = True

[tvalenciaz]

Fixed by PR #308