mlos_bench: error handling improvements

[bpkroth]

Sometimes user scripts don't return a score value, even though they exit 0 (indicating SUCCESS).

In that case we can do a couple of things:

• abort immediately in order to notify the experimenter and let them figure out what to do
• assume it's a bad config and that's why the benchmark aborted early
  • in which case we should fabricate a "fake" score that looks "bad" (i.e., much worse than any we've actually recorded with a good config) so that the optimizer learns that this is an infeasible region (there are already TODO markers in the code to implement this)
• some cominbation of the two
  for instance, tolerate no more than N "bad" configs in a row before we assume its a script error and abort entirely to notify the user that they should manually inspect and deal with things

[bpkroth]

@eujing

[bpkroth]

One thing that might make this easier to implement is if we clearly separated the phases of "setup" (e.g., basic system preparation) vs. "configure" (e.g., configure the target system with the tunables).

That way, if "setup" failed, we could alert that the script was the problem, whereas if "configure" failed, we could inform the optimizer that it was a bad region.

See Also: #498