Multi-objective ensemble API

[mfeurer]

On a high-level, this PR adds

• API to pass custom ensemble classes to Auto-sklearn,
• API updates for use multi-objective ensembles, and
• possibility to use a metric that requires access to X.

On a lower level, this PR also adds:

• Updates to the ensemble building module:
  • New functionality to retrieve the identifiers and weights of an ensemble.
  • Because of that I was able to improve type definitions for the ensemble building submodule. More concretely, there are no more ensemble files exempt from type checking.
  • pass a different seed to the ensemble builder every time it is called
  • make candidate selection (n_best) aware of multiple objectives
  • pass information about all available runs to the ensemble class
  • available ensemble classes are now shown in the docs
  • improved the docstring of ensemble selection
• estimators API
  • deprecate ensemble_size argument
• examples
  • add a pareto-front plot to the multi-objective Auto-sklearn example
• tests
  • add a new case for AutoML tests that checks for multi-objective optimization

[codecov]

Codecov Report

Merging #1485 (d8a863c) into development (4b21134) will decrease coverage by 0.41%.
The diff coverage is 75.08%.

❗ Current head d8a863c differs from pull request most recent head 0ba05e9. Consider uploading reports for the commit 0ba05e9 to get more accurate results

@@               Coverage Diff               @@
##           development    #1485      +/-   ##
===============================================
- Coverage        84.22%   83.81%   -0.42%     
===============================================
  Files              151      152       +1     
  Lines            11488    11662     +174     
  Branches          1994     2037      +43     
===============================================
+ Hits              9676     9774      +98     
- Misses            1279     1339      +60     
- Partials           533      549      +16     

[eddiebergman]

I assume this is mostly the same as the other code I saw, just brief question on RandomState

[eddiebergman]

It's not very clear why this should be only one scikit learn ensemble expected but I assume it's because of the default parameter for ensemble selection.

It also seems this test is very specific to this single case (fitted multiobjective iris classifier).

I had the same problem when considering cases and my solution was just to have general tests. We can just push this through for now, knowing it will break if we add any other cases with the "multiobjective" tag.

The longer term solution to this, I have a few ideas:

• We just use make_automl, make_dataset and construct the specific automl instance for this test where the specifics that are being tested are directly evident in the test. Same as old way of doing things and leads to no caching but at least all relevant setup assumptions are stated clearly in test.
• We encode these extra specifics somehow:
  • The case just returns extra info

  def case_classifier_fitted_holdout_multiobjective(...):
      ...
      return (model, extra_info)

  • The extra specifics are directly saved and access on the model object. This does add a lot more introspection capabilites to the model which may be helpful for future additions

Happy to hear any other ideas on this though, I admit the caching solution as is, is not perfect for this reason, but it does allow the tests to be a lot more modular.

[mfeurer]

It's not very clear why this should be only one scikit learn ensemble expected but I assume it's because of the default parameter for ensemble selection.

Correct.

It also seems this test is very specific to this single case (fitted multiobjective iris classifier).

Correct as well.

I had the same problem when considering cases and my solution was just to have general tests. We can just push this through for now, knowing it will break if we add any other cases with the "multiobjective" tag.

Very glad you see it this way.

Happy to hear any other ideas on this though

Would we for the 2nd idea check whether the AutoML was built on iris and then use it? Besides that, could we maybe add a filter on which dataset(s) were used to build the AutoML system?

[eddiebergman]

Would we for the 2nd idea check whether the AutoML was built on iris and then use it? Besides that, could we maybe add a filter on which dataset(s) were used to build the AutoML system?

Yup it's definitely possible, the easiest way is to just do so in the test itself, i.e. if extra_info["dataset"] != "iris": pass but I'm not the biggest fan of the solution.

The overarching problem is that you can't use @parametrize and @tags together, i.e. you can't associate a parameter with a tag.

I guess my prefered solution is to include more general things in the extra_info or encode it on the model, meaning the tests don't have to do any filtering.

extra_info = {
    "X_shape": X.shape,
    "y_shape": y.shape,
    "labels": ...
}
return (automl, extra_info)

It's not the cleanest but at least it means this test could theoretically work for any other "multiobjective" tagged case, as long as it provides the necessary extra_info.