AutoSklearnClassifier vs AutoSklearn2Classifier (parameters): "scoring_functions" and "initial_configurations_via_metalearning"

[guilhermefarto]

Good afternoon.

I'm implementing a project that compares traditional Machine Learnind and AutoML-based metrics.

I verified that there are parameters (arguments) that exist in AutoSklearnClassifier, but that do not exist in the implementation of AutoSklearn2Classifier.

One of the most important parameters that I use is "scoring_functions", which allows me to pass a list of objects of the Scorer type so that, when returning from the fit method, I can analyze the results of the evaluated metrics.

How can I make use of the "scoring_functions" and initial_configurations_via_metalearning parameters when implementing AutoSklearn2Classifier?

Why don't we have the same AutoSklearnClassifier parameters available in AutoSklearn2Classifier (at least the parameters linked to metrics)?

AutoSklearnClassifier

  estimator_ensemble1 = AutoSklearnClassifier(
    seed=42,
    ensemble_size=1, ensemble_nbest=50,
    **initial_configurations_via_metalearning=0,**
    time_left_for_this_task=60*MINUTES,
    per_run_time_limit=round((60*MINUTES)/10),
    memory_limit=10240, # 3072
    n_jobs=n_jobs,
    resampling_strategy='cv', resampling_strategy_arguments={'folds': cv},
    **scoring_functions=scoring_functions_classifier,**
    metric=refit_scorer_classifier,
    disable_evaluator_output=False)

AutoSklearn2Classifier

  estimator_ensemble1 = AutoSklearn2Classifier(
    seed=42,
    ensemble_size=1, ensemble_nbest=50,
    time_left_for_this_task=60*MINUTES,
    memory_limit=10240, # 3072
    n_jobs=n_jobs,
    metric=refit_scorer_classifier,
    disable_evaluator_output=False)

Thanks

[mfeurer]

Hi @guilhermefarto, Thanks a lot for raising this issue.

Why don't we have the same AutoSklearnClassifier parameters available in AutoSklearn2Classifier (at least the parameters linked to metrics)?

Some of these are automatically set by the AutoSklearn2Classifier based on the dataset characteristics, such as the resampling strategy and the meta-learning.

The scoring functions was an oversight when implementing this new feature, they should be available to the AutoSklearn2Classifier as well. Would you like to create a PR to add them?

[mfeurer]

This was fixed via #1084 and will be available in the next release. Thank you very much for reporting this @guilhermefarto.