Leaderboard #1185

eddiebergman · 2021-07-27T14:27:28Z

Leaderboard functionality

Show's each model's performance on the training set as optimized by SMAC

Main Changes

Add's a function to the AutoSklearnEstimator class called leaderboard with signature

def leaderboard(
    self,
    detailed: bool = False,
    ensemble_only: bool = True,
    top_k: Union[int, Literal['all']] = 'all',
    sort_by: str = 'cost',
    sort_order: Literal['auto', 'ascending', 'descending'] = 'auto',
    include: Optional[Union[str, Iterable[str]]] = None
) -> pd.DataFrame:

Note

This is a clean branch based off the PR at (#1177) due to git difing issues once merged with an updated development branch.

Still requires testing, only works for classification

For the autoML models to be useable for the entire session without training, they require a session scoped tmp_dir. I tried to figure out how to make the tmp_dir more dynamic but documentation seems to imply that the scope is set at *function definition*, not on function call. This means either call the _tmp_dir and manually clean up or just duplicate the tmp_dir function but aptly named for session scope. It's a bit ugly but couldn't find an alternative.

Doesn't populate the request.module object if requesting from a session scope. For now module will have to do

Generating the sphinx examples causes output to be generated in doc/examples. Not sure if this should be pushed considering docs/build is not.

Found a bug: /home/skantify/code/auto-sklearn/examples/20_basic/example_multilabel_classification.py failed to execute correctly: Traceback (most recent call last): File "/home/skantify/code/auto-sklearn/examples/20_basic/example_multilabel_classification.py", line 61, in <module> print(automl.leaderboard()) File "/home/skantify/code/auto-sklearn/autosklearn/estimators.py", line 738, in leaderboard model_runs[model_id]['ensemble_weight'] = self.automl_.ensemble_.weights_[i] KeyError: 2

There is a discrepency between identifiers used by SMAC and and the identifiers used by an Ensemble class. SMAC uses `config_id` which is available for every run of SMAC while Ensemble uses `model_id == num_run` which is only available in runinfo.additional_info. However, this is not always included in additional_info, nor is additional_info garunteed to exist. Therefore the only garunteed unique identifier for models are `config_id`s which can confuse the user if they wise to interact with the ensembler.

There are two indexes that can be used, SMAC uses `config_id` and asklearn uses `num_run`, these are not garunteed to be equal and also `num_run` is not always present. As the user should not care that there is possible 2 indexes for models, made the choice to show `config_id` as this allows displaying info on failed runs. An alternative to show asklearn's `num_run` index is just to exclude any failed runs from showing up in the leaderboard.

Any runs which do not provide a model_id == num_run are essentially discarded. This hsould change in the future but the fix is outside the scope of the PR.

Once Python 3.7 is dropped, we can drop typing_extensions

Co-authored-by: Matthias Feurer <[email protected]>

codecov · 2021-07-27T15:01:08Z

Codecov Report

Merging #1185 (f36eec4) into development (611cf5c) will decrease coverage by 0.24%.
The diff coverage is 36.23%.

@@               Coverage Diff               @@
##           development    #1185      +/-   ##
===============================================
- Coverage        85.86%   85.62%   -0.25%     
===============================================
  Files              138      138              
  Lines            10790    10857      +67     
===============================================
+ Hits              9265     9296      +31     
- Misses            1525     1561      +36

Impacted Files	Coverage Δ
autosklearn/automl.py	`85.00% <ø> (ø)`
autosklearn/estimators.py	`73.76% <33.33%> (-19.67%)`	⬇️
autosklearn/ensembles/ensemble_selection.py	`67.80% <100.00%> (+0.44%)`	⬆️
...ine/components/classification/gradient_boosting.py	`91.30% <0.00%> (-0.87%)`	⬇️
autosklearn/ensemble_builder.py	`77.17% <0.00%> (+0.40%)`	⬆️
..._preprocessing/select_percentile_classification.py	`89.65% <0.00%> (+1.72%)`	⬆️
...ature_preprocessing/select_rates_classification.py	`87.32% <0.00%> (+4.22%)`	⬆️
...eline/components/feature_preprocessing/fast_ica.py	`97.82% <0.00%> (+6.52%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 611cf5c...f36eec4. Read the comment docs.

eddiebergman added 30 commits July 27, 2021 15:27

Implemented def leaderboard

fa06a62

Still requires testing, only works for classification

Fixed some bugs

2712a32

Updated function with new params

7109f97

Cleaned info gathering a little

cff371b

Identifies if classifier or regressor models

fa38597

Implemented sort_by param

3ac0ab7

Added ranking column

72bef65

Implemented ensemble_only param for leadboard

8a80ef4

Implemented param top_k

25948c7

flake8'd

c52a91b

Created fixtures for use with test_leaderboard

0ba9f2e

Can't make tmp_dir for session scope fixtures

0241b1f

Doesn't populate the request.module object if requesting from a session scope. For now module will have to do

Reverted back, models trained in test

57cc340

Moved leaderboard AutoML -> AutoSklearnEstimator

8bb2436

Added fuzzing test for test_leaderboard

dbf06a8

Added tests for leaderboard, added sort_order

1dbecbf

Removed Type Final to support python 3.7

37b558e

Removed old solution to is_classication for leaderboard

8953ff7

I should really force pre-commit to run before commit (flake8 fixes)

acf4f2b

More occurences of Literal

455fc0c

Readded Literal but imported from typing_extensions

4a33a69

Fixed docstring for sphinx

5bb51ea

Added make command to build html without running examples

704fe5f

Added doc/examples to gitignore

a9b396f

Generating the sphinx examples causes output to be generated in doc/examples. Not sure if this should be pushed considering docs/build is not.

Cleaned up _str_ of EnsembleSelection

fef0c11

Removed Literal again as typing_extensions is external module

819e5ee

eddiebergman and others added 12 commits July 27, 2021 15:28

Switched to model_id as primary id

a658979

Any runs which do not provide a model_id == num_run are essentially discarded. This hsould change in the future but the fix is outside the scope of the PR.

pre-commit flake8 fix

4e7ba41

Logger gives warning if sort_by is not in columns asked for

0b5f448

Moved column types to static method

292898d

Fixed rank to be based on cost

8c8f41a

Fixed so model_id can be requested, even though it always exists

ff261d6

Fixed so rank can be calculated even if cost not requested

c3b2caf

Readded Literal and included typing_extension dependancy

9af9d8b

Once Python 3.7 is dropped, we can drop typing_extensions

Changed default sort_order to 'auto'

935da0c

Changed leaderboard columns to be static attributes

5de2e63

Update budget doc

407dcc5

Co-authored-by: Matthias Feurer <[email protected]>

flake8'd

f36eec4

eddiebergman marked this pull request as ready for review July 27, 2021 14:27

eddiebergman requested a review from mfeurer July 27, 2021 14:29

mfeurer approved these changes Jul 27, 2021

View reviewed changes

mfeurer merged commit 6231b1c into automl:development Jul 27, 2021

github-actions bot pushed a commit that referenced this pull request Jul 27, 2021

Eddie Bergman: Leaderboard (#1185)

c591ff8

eddiebergman deleted the leaderboard_fresh branch July 28, 2021 08:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Leaderboard #1185

Leaderboard #1185

eddiebergman commented Jul 27, 2021 •

edited

Loading

codecov bot commented Jul 27, 2021 •

edited

Loading

Leaderboard #1185

Leaderboard #1185

Conversation

eddiebergman commented Jul 27, 2021 • edited Loading

Leaderboard functionality

Main Changes

Note

codecov bot commented Jul 27, 2021 • edited Loading

Codecov Report

eddiebergman commented Jul 27, 2021 •

edited

Loading

codecov bot commented Jul 27, 2021 •

edited

Loading