Skip to content

Commit

Permalink
Rohit Agarwal: Fix rank for metrics for which greater value is not go…
Browse files Browse the repository at this point in the history
…od (#1079)
  • Loading branch information
Github Actions committed Feb 16, 2021
1 parent 2ea9cd1 commit c3d89e2
Show file tree
Hide file tree
Showing 68 changed files with 307 additions and 349 deletions.
Binary file not shown.
Binary file not shown.
Binary file modified development/_images/sphx_glr_example_pandas_train_test_001.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified development/_images/sphx_glr_example_pandas_train_test_thumb.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion development/_modules/autosklearn/estimators.html
Original file line number Diff line number Diff line change
Expand Up @@ -889,7 +889,7 @@ <h1>Source code for autosklearn.estimators</h1><div class="highlight"><pre>
</p>
<p>
&copy; Copyright 2014-2019, Machine Learning Professorship Freiburg.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.0.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.1.<br/>
</p>
</div>
</footer>
Expand Down
2 changes: 1 addition & 1 deletion development/_modules/autosklearn/experimental/askl2.html
Original file line number Diff line number Diff line change
Expand Up @@ -534,7 +534,7 @@ <h1>Source code for autosklearn.experimental.askl2</h1><div class="highlight"><p
</p>
<p>
&copy; Copyright 2014-2019, Machine Learning Professorship Freiburg.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.0.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.1.<br/>
</p>
</div>
</footer>
Expand Down
2 changes: 1 addition & 1 deletion development/_modules/autosklearn/metrics.html
Original file line number Diff line number Diff line change
Expand Up @@ -596,7 +596,7 @@ <h1>Source code for autosklearn.metrics</h1><div class="highlight"><pre>
</p>
<p>
&copy; Copyright 2014-2019, Machine Learning Professorship Freiburg.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.0.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.1.<br/>
</p>
</div>
</footer>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -559,7 +559,7 @@ <h1>Source code for autosklearn.pipeline.components.base</h1><div class="highlig
</p>
<p>
&copy; Copyright 2014-2019, Machine Learning Professorship Freiburg.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.0.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.1.<br/>
</p>
</div>
</footer>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,7 @@ <h1>Source code for autosklearn.pipeline.components.classification</h1><div clas
</p>
<p>
&copy; Copyright 2014-2019, Machine Learning Professorship Freiburg.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.0.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.1.<br/>
</p>
</div>
</footer>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ <h1>Source code for autosklearn.pipeline.components.feature_preprocessing</h1><d
</p>
<p>
&copy; Copyright 2014-2019, Machine Learning Professorship Freiburg.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.0.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.1.<br/>
</p>
</div>
</footer>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,7 @@ <h1>Source code for autosklearn.pipeline.components.regression</h1><div class="h
</p>
<p>
&copy; Copyright 2014-2019, Machine Learning Professorship Freiburg.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.0.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.1.<br/>
</p>
</div>
</footer>
Expand Down
2 changes: 1 addition & 1 deletion development/_modules/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ <h1>All modules for which code is available</h1>
</p>
<p>
&copy; Copyright 2014-2019, Machine Learning Professorship Freiburg.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.0.<br/>
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.1.<br/>
</p>
</div>
</footer>
Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ Get the Score of the final ensemble
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 0 minutes 18.521 seconds)
**Total running time of the script:** ( 0 minutes 17.562 seconds)


.. _sphx_glr_download_examples_20_basic_example_multilabel_classification.py:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ Get the Score of the final ensemble
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 1 minutes 56.918 seconds)
**Total running time of the script:** ( 1 minutes 56.816 seconds)


.. _sphx_glr_download_examples_20_basic_example_regression.py:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@

Computation times
=================
**04:15.407** total execution time for **examples_20_basic** files:
**04:17.262** total execution time for **examples_20_basic** files:

+-------------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_20_basic_example_classification.py` (``example_classification.py``) | 01:59.969 | 0.0 MB |
| :ref:`sphx_glr_examples_20_basic_example_classification.py` (``example_classification.py``) | 02:02.885 | 0.0 MB |
+-------------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_20_basic_example_regression.py` (``example_regression.py``) | 01:56.918 | 0.0 MB |
| :ref:`sphx_glr_examples_20_basic_example_regression.py` (``example_regression.py``) | 01:56.816 | 0.0 MB |
+-------------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_20_basic_example_multilabel_classification.py` (``example_multilabel_classification.py``) | 00:18.521 | 0.0 MB |
| :ref:`sphx_glr_examples_20_basic_example_multilabel_classification.py` (``example_multilabel_classification.py``) | 00:17.562 | 0.0 MB |
+-------------------------------------------------------------------------------------------------------------------+-----------+--------+
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ Get the Score of the final ensemble
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 1 minutes 55.919 seconds)
**Total running time of the script:** ( 1 minutes 58.881 seconds)


.. _sphx_glr_download_examples_40_advanced_example_calc_multiple_metrics.py:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ Get the Score of the final ensemble
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 0 minutes 15.920 seconds)
**Total running time of the script:** ( 0 minutes 19.177 seconds)


.. _sphx_glr_download_examples_40_advanced_example_feature_types.py:
Expand Down

Large diffs are not rendered by default.

31 changes: 17 additions & 14 deletions development/_sources/examples/40_advanced/example_metrics.rst.txt
Original file line number Diff line number Diff line change
Expand Up @@ -280,9 +280,10 @@ Second example: Use own accuracy metric
Accuracy score 0.937063 using accu
################################################################################
Use self defined error metric
[WARNING] [2021-02-16 06:44:31,458:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 06:44:32,305:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 06:44:32,931:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 17:59:56,760:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 17:59:57,574:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 17:59:58,179:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:00:02,081:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
Error rate -0.0629371 using error
Expand Down Expand Up @@ -366,16 +367,18 @@ Third example: Use own accuracy metric with additional argument
Accuracy score 0.937063 using accu_add
################################################################################
Use self defined error with additional argument
[WARNING] [2021-02-16 06:45:40,989:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 06:45:44,024:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 06:45:48,714:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 06:45:50,140:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 06:46:23,056:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 06:46:23,056:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 06:46:23,828:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 06:46:23,828:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 06:46:24,405:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 06:46:24,405:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:01:23,649:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:01:26,506:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:01:30,853:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:01:32,225:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:02:05,027:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:02:05,027:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:02:05,765:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:02:05,766:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:02:06,313:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:02:06,314:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:02:10,045:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
[WARNING] [2021-02-16 18:02:10,046:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.
Error rate 0.615385 using error_add
Expand All @@ -384,7 +387,7 @@ Third example: Use own accuracy metric with additional argument
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 4 minutes 37.876 seconds)
**Total running time of the script:** ( 5 minutes 0.171 seconds)


.. _sphx_glr_download_examples_40_advanced_example_metrics.py:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ Plot the ensemble performance

.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 1 minutes 56.114 seconds)
**Total running time of the script:** ( 1 minutes 56.137 seconds)


.. _sphx_glr_download_examples_40_advanced_example_pandas_train_test.py:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -357,7 +357,7 @@ splitting it on the first feature.
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 6 minutes 4.519 seconds)
**Total running time of the script:** ( 6 minutes 0.487 seconds)


.. _sphx_glr_download_examples_40_advanced_example_resampling.py:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,18 @@

Computation times
=================
**15:17.627** total execution time for **examples_40_advanced** files:
**15:51.578** total execution time for **examples_40_advanced** files:

+------------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_40_advanced_example_resampling.py` (``example_resampling.py``) | 06:04.519 | 0.0 MB |
| :ref:`sphx_glr_examples_40_advanced_example_resampling.py` (``example_resampling.py``) | 06:00.487 | 0.0 MB |
+------------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_40_advanced_example_metrics.py` (``example_metrics.py``) | 04:37.876 | 0.0 MB |
| :ref:`sphx_glr_examples_40_advanced_example_metrics.py` (``example_metrics.py``) | 05:00.171 | 0.0 MB |
+------------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_40_advanced_example_pandas_train_test.py` (``example_pandas_train_test.py``) | 01:56.114 | 0.0 MB |
| :ref:`sphx_glr_examples_40_advanced_example_calc_multiple_metrics.py` (``example_calc_multiple_metrics.py``) | 01:58.881 | 0.0 MB |
+------------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_40_advanced_example_calc_multiple_metrics.py` (``example_calc_multiple_metrics.py``) | 01:55.919 | 0.0 MB |
| :ref:`sphx_glr_examples_40_advanced_example_pandas_train_test.py` (``example_pandas_train_test.py``) | 01:56.137 | 0.0 MB |
+------------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_40_advanced_example_get_pipeline_components.py` (``example_get_pipeline_components.py``) | 00:27.279 | 0.0 MB |
| :ref:`sphx_glr_examples_40_advanced_example_get_pipeline_components.py` (``example_get_pipeline_components.py``) | 00:36.725 | 0.0 MB |
+------------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_40_advanced_example_feature_types.py` (``example_feature_types.py``) | 00:15.920 | 0.0 MB |
| :ref:`sphx_glr_examples_40_advanced_example_feature_types.py` (``example_feature_types.py``) | 00:19.177 | 0.0 MB |
+------------------------------------------------------------------------------------------------------------------+-----------+--------+
Original file line number Diff line number Diff line change
Expand Up @@ -301,16 +301,16 @@ Start Auto-sklearn
.. code-block:: none
auto-sklearn results:
Dataset name: 8b9cede7-7023-11eb-8aae-000d3a968487
Dataset name: f16d305f-7081-11eb-8a74-000d3a1505e7
Metric: accuracy
Best validation score: 0.978723
Number of target algorithm runs: 15
Number of successful target algorithm runs: 14
Number of target algorithm runs: 16
Number of successful target algorithm runs: 15
Number of crashed target algorithm runs: 0
Number of target algorithms that exceeded the time limit: 1
Number of target algorithms that exceeded the memory limit: 0
Accuracy score 0.951048951048951
Accuracy score 0.9440559440559441
Expand Down Expand Up @@ -342,7 +342,7 @@ line.
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 0 minutes 39.228 seconds)
**Total running time of the script:** ( 0 minutes 40.036 seconds)


.. _sphx_glr_download_examples_60_search_example_parallel_manual_spawning_cli.py:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -210,28 +210,28 @@ which means that it is automatically stopped once all computation is done.

.. code-block:: none
[ERROR] [2021-02-16 06:53:19,613:asyncio] _GatheringFuture exception was never retrieved
[ERROR] [2021-02-16 18:09:03,972:asyncio] _GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=CancelledError()>
asyncio.exceptions.CancelledError
auto-sklearn results:
Dataset name: a27fca04-7023-11eb-8aae-000d3a968487
Dataset name: 08cd4638-7082-11eb-8a74-000d3a1505e7
Metric: accuracy
Best validation score: 0.978723
Number of target algorithm runs: 17
Number of successful target algorithm runs: 16
Number of target algorithm runs: 15
Number of successful target algorithm runs: 15
Number of crashed target algorithm runs: 0
Number of target algorithms that exceeded the time limit: 1
Number of target algorithms that exceeded the time limit: 0
Number of target algorithms that exceeded the memory limit: 0
Accuracy score 0.958041958041958
Accuracy score 0.951048951048951
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 0 minutes 46.899 seconds)
**Total running time of the script:** ( 0 minutes 46.918 seconds)


.. _sphx_glr_download_examples_60_search_example_parallel_manual_spawning_python.py:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,8 @@ To use ``n_jobs_`` we must guard the code
Dataset name: breast_cancer
Metric: accuracy
Best validation score: 0.992908
Number of target algorithm runs: 47
Number of successful target algorithm runs: 42
Number of target algorithm runs: 48
Number of successful target algorithm runs: 43
Number of crashed target algorithm runs: 2
Number of target algorithms that exceeded the time limit: 3
Number of target algorithms that exceeded the memory limit: 0
Expand All @@ -127,7 +127,7 @@ To use ``n_jobs_`` we must guard the code
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 2 minutes 5.557 seconds)
**Total running time of the script:** ( 2 minutes 3.905 seconds)


.. _sphx_glr_download_examples_60_search_example_parallel_n_jobs.py:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -625,7 +625,7 @@ Fit a classifier using Random Search
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 1 minutes 51.289 seconds)
**Total running time of the script:** ( 1 minutes 50.441 seconds)


.. _sphx_glr_download_examples_60_search_example_random_search.py:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,7 @@ Get the Score of the final ensemble
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 1 minutes 56.253 seconds)
**Total running time of the script:** ( 1 minutes 55.738 seconds)


.. _sphx_glr_download_examples_60_search_example_sequential.py:
Expand Down
Loading

0 comments on commit c3d89e2

Please sign in to comment.