ENH Emphasize discussion on multi-class classification in tree notebook

python_scripts/trees_classification.py

@@ -8,8 +8,8 @@
 # %% [markdown]
 # # Build a classification decision tree
 #
-# We will illustrate how decision tree fit data with a simple classification
-# problem using the penguins dataset.
+# In this notebook we illustrate decision trees in a multiclass classification
+# problem by using the penguins dataset with 2 features and 3 classes.
 
 # %% [markdown]
 # ```{note}
@@ -25,8 +25,8 @@
 target_column = "Species"
 
 # %% [markdown]
-# Besides, we split the data into two subsets to investigate how trees will
-# predict values based on an out-of-samples dataset.
+# First, we split the data into two subsets to investigate how trees predict
+# values based on unseen data.
 
 # %%
 from sklearn.model_selection import train_test_split
@@ -37,16 +37,13 @@
 )
 
 # %% [markdown]
-# In a previous notebook, we learnt that a linear classifier will define a
-# linear separation to split classes using a linear combination of the input
-# features. In our 2-dimensional space, it means that a linear classifier will
-# define some oblique lines that best separate our classes. We define a function
-# below that, given a set of data points and a classifier, will plot the
-# decision boundaries learnt by the classifier.
-#
-# Thus, for a linear classifier, we will obtain the following decision
-# boundaries. These boundaries lines indicate where the model changes its
-# prediction from one class to another.
+# In a previous notebook, we learnt that linear classifiers define a linear
+# separation to split classes using a linear combination of the input features.
+# In our 2-dimensional feature space, it means that a linear classifier finds
+# the oblique lines that best separate the classes. This is still true for
+# multiclass problems, except that more than one line is fitted. We can use
+# `DecisionBoundaryDisplay` to plot the decision boundaries learnt by the
+# classifier.
 
 # %%
 from sklearn.linear_model import LogisticRegression
@@ -80,7 +77,7 @@
 # %% [markdown]
 # We see that the lines are a combination of the input features since they are
 # not perpendicular a specific axis. Indeed, this is due to the model
-# parametrization that we saw in the previous notebook, controlled by the
+# parametrization that we saw in some previous notebooks, i.e. controlled by the
 # model's weights and intercept.
 #
 # Besides, it seems that the linear model would be a good candidate for such
@@ -93,12 +90,12 @@
 
 # %% [markdown]
 # Unlike linear models, decision trees are non-parametric models: they are not
-# controlled by a mathematical decision function and do not have weights or
+# controlled by a mathematical decision function and do not have weights or an
 # intercept to be optimized.
 #
-# Indeed, decision trees will partition the space by considering a single
-# feature at a time. Let's illustrate this behaviour by having a decision tree
-# make a single split to partition the feature space.
+# Indeed, decision trees partition the space by considering a single feature at
+# a time. Let's illustrate this behaviour by having a decision tree make a
+# single split to partition the feature space.
 
 # %%
 from sklearn.tree import DecisionTreeClassifier
@@ -123,7 +120,7 @@
 # %% [markdown]
 # The partitions found by the algorithm separates the data along the axis
 # "Culmen Depth", discarding the feature "Culmen Length". Thus, it highlights
-# that a decision tree does not use a combination of feature when making a
+# that a decision tree does not use a combination of features when making a
 # split. We can look more in depth at the tree structure.
 
 # %%
@@ -150,44 +147,48 @@
 # dataset was subdivided into 2 sets based on the culmen depth (inferior or
 # superior to 16.45 mm).
 #
-# This partition of the dataset minimizes the class diversities in each
+# This partition of the dataset minimizes the class diversity in each
 # sub-partitions. This measure is also known as a **criterion**, and is a
 # settable parameter.
 #
 # If we look more closely at the partition, we see that the sample superior to
-# 16.45 belongs mainly to the Adelie class. Looking at the values, we indeed
-# observe 103 Adelie individuals in this space. We also count 52 Chinstrap
-# samples and 6 Gentoo samples. We can make similar interpretation for the
+# 16.45 belongs mainly to the "Adelie" class. Looking at the values, we indeed
+# observe 103 "Adelie" individuals in this space. We also count 52 "Chinstrap"
+# samples and 6 "Gentoo" samples. We can make similar interpretation for the
 # partition defined by a threshold inferior to 16.45mm. In this case, the most
-# represented class is the Gentoo species.
+# represented class is the "Gentoo" species.
 #
 # Let's see how our tree would work as a predictor. Let's start with a case
 # where the culmen depth is inferior to the threshold.
 
 # %%
-sample_1 = pd.DataFrame({"Culmen Length (mm)": [0], "Culmen Depth (mm)": [15]})
-tree.predict(sample_1)
+test_penguin_1 = pd.DataFrame(
+    {"Culmen Length (mm)": [0], "Culmen Depth (mm)": [15]}
+)
+tree.predict(test_penguin_1)
 
 # %% [markdown]
-# The class predicted is the Gentoo. We can now check what happens if we pass a
+# The class predicted is the "Gentoo". We can now check what happens if we pass a
 # culmen depth superior to the threshold.
 
 # %%
-sample_2 = pd.DataFrame({"Culmen Length (mm)": [0], "Culmen Depth (mm)": [17]})
-tree.predict(sample_2)
+test_penguin_2 = pd.DataFrame(
+    {"Culmen Length (mm)": [0], "Culmen Depth (mm)": [17]}
+)
+tree.predict(test_penguin_2)
 
 # %% [markdown]
-# In this case, the tree predicts the Adelie specie.
+# In this case, the tree predicts the "Adelie" specie.
 #
-# Thus, we can conclude that a decision tree classifier will predict the most
+# Thus, we can conclude that a decision tree classifier predicts the most
 # represented class within a partition.
 #
 # During the training, we have a count of samples in each partition, we can also
 # compute the probability of belonging to a specific class within this
 # partition.
 
 # %%
-y_pred_proba = tree.predict_proba(sample_2)
+y_pred_proba = tree.predict_proba(test_penguin_2)
 y_proba_class_0 = pd.Series(y_pred_proba[0], index=tree.classes_)
 
 # %%
@@ -212,14 +213,14 @@
 
 # %% [markdown]
 # It is also important to note that the culmen length has been disregarded for
-# the moment. It means that whatever the value given, it will not be used during
-# the prediction.
+# the moment. It means that regardless of its value, it is not used during the
+# prediction.
 
 # %%
-sample_3 = pd.DataFrame(
+test_penguin_3 = pd.DataFrame(
     {"Culmen Length (mm)": [10_000], "Culmen Depth (mm)": [17]}
 )
-tree.predict_proba(sample_3)
+tree.predict_proba(test_penguin_3)
 
 # %% [markdown]
 # Going back to our classification problem, the split found with a maximum depth
@@ -232,9 +233,10 @@
 print(f"Accuracy of the DecisionTreeClassifier: {test_score:.2f}")
 
 # %% [markdown]
-# Indeed, it is not a surprise. We saw earlier that a single feature will not be
-# able to separate all three species. However, from the previous analysis we saw
-# that by using both features we should be able to get fairly good results.
+# Indeed, it is not a surprise. We saw earlier that a single feature is not able
+# to separate all three species: it underfits. However, from the previous
+# analysis we saw that by using both features we should be able to get fairly
+# good results.
 #
-# In the next exercise, you will increase the size of the tree depth. You will
-# get intuitions on how the space partitioning is repeated over time.
+# In the next exercise, you will increase the tree depth to get an intuition on
+# how such parameter affects the space partitioning.

python_scripts/trees_sol_01.py

@@ -8,16 +8,13 @@
 # %% [markdown]
 # # 📃 Solution for Exercise M5.01
 #
-# In the previous notebook, we showed how a tree with a depth of 1 level was
-# working. The aim of this exercise is to repeat part of the previous experiment
-# for a depth with 2 levels to show how the process of partitioning is repeated
-# over time.
+# In the previous notebook, we showed how a tree with 1 level depth works. The
+# aim of this exercise is to repeat part of the previous experiment for a tree
+# with 2 levels depth to show how such parameter affects the feature space
+# partitioning.
 #
-# Before to start, we will:
-#
-# * load the dataset;
-# * split the dataset into training and testing dataset;
-# * define the function to show the classification decision function.
+# We first load the penguins dataset and split it into a training and a testing
+# sets:
 
 # %%
 import pandas as pd
@@ -42,10 +39,7 @@
 
 # %% [markdown]
 # Create a decision tree classifier with a maximum depth of 2 levels and fit the
-# training data. Once this classifier trained, plot the data and the decision
-# boundary to see the benefit of increasing the depth. To plot the decision
-# boundary, you should import the class `DecisionBoundaryDisplay` from the
-# module `sklearn.inspection` as shown in the previous course notebook.
+# training data.
 
 # %%
 # solution
@@ -54,7 +48,15 @@
 tree = DecisionTreeClassifier(max_depth=2)
 tree.fit(data_train, target_train)
 
-# %% tags=["solution"]
+# %% [markdown]
+# Now plot the data and the decision boundary of the trained classifier to see
+# the effect of increasing the depth of the tree.
+#
+# Hint: Use the class `DecisionBoundaryDisplay` from the module
+# `sklearn.inspection` as shown in previous course notebooks.
+
+# %%
+# solution
 import matplotlib.pyplot as plt
 import seaborn as sns
 
@@ -114,3 +116,64 @@
 # which is not surprising since this partition was almost pure. If the feature
 # value is above the threshold, we predict the Gentoo penguin, the class that is
 # most probable.
+#
+# ## (Estimated) predicted probabilities in multi-class problems
+#
+# For those interested, one can further try to visualize the output of
+# `predict_proba` for a multiclass problem using `DecisionBoundaryDisplay`,
+# except that for a K-class problem you have K probability outputs for each
+# data point. Visualizing all these on a single plot can quickly become tricky
+# to interpret. It is then common to instead produce K separate plots, one for
+# each class, in a one-vs-rest (or one-vs-all) fashion.
+#
+# For example, in the plot below, the first plot on the left shows in red the
+# certainty on classifying a data point as belonging to the "Adelie" class. In
+# the same plot, the blue color represents the certainty of **not** belonging to
+# the "Adelie" class. The same logic applies to the other plots in the figure.
+
+# %% tags=["solution"]
+import numpy as np
+
+xx = np.linspace(30, 60, 100)
+yy = np.linspace(10, 23, 100)
+xx, yy = np.meshgrid(xx, yy)
+Xfull = pd.DataFrame(
+    {"Culmen Length (mm)": xx.ravel(), "Culmen Depth (mm)": yy.ravel()}
+)
+
+probas = tree.predict_proba(Xfull)
+n_classes = len(np.unique(tree.classes_))
+
+_, axs = plt.subplots(ncols=3, nrows=1, sharey=True, figsize=(12, 5))
+plt.suptitle("Predicted probabilities for decision tree model", y=0.8)
+
+for k in range(n_classes):
+    axs[k].set_title(f"Class {tree.classes_[k]}")
+    imshow_handle = axs[k].imshow(
+        probas[:, k].reshape((100, 100)),
+        extent=(30, 60, 10, 23),
+        vmin=0.0,
+        vmax=1.0,
+        origin="lower",
+        cmap="RdBu_r",
+    )
+    axs[k].set_xlabel("Culmen Length (mm)")
+    if k == 0:
+        axs[k].set_ylabel("Culmen Depth (mm)")
+    idx = target_test == tree.classes_[k]
+    axs[k].scatter(
+        data_test["Culmen Length (mm)"].loc[idx],
+        data_test["Culmen Depth (mm)"].loc[idx],
+        marker="o",
+        c="w",
+        edgecolor="k",
+    )
+
+ax = plt.axes([0.15, 0.04, 0.7, 0.05])
+plt.colorbar(imshow_handle, cax=ax, orientation="horizontal")
+_ = plt.title("Probability")
+
+# %% [markdown] tags=["solution"]
+# In scikit-learn v1.4 `DecisionBoundaryDisplay` will support a `class_of_interest`
+# parameter that will allow in particular for a visualization of `predict_proba` in
+# multi-class settings.