notebooks/3-Supervised-2-Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.2 Classification \n",
    "\n",
    "## Objectives:\n",
    "1. Understand the concept of classification and its role in supervised learning.\n",
    "2. Explore the types of classification: binary, multi-class, and multi-label.\n",
    "3. Learn about common classification algorithms: Logistic Regression, Decision Trees, and K-Nearest Neighbors (KNN).\n",
    "4. Explore evaluation metrics for classification models.\n",
    "5. Hands-on: Implement a binary classification model using Scikit-learn.\n",
    "\n",
    "By the end of this session, you will be able to classify data into discrete categories, handle different classification scenarios, and evaluate model performance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🤔 What is Classification?\n",
    "\n",
    "Classification is a supervised learning technique where the goal is to predict a discrete category (class) for an input.\n",
    "\n",
    "---\n",
    "\n",
    "### Examples of Classification:\n",
    "1. **Spam Detection**:\n",
    "   - Input: Email text\n",
    "   - Output: Spam or Not Spam (binary classification).\n",
    "\n",
    "2. **Disease Diagnosis**:\n",
    "   - Input: Patient symptoms.\n",
    "   - Output: Disease type (multi-class classification).\n",
    "\n",
    "3. **Image Recognition**:\n",
    "   - Input: Image pixels.\n",
    "   - Output: Multiple objects in the image (multi-label classification).\n",
    "\n",
    "---\n",
    "\n",
    "### How Classification Works:\n",
    "1. Train the model on labeled data (inputs with corresponding class labels).\n",
    "2. Learn decision boundaries or probabilities for each class.\n",
    "3. Use the trained model to classify new, unseen inputs.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🔑 Types of Classification\n",
    "\n",
    "### 1️⃣ **Binary Classification**:\n",
    "- **Definition**: Two possible class labels.\n",
    "- **Examples**:\n",
    "  - Spam vs. Not Spam.\n",
    "  - Disease diagnosis: Positive vs. Negative.\n",
    "- **Decision Boundary**: A line, curve, or hyperplane separates the two classes.\n",
    "\n",
    "---\n",
    "\n",
    "### 2️⃣ **Multi-Class Classification**:\n",
    "- **Definition**: More than two possible class labels, but each input is assigned to only one class.\n",
    "- **Examples**:\n",
    "  - Image classification: Dog, Cat, or Bird.\n",
    "  - Flower classification: Iris Setosa, Iris Versicolor, Iris Virginica.\n",
    "- **Special Requirement**: Algorithms like Logistic Regression and Decision Trees need to handle more than two classes.\n",
    "\n",
    "---\n",
    "\n",
    "### 3️⃣ **Multi-Label Classification**:\n",
    "- **Definition**: One input can belong to multiple classes simultaneously.\n",
    "- **Examples**:\n",
    "  - Image with multiple objects: Dog, Ball, Tree.\n",
    "  - Movie genres: A film can be both Comedy and Drama.\n",
    "- **Approach**: Typically solved by transforming the problem into multiple binary classification tasks.\n",
    "\n",
    "---\n",
    "\n",
    "### Key Differences:\n",
    "| Classification Type | Input Example            | Output Example            |\n",
    "|---------------------|--------------------------|---------------------------|\n",
    "| **Binary**          | Patient symptoms         | Disease: Yes or No        |\n",
    "| **Multi-Class**     | Image of a fruit         | Apple, Banana, or Orange  |\n",
    "| **Multi-Label**     | Movie description        | Genres: Comedy, Action    |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🔑 Common Algorithms:\n",
    "1. **Logistic Regression**:\n",
    "   - Despite its name, Logistic Regression is used for classification tasks.\n",
    "   - Uses the sigmoid function to map predictions to probabilities:\n",
    "     \\[\n",
    "     \\sigma(x) = \\frac{1}{1 + e^{-x}}\n",
    "     \\]\n",
    "   - Example: Classify customers as \"churn\" or \"not churn\".\n",
    "\n",
    "2. **Decision Trees**:\n",
    "   - Splits data into branches based on feature values, leading to leaf nodes that represent classes.\n",
    "   - Easy to visualize and interpret.\n",
    "   - Example: Classify animals based on characteristics (e.g., feathers, number of legs).\n",
    "\n",
    "3. **K-Nearest Neighbors (KNN)**:\n",
    "   - A non-parametric method that assigns a class based on the majority class of the \\( k \\) nearest neighbors.\n",
    "   - Works well for small datasets but can be computationally expensive for large datasets.\n",
    "\n",
    "---\n",
    "\n",
    "### Other Algorithms (Briefly Mentioned):\n",
    "- **Support Vector Machines (SVM)**: Separates classes with a hyperplane that maximizes the margin.\n",
    "- **Random Forest**: An ensemble method that combines multiple decision trees.\n",
    "- **Naive Bayes**: A probabilistic classifier based on Bayes' theorem.\n",
    "\n",
    "**Scikit-learn** provides a wide range of classification algorithms and tools to build and evaluate models. See the [documentation](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning) for more details.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation Metrics for Classification\n",
    "\n",
    "We evaluate classification models using various metrics to understand their performance. The common evaluation metrics are \"Accuracy,\" \"Precision,\" \"Recall,\" \"F1 Score,\" which are calculated based on the confusion matrix.\n",
    "\n",
    "### 1️⃣ **Confusion Matrix**:\n",
    "- A table that summarizes the performance of a classification model.\n",
    "- Contains four metrics: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN).\n",
    "- Used to calculate other evaluation metrics.\n",
    "\n",
    "| Actual/Predicted | Positive | Negative |\n",
    "|------------------|----------|----------|\n",
    "| Positive         | TP       | FP       |\n",
    "| Negative         | FN       | TN       |\n",
    "\n",
    "True Positive (TP): Correctly predicted positive instances.\n",
    "True Negative (TN): Correctly predicted negative instances.\n",
    "False Positive (FP): Incorrectly predicted as positive.\n",
    "False Negative (FN): Incorrectly predicted as negative.\n",
    "\n",
    "### 2️⃣ **Accuracy**:\n",
    "- **Definition**: The proportion of correctly classified instances.\n",
    "- **Formula**: Accuracy = correct predictions / total predictions = (TP + TN) / (TP + TN + FP + FN).\n",
    "- Can be misleading for imbalanced datasets. :warning:\n",
    "\n",
    "### 3️⃣ **Precision**:\n",
    "- **Definition**: The proportion of correctly predicted positive instances among all predicted positives.\n",
    "- **Formula**: Precision = TP / (TP + FP).\n",
    "- High precision indicates low false positive rate \n",
    "  \n",
    "### 4️⃣ **Recall (Sensitivity)**:\n",
    "- **Definition**: The proportion of correctly predicted positive instances among all actual positives.\n",
    "- **Formula**: Recall = TP / (TP + FN).\n",
    "- High recall indicates low false negative rate.\n",
    "\n",
    "### 5️⃣ **F1 Score**:\n",
    "- **Definition**: The harmonic mean of precision and recall.\n",
    "- **Formula**: F1 Score = 2 * (Precision * Recall) / (Precision + Recall).\n",
    "- Useful when you want to balance precision and recall.\n",
    "- F1 Score ranges from 0 to 1, where 1 is the best score.\n",
    "- F1 Score is a better metric for imbalanced datasets.\n",
    "\n",
    "For multi-class classification, these metrics can be calculated for each class separately or averaged across all classes. Also, you can use the \"macro\" or \"weighted\" average to handle class imbalance. \n",
    "\n",
    "For more details on evaluation metrics, refer to the Scikit-learn documentation on [classification metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📏 Evaluation Metrics for Classification\n",
    "\n",
    "We evaluate classification models using various metrics to understand their performance. Common evaluation metrics include accuracy, precision, recall, F1-score, and Matthew's correlation coefficient (MCC) based on the confusion matrix.\n",
    "\n",
    "### Confusion Matrix:\n",
    "- A table summarizing true positives, false positives, true negatives, and false negatives.\n",
    "\n",
    "|                | Predicted Positive | Predicted Negative |\n",
    "|----------------|---------------------|---------------------|\n",
    "| **Actual Positive** | True Positive (TP)     | False Negative (FN)     |\n",
    "| **Actual Negative** | False Positive (FP)    | True Negative (TN)      |\n",
    "\n",
    "**True Positive (TP)**: Correctly predicted positive samples.\n",
    "\n",
    "**False Positive (FP)**: Incorrectly predicted positive samples.\n",
    "\n",
    "**True Negative (TN)**: Correctly predicted negative samples.\n",
    "\n",
    "**False Negative (FN)**: Incorrectly predicted negative samples.\n",
    "\n",
    "### 1️⃣ Accuracy:\n",
    "- Percentage of correctly predicted samples.\n",
    "- Formula: Accuracy = \\(\\frac{TP + TN}{TP + TN + FP + FN}\\)\n",
    "- Limitation: Can be misleading for imbalanced datasets.\n",
    "\n",
    "### 2️⃣ Precision:\n",
    "- Of all the predicted positives, how many are truly positive?\n",
    "- Formula:\n",
    "    \\[\n",
    "    \\text{Precision} = \\frac{\\text{True Positives}}{\\text{True Positives} + \\text{False Positives}}\n",
    "    \\]\n",
    "\n",
    "### 3️⃣ Recall (Sensitivity):\n",
    "- Of all the actual positives, how many were correctly predicted?\n",
    "- Formula:\n",
    "    \\[\n",
    "    \\text{Recall} = \\frac{\\text{True Positives}}{\\text{True Positives} + \\text{False Negatives}}\n",
    "    \\]\n",
    "\n",
    "### 4️⃣ F1-Score:\n",
    "- Harmonic mean of precision and recall. Useful when you want to balance both.\n",
    "- Formula:\n",
    "    \\[\n",
    "    \\text{F1-Score} = 2 \\times \\frac{\\text{Precision} \\times \\text{Recall}}{\\text{Precision} + \\text{Recall}}\n",
    "    \\]\n",
    "\n",
    "### 5️⃣ Matthew's Correlation Coefficient (MCC):\n",
    "- Takes into account true and false positives and negatives. Useful for imbalanced datasets.\n",
    "- Formula:\n",
    "    \\[\n",
    "    \\text{MCC} = \\frac{(TP \\times TN) - (FP \\times FN)}{\\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}\n",
    "    \\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🛠️ Hands-on: Binary Classification with Scikit-learn\n",
    "\n",
    "Let's implement a binary classification model using Scikit-learn. We will use the Breast Cancer Wisconsin dataset to classify tumors as benign or malignant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import required libraries\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import classification_report, confusion_matrix, ConfusionMatrixDisplay\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "\n",
    "# print the shape of X and y\n",
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# see dataset as a panda dataframe\n",
    "import pandas as pd\n",
    "df = pd.DataFrame(data=X)\n",
    "df['target'] = y\n",
    "print(df.head())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df['target'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize the data\n",
    "plt.scatter(X[:, 0], X[:, 1], c=y, cmap='viridis', label='Classes')\n",
    "plt.title('Breast Cancer Dataset')\n",
    "plt.xlabel('Feature 1')\n",
    "plt.ylabel('Feature 2')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2. Split Data into Training and Test Sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3. Train a Logistic Regression Model\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 4. Make Predictions\n",
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 5. Evaluate the Model\n",
    "print(\"\\nClassification Report:\\n\", classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Confusion Matrix\n",
    "conf_matrix = confusion_matrix(y_test, y_pred)\n",
    "disp = ConfusionMatrixDisplay(confusion_matrix=conf_matrix, display_labels=model.classes_)\n",
    "disp.plot(cmap='Blues')\n",
    "plt.title('Confusion Matrix')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the model using the metrics\n",
    "accuracy = np.mean(y_test == y_pred)\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "\n",
    "# Precision, Recall, F1-Score\n",
    "precision = np.sum((y_test == 1) & (y_pred == 1)) / np.sum(y_pred == 1)\n",
    "recall = np.sum((y_test == 1) & (y_pred == 1)) / np.sum(y_test == 1)\n",
    "f1_score = 2 * precision * recall / (precision + recall)\n",
    "\n",
    "print(f\"Precision: {precision}\")\n",
    "print(f\"Recall: {recall}\")\n",
    "print(f\"F1 Score: {f1_score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📝 Key Takeaways:\n",
    "1. Classification is used to predict discrete categories (e.g., spam vs. not spam).\n",
    "2. Types include binary, multi-class, and multi-label classification.\n",
    "3. Common algorithms include Logistic Regression, Decision Trees, K-Nearest Neighbors (KNN) and many more.\n",
    "4. Evaluation metrics like accuracy, precision, recall, and F1-score help assess model performance.\n",
    "\n",
    "Next, we will explore **Unsupervised Learning** techniques!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In-class Activity:\n",
    "- Implement another classification algorithm using a different dataset.\n",
    "- Use scikit-learn to choose dataset and new classification algorithm.\n",
    "- Datasets: [Toy datasets](https://scikit-learn.org/1.5/datasets/toy_dataset.html)\n",
    "- Algorithms: [Scikit-learn algorithms](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}