lab_5_Clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DVL7_bgmIAPR"
   },
   "source": [
    "# Unsupervised Learning: Clustering Lab\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6ZbYjZZZ_yLV"
   },
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, ClassifierMixin\n",
    "from sklearn.linear_model import Perceptron\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sCcEPx5VIORj"
   },
   "source": [
    "## 1. (50%) Implement the k-means clustering algorithm and the HAC (Hierarchical Agglomerative Clustering) algorithm.\n",
    "\n",
    "### 1.1.1 HAC\n",
    "\n",
    "### Code requirements \n",
    "- HAC should support both single link and complete link options.\n",
    "- HAC automatically generates all clusterings from n to 1.  To simplify the amount of output you may want to implement a mechanism to specify for which k values actual output will be generated.\n",
    "\n",
    "\n",
    "---\n",
    "The output should include the following:\n",
    "- The number of clusters (k).\n",
    "- The total SSE of the full clustering. \n",
    "\n",
    "\n",
    "For each cluster report include:\n",
    "\n",
    "\n",
    "- The centroid id.\n",
    "- The number of instances tied to that centroid. The SSE of that cluster. The sum squared error (SSE) of a single cluster is the sum of the squared euclidean distance of each cluster member to the cluster centroid.\n",
    "---\n",
    "You only need to handle continuous features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_a2KSZ_7AN0G"
   },
   "outputs": [],
   "source": [
    "class HAC(BaseEstimator,ClassifierMixin):\n",
    "\n",
    "    def __init__(self, k=1, shuffle=True):\n",
    "        \"\"\" \n",
    "            Initialize class with chosen hyperparameters.\n",
    "        Args:\n",
    "            lr (float): A learning rate / step size.\n",
    "            shuffle: Whether to shuffle the training data each epoch. DO NOT \n",
    "            SHUFFLE for evaluation / debug datasets.\n",
    "        \"\"\"\n",
    "        self.lr = lr\n",
    "        self.shuffle = shuffle\n",
    "\n",
    "    def fit(self, X, y, initial_weights=None):\n",
    "        \"\"\" \n",
    "            Fit the data; run the algorithm and adjust the weights to find a \n",
    "            good solution\n",
    "        Args:\n",
    "            X (array-like): A 2D numpy array with the training data, excluding\n",
    "            targets\n",
    "            y (array-like): A 2D numpy array with the training targets\n",
    "            initial_weights (array-like): allows the user to provide initial \n",
    "            weights\n",
    "        Returns:\n",
    "            self: this allows this to be chained, e.g. model.fit(X,y).predict(X_test)\n",
    "        \"\"\"\n",
    "        self.initial_weights = self.initialize_weights() if not initial_weights else initial_weights\n",
    "\n",
    "        return self\n",
    "\n",
    "    def predict(self, X):\n",
    "        \"\"\" \n",
    "            Predict all classes for a dataset X\n",
    "        Args:\n",
    "            X (array-like): A 2D numpy array with the training data, excluding \n",
    "            targets\n",
    "        Returns:\n",
    "            array, shape (n_samples,)\n",
    "                Predicted target values per element in X.\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "\n",
    "    def score(self, X, y):\n",
    "        \"\"\" \n",
    "            Return accuracy of model on a given dataset. Must implement own \n",
    "            score function.\n",
    "        Args:\n",
    "            X (array-like): A 2D numpy array with data, excluding targets\n",
    "            y (array-like): A 2D numpy array with targets\n",
    "        Returns:\n",
    "            score : float\n",
    "                Mean accuracy of self.predict(X) wrt. y.\n",
    "        \"\"\"\n",
    "\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KibCIXIThpbE"
   },
   "source": [
    "### 1.1.2 Debug \n",
    "\n",
    "Debug your model by running it on the [Debug Dataset](https://raw.githubusercontent.com/cs472ta/CS472/master/datasets/abalone.arff)\n",
    "\n",
    "\n",
    "---\n",
    "The dataset was modified to be a lot smaller. The last datapoint should be on line 359 or the point 0.585,0.46,0.185,0.922,0.3635,0.213,0.285,10. The remaining points should be commented out.\n",
    "\n",
    "\n",
    "\n",
    "- Normalize Data\n",
    "- K = 5\n",
    "- Use the first k instances as the initial centroids\n",
    "- Use 4 decimal places and DO NOT ROUND when reporting SSE and centroid values.\n",
    "\n",
    "\n",
    "---\n",
    "Solutions in files:\n",
    "\n",
    "Debug HAC Single.txt\n",
    "\n",
    "Debug HAC Complete.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Debug Here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kY3VNB1ui03N"
   },
   "source": [
    "### 1.1.3 Evaluation\n",
    "\n",
    "We will evaluate your model based on it's performance on the [Evaluation Dataset](https://raw.githubusercontent.com/cs472ta/CS472/master/datasets/seismic-bumps_train.arff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2yAxA78QjDh2"
   },
   "outputs": [],
   "source": [
    "# Load evaluation data\n",
    "\n",
    "# Train on evaluation data\n",
    "\n",
    "# Print weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.1 K-Means\n",
    "\n",
    "### Code requirements \n",
    "- Ability to choose k and specify k initial centroids\n",
    "- Use Euclidean Distance as metric\n",
    "- Ability to handle distance ties\n",
    "- Include output label as a cluster feature\n",
    "\n",
    "\n",
    "---\n",
    "The output should include the following:\n",
    "- The number of clusters (k).\n",
    "- The total SSE of the full clustering. \n",
    "\n",
    "\n",
    "For each cluster report include:\n",
    "\n",
    "\n",
    "- The centroid id.\n",
    "- The number of instances tied to that centroid. The SSE of that cluster. The sum squared error (SSE) of a single cluster is the sum of the squared euclidean distance of each cluster member to the cluster centroid.\n",
    "---\n",
    "You only need to handle continuous features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_a2KSZ_7AN0G"
   },
   "outputs": [],
   "source": [
    "class KMEANS(BaseEstimator,ClassifierMixin):\n",
    "\n",
    "    def __init__(self, k=1, shuffle=True):\n",
    "        \"\"\" \n",
    "            Initialize class with chosen hyperparameters.\n",
    "        Args:\n",
    "            lr (float): A learning rate / step size.\n",
    "            shuffle: Whether to shuffle the training data each epoch. DO NOT \n",
    "            SHUFFLE for evaluation / debug datasets.\n",
    "        \"\"\"\n",
    "        self.lr = lr\n",
    "        self.shuffle = shuffle\n",
    "\n",
    "    def fit(self, X, y, initial_weights=None):\n",
    "        \"\"\" \n",
    "            Fit the data; run the algorithm and adjust the weights to find a \n",
    "            good solution\n",
    "        Args:\n",
    "            X (array-like): A 2D numpy array with the training data, excluding\n",
    "            targets\n",
    "            y (array-like): A 2D numpy array with the training targets\n",
    "            initial_weights (array-like): allows the user to provide initial \n",
    "            weights\n",
    "        Returns:\n",
    "            self: this allows this to be chained, e.g. model.fit(X,y).predict(X_test)\n",
    "        \"\"\"\n",
    "        self.initial_weights = self.initialize_weights() if not initial_weights else initial_weights\n",
    "\n",
    "        return self\n",
    "\n",
    "    def predict(self, X):\n",
    "        \"\"\" \n",
    "            Predict all classes for a dataset X\n",
    "        Args:\n",
    "            X (array-like): A 2D numpy array with the training data, excluding \n",
    "            targets\n",
    "        Returns:\n",
    "            array, shape (n_samples,)\n",
    "                Predicted target values per element in X.\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "\n",
    "    def score(self, X, y):\n",
    "        \"\"\" \n",
    "            Return accuracy of model on a given dataset. Must implement own \n",
    "            score function.\n",
    "        Args:\n",
    "            X (array-like): A 2D numpy array with data, excluding targets\n",
    "            y (array-like): A 2D numpy array with targets\n",
    "        Returns:\n",
    "            score : float\n",
    "                Mean accuracy of self.predict(X) wrt. y.\n",
    "        \"\"\"\n",
    "\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KibCIXIThpbE"
   },
   "source": [
    "### 1.2.2 Debug \n",
    "\n",
    "Debug your model by running it on the [Debug Dataset](https://raw.githubusercontent.com/cs472ta/CS472/master/datasets/abalone.arff)\n",
    "\n",
    "\n",
    "- Train until convergance\n",
    "- Normalize Data\n",
    "- K = 5\n",
    "- Use the first k instances as the initial centroids\n",
    "- Use 4 decimal places and DO NOT ROUND when reporting SSE and centroid values\n",
    "\n",
    "\n",
    "---\n",
    "Solutions in files:\n",
    "\n",
    "Debug K Means.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KgAyy82gixIF"
   },
   "outputs": [],
   "source": [
    "# Load debug data\n",
    "\n",
    "# Train on debug data\n",
    "\n",
    "# Check weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kY3VNB1ui03N"
   },
   "source": [
    "### 1.2.3 Evaluation\n",
    "\n",
    "We will evaluate your model based on it's performance on the [Evaluation Dataset](https://raw.githubusercontent.com/cs472ta/CS472/master/datasets/seismic-bumps_train.arff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2yAxA78QjDh2"
   },
   "outputs": [],
   "source": [
    "# Load evaluation data\n",
    "\n",
    "# Train on evaluation data\n",
    "\n",
    "# Print weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9vWiTdlbR2Xh"
   },
   "source": [
    "## 2.1 (12.5%) Clustering the Iris Classification problem - HAC\n",
    "\n",
    "Load the Iris Dataset [Iris Dataset](https://raw.githubusercontent.com/rmorain/CS472-1/master/datasets/perceptron/iris.arff)\n",
    "\n",
    "Requirements:\n",
    "- State whether you normalize or not (your choice).  \n",
    "- For both algorithms show your results for clusterings with k = 2-7.  \n",
    "- Graph the total SSE for each k and discuss your results (i.e. what kind of clusters are being made).\n",
    "- Now do it again where you include the output label as one of the input features and discuss your results and any differences. For this final data set, also run k-means 5 times with k=4, each time with different initial random centroids and discuss any variations in the results.\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4SSoasDQSKXb"
   },
   "outputs": [],
   "source": [
    "# Iris Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9vWiTdlbR2Xh"
   },
   "source": [
    "## 2.2 (12.5%) Clustering the Iris Classification problem: K-Means\n",
    "\n",
    "Load the Iris Dataset [Iris Dataset](https://raw.githubusercontent.com/rmorain/CS472-1/master/datasets/perceptron/iris.arff)\n",
    "\n",
    "Requirements:\n",
    "- State whether you normalize or not (your choice).  \n",
    "- For both algorithms show your results for clusterings with k = 2-7.  \n",
    "- Graph the total SSE for each k and discuss your results (i.e. what kind of clusters are being made).\n",
    "- Now do it again where you include the output label as one of the input features and discuss your results and any differences. For this final data set, also run k-means 5 times with k=4, each time with different initial random centroids and discuss any variations in the results.\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4SSoasDQSKXb"
   },
   "outputs": [],
   "source": [
    "# Iris Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oBBmeNQ7jvcQ"
   },
   "source": [
    "## 3.1 (12.5%) Run the SK versions of HAC (both single and complete link) on iris including the output label and compare your results with those above.\n",
    "\n",
    "Requirements\n",
    "- Use the Sillhouette score for this iris problem (k= 2-7) \n",
    "- Use at least one other scoring function form sklearn.metrics and compare the results. State which metric was used\n",
    "- Experiment different hypder-parameters. Discuss Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OFQv70W2VyqJ"
   },
   "outputs": [],
   "source": [
    "# Load sklearn\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BqSFAXwlk3Ms"
   },
   "source": [
    "*Record impressions*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oBBmeNQ7jvcQ"
   },
   "source": [
    "## 3.2 (12.5%) Run the SK versions of k-means on iris including the output label and compare your results with those above. \n",
    "\n",
    "Requirements\n",
    "- Use the Sillhouette score for this iris problem (k= 2-7) \n",
    "- Use at least one other scoring function form sklearn.metrics and compare the results. State which metric was used\n",
    "- Experiment different hypder-parameters. Discuss Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OFQv70W2VyqJ"
   },
   "outputs": [],
   "source": [
    "# Load sklearn \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BqSFAXwlk3Ms"
   },
   "source": [
    "*Record impressions*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cTlK-kijk8Mg"
   },
   "source": [
    "## 4. (Optional 5% extra credit) For your silhouette experiment above, write and use your own code to calculate the silhouette scores, rather than the SK or other version. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BqSFAXwlk3Ms"
   },
   "source": [
    "*Show findings here*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy function Below"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "lab 1 - perceptron",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}