ML_cup_KRR.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8d1ed405",
   "metadata": {},
   "source": [
    "# ML CUP 2022\n",
    "\n",
    "## Kernel Ridge Regression\n",
    "\n",
    "This notebook creates two Kernel Ridge Regression (KRR) models to generalize the problem of the ML cup 2022. It searches the best combination of hyperparameters performing a grid searches over a given range of values. Two different models are given as output in this phase, one for each target, and for both models there is a tuning phase based on the same hyperparameters.\n",
    "\n",
    "Hyperparameters considered for the grid search:\n",
    "\n",
    "1. kernel\n",
    "1. alpha\n",
    "2. gamma (only for rbf and poly kernels)\n",
    "3. degree (only for poly kernel)\n",
    "\n",
    "Model selection performed using a cross validation.\\\n",
    "The model assessment phase is not included in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ac196dd",
   "metadata": {},
   "source": [
    "### Importing libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "26bf163a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from numpy import linalg as LA\n",
    "\n",
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.kernel_ridge import KernelRidge\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import joblib\n",
    "\n",
    "import math\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "088322b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# choosing a seed for reproducibility\n",
    "seed = 1\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f111416",
   "metadata": {},
   "source": [
    "### Definition of the Mean Euclidean Distance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ae1b1894",
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_mean_euclidean_distance(y_true, y_pred):\n",
    "    points = len(y_true)\n",
    "    tot_sum = 0\n",
    "    for i in range (points):\n",
    "        tot_sum += LA.norm(y_true[i] - y_pred[i])\n",
    "    \n",
    "    return tot_sum / points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8c471824",
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_euclidean_distance = make_scorer(my_mean_euclidean_distance, greater_is_better=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "878ff479",
   "metadata": {},
   "source": [
    "### Loading data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "80e5bb96",
   "metadata": {},
   "outputs": [],
   "source": [
    "colnames = ['id', 'a1', 'a2', 'a3', 'a4', 'a5', 'a6', 'a7', 'a8', 'a9', 'target1', 'target2']\n",
    "mlcup_tr = pd.read_csv(\"./dataset/ml_cup22/ML-CUP22-TR.csv\", sep = \",\", names=colnames)\n",
    "mlcup_tr = mlcup_tr.iloc[1:, :]\n",
    "mlcup_tr = mlcup_tr.drop('id', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5f5f44a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_mlcup_tr = mlcup_tr.iloc[:, 0:9].values\n",
    "y_mlcup_tr = mlcup_tr.iloc[:, 9:11].values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "911aecdd",
   "metadata": {},
   "source": [
    "We used the function below to normalize our training set (both input and target) according to a min-max normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e7bade63",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_cols = len(x_mlcup_tr[0])\n",
    "\n",
    "max_col_value_x = [None]*x_cols\n",
    "max_vl = None\n",
    "\n",
    "min_col_value_x = [None]*x_cols\n",
    "min_vl = None\n",
    "\n",
    "for i in range(x_cols):\n",
    "    col = x_mlcup_tr[:, i]\n",
    "    max_vl = np.amax(col)\n",
    "    min_vl = np.amin(col)\n",
    "    \n",
    "    x_mlcup_tr[:, i] = (x_mlcup_tr[:, i] - min_vl) / (max_vl - min_vl)\n",
    "    \n",
    "    max_col_value_x[i] = max_vl\n",
    "    min_col_value_x[i] = min_vl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3b83cbb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "y1_mlcup_tr = y_mlcup_tr[:, 0]\n",
    "y2_mlcup_tr = y_mlcup_tr[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "479b343a",
   "metadata": {},
   "source": [
    "### Grid search for target 1\n",
    "#### rbf kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "eead24e1",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best parameters are {'alpha': 0.03968747494481696, 'gamma': 1.259921049894872} with a score of -0.80567\n"
     ]
    }
   ],
   "source": [
    "alpha_range = np.logspace(-9, 0, 30, base = 2)\n",
    "gamma_range = np.logspace(-9, 3, 10, base = 2)\n",
    "\n",
    "param_grid = [\n",
    "    {'alpha': alpha_range, 'gamma': gamma_range},\n",
    "    {'alpha': alpha_range}\n",
    "]\n",
    "\n",
    "kr = GridSearchCV(\n",
    "    KernelRidge(kernel=\"rbf\"),\n",
    "    param_grid = param_grid,\n",
    "    cv = 4,\n",
    "    scoring = mean_euclidean_distance,\n",
    "    n_jobs = -1\n",
    ")\n",
    "\n",
    "kr.fit(x_mlcup_tr, y1_mlcup_tr)\n",
    "\n",
    "print(\n",
    "    \"The best parameters are %s with a score of %0.5f\"\n",
    "    % (kr.best_params_, kr.best_score_)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "69a32af2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "rbf_krr_1 = kr.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a749df6",
   "metadata": {},
   "source": [
    "#### linear kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "852b1ca0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best parameters are {'alpha': 0.8064489817576826} with a score of -1.55315\n"
     ]
    }
   ],
   "source": [
    "alpha_range = np.logspace(-9, 0, 30, base = 2)\n",
    "\n",
    "param_grid = dict(\n",
    "    alpha = alpha_range\n",
    ")\n",
    "\n",
    "kr = GridSearchCV(\n",
    "    KernelRidge(kernel=\"linear\"),\n",
    "    param_grid = param_grid,\n",
    "    cv = 4,\n",
    "    scoring = mean_euclidean_distance,\n",
    "    n_jobs = -1\n",
    ")\n",
    "\n",
    "kr.fit(x_mlcup_tr, y1_mlcup_tr)\n",
    "\n",
    "print(\n",
    "    \"The best parameters are %s with a score of %0.5f\"\n",
    "    % (kr.best_params_, kr.best_score_)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "971351e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_krr_1 = kr.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dacd8e4",
   "metadata": {},
   "source": [
    "#### polynomial kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9561e70b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best parameters are {'alpha': 0.03125, 'degree': 7, 'gamma': 0.19842513149602486} with a score of -0.84055\n"
     ]
    }
   ],
   "source": [
    "degree_range = np.arange(2, 8, 1)\n",
    "alpha_range = np.logspace(-9, 0, 10, base = 2)\n",
    "gamma_range = np.logspace(-9, 3, 10, base = 2)\n",
    "\n",
    "param_grid = dict(\n",
    "    degree = degree_range,\n",
    "    alpha = alpha_range,\n",
    "    gamma = gamma_range\n",
    ")\n",
    "\n",
    "param_grid = [\n",
    "    {'alpha': alpha_range, 'gamma': gamma_range, 'degree': degree_range},\n",
    "    {'alpha': alpha_range, 'degree': degree_range}\n",
    "]\n",
    "\n",
    "kr = GridSearchCV(\n",
    "    KernelRidge(kernel = 'poly'),\n",
    "    param_grid = param_grid,\n",
    "    cv = 4,\n",
    "    scoring = mean_euclidean_distance,\n",
    "    n_jobs = -1\n",
    ")\n",
    "\n",
    "kr.fit(x_mlcup_tr, y1_mlcup_tr)\n",
    "\n",
    "print(\n",
    "    \"The best parameters are %s with a score of %0.5f\"\n",
    "    % (kr.best_params_, kr.best_score_)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "53ff8abc",
   "metadata": {},
   "outputs": [],
   "source": [
    "poly_krr_1 = kr.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abaa9154",
   "metadata": {},
   "source": [
    "### Grid search for target 2\n",
    "#### rbf kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ce875425",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best parameters are {'alpha': 0.004617665262461984, 'gamma': 0.5} with a score of -1.09913\n"
     ]
    }
   ],
   "source": [
    "alpha_range = np.logspace(-9, 0, 30, base = 2)\n",
    "gamma_range = np.logspace(-9, 3, 10, base = 2)\n",
    "\n",
    "param_grid = [\n",
    "    {'alpha': alpha_range, 'gamma': gamma_range},\n",
    "    {'alpha': alpha_range}\n",
    "]\n",
    "\n",
    "kr = GridSearchCV(\n",
    "    KernelRidge(kernel=\"rbf\"),\n",
    "    param_grid = param_grid,\n",
    "    cv = 4,\n",
    "    scoring = mean_euclidean_distance,\n",
    "    n_jobs = -1\n",
    ")\n",
    "\n",
    "kr.fit(x_mlcup_tr, y2_mlcup_tr)\n",
    "\n",
    "print(\n",
    "    \"The best parameters are %s with a score of %0.5f\"\n",
    "    % (kr.best_params_, kr.best_score_)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c1a54e06",
   "metadata": {},
   "outputs": [],
   "source": [
    "rbf_krr_2 = kr.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1ab84a4",
   "metadata": {},
   "source": [
    "#### linear kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "b1f41183",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best parameters are {'alpha': 1.0} with a score of -1.69602\n"
     ]
    }
   ],
   "source": [
    "alpha_range = np.logspace(-9, 0, 30, base = 2)\n",
    "\n",
    "param_grid = dict(\n",
    "    alpha = alpha_range\n",
    ")\n",
    "\n",
    "kr = GridSearchCV(\n",
    "    KernelRidge(kernel=\"linear\"),\n",
    "    param_grid = param_grid,\n",
    "    cv = 4,\n",
    "    scoring = mean_euclidean_distance,\n",
    "    n_jobs = -1\n",
    ")\n",
    "\n",
    "kr.fit(x_mlcup_tr, y2_mlcup_tr)\n",
    "\n",
    "print(\n",
    "    \"The best parameters are %s with a score of %0.5f\"\n",
    "    % (kr.best_params_, kr.best_score_)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "569a36f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_krr_2 = kr.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "256e281e",
   "metadata": {},
   "source": [
    "#### polynomial kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "9ebd1705",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best parameters are {'alpha': 0.0625, 'degree': 7, 'gamma': 0.19842513149602486} with a score of -1.12928\n"
     ]
    }
   ],
   "source": [
    "degree_range = np.arange(2, 8, 1)\n",
    "alpha_range = np.logspace(-9, 0, 10, base = 2)\n",
    "gamma_range = np.logspace(-9, 3, 10, base = 2)\n",
    "\n",
    "param_grid = [\n",
    "    {'alpha': alpha_range, 'gamma': gamma_range, 'degree': degree_range},\n",
    "    {'alpha': alpha_range, 'degree': degree_range}\n",
    "]\n",
    "\n",
    "kr = GridSearchCV(\n",
    "    KernelRidge(kernel = 'poly'),\n",
    "    param_grid = param_grid,\n",
    "    cv = 4,\n",
    "    scoring = mean_euclidean_distance,\n",
    "    n_jobs = -1\n",
    ")\n",
    "\n",
    "kr.fit(x_mlcup_tr, y2_mlcup_tr)\n",
    "\n",
    "print(\n",
    "    \"The best parameters are %s with a score of %0.5f\"\n",
    "    % (kr.best_params_, kr.best_score_)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a1bbbbd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "poly_krr_2 = kr.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7510628",
   "metadata": {},
   "source": [
    "## Model selection\n",
    "### Target 1 and 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e52b1e7b",
   "metadata": {},
   "source": [
    "Since the best results are given by the Kernel ridge regression classifier with the rbf kernel on the first target and the poly kernel on the second target, we choose these two models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "9266794f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./results/ml_cup/KRR/rbf_krr_2.z']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(rbf_krr_1, './results/ml_cup/KRR/rbf_krr_1.z')\n",
    "joblib.dump(poly_krr_2, './results/ml_cup/KRR/rbf_krr_2.z')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08c94312",
   "metadata": {},
   "source": [
    "### MEE on both targets on the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "22b57e27",
   "metadata": {},
   "outputs": [],
   "source": [
    "krr1 = joblib.load('./results/ml_cup/KRR/rbf_krr_1.z')\n",
    "krr2 = joblib.load('./results/ml_cup/KRR/rbf_krr_2.z')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d9496548",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_label_krr_1 = krr1.predict(x_mlcup_tr)\n",
    "pred_label_krr_2 = krr2.predict(x_mlcup_tr)\n",
    "pred_label_krr = np.vstack((pred_label_krr_1, pred_label_krr_2)).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "b7d92803",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MEE on the training set: 1.3637822027846693\n"
     ]
    }
   ],
   "source": [
    "# Mean euclidean distance\n",
    "points = y_mlcup_tr.shape[0]\n",
    "tot_sum = 0\n",
    "for i in range (points):\n",
    "    tot_sum += math.sqrt(math.pow((y_mlcup_tr[i][0] - pred_label_krr[i][0]), 2)\n",
    "                         + math.pow((y_mlcup_tr[i][1] - pred_label_krr[i][1]), 2))\n",
    "    \n",
    "print('MEE on the training set:', tot_sum / points)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}