autogluon/tabular_prediction_AutoGluon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1117aab6-4c77-4732-9434-3c56a367fc6e",
   "metadata": {},
   "source": [
    "# Tabular prediction using AutoGluon and Preprocessed Parquet files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a934edbd-1d4d-4ae9-8307-89b819426046",
   "metadata": {},
   "source": [
    "AutoGluon can be used to preprocess data. However, as we have 48 GB of different\n",
    "files and a future Pandas dataframe to preprocess, we don’t have enough memory for this expensive\n",
    "operation. Therefore, we employ the same dataset of our previous project “IoT Traffic Malware\n",
    "Detection Using Machine Learning Models”, which is already balanced and preprocessed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7b3ef9d-a282-4de7-a73f-df683347d317",
   "metadata": {},
   "source": [
    "Load training data from a parquet file into an AutoGluon Dataset object. This object is essentially equivalent to a Pandas DataFrame and the same methods can be applied to both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0a3730f1-9cc3-4e0c-ba9e-91c519e0f160",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/fuomag9/.conda/envs/cybersec/lib/python3.8/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id.orig_h</th>\n",
       "      <th>id.orig_p</th>\n",
       "      <th>id.resp_h</th>\n",
       "      <th>id.resp_p</th>\n",
       "      <th>proto</th>\n",
       "      <th>resp_bytes</th>\n",
       "      <th>conn_state</th>\n",
       "      <th>history</th>\n",
       "      <th>orig_pkts</th>\n",
       "      <th>orig_ip_bytes</th>\n",
       "      <th>resp_pkts</th>\n",
       "      <th>resp_ip_bytes</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ts</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-01-10 19:27:37.433427968</th>\n",
       "      <td>10</td>\n",
       "      <td>52520</td>\n",
       "      <td>4937</td>\n",
       "      <td>45</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-09-06 13:49:14.057720832</th>\n",
       "      <td>15</td>\n",
       "      <td>17832</td>\n",
       "      <td>29019</td>\n",
       "      <td>37215</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>80</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-10 18:49:34.581686016</th>\n",
       "      <td>10</td>\n",
       "      <td>13486</td>\n",
       "      <td>24875</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-09 21:24:05.414142976</th>\n",
       "      <td>6</td>\n",
       "      <td>33286</td>\n",
       "      <td>23351</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>1419</td>\n",
       "      <td>9</td>\n",
       "      <td>28</td>\n",
       "      <td>14</td>\n",
       "      <td>2526</td>\n",
       "      <td>15</td>\n",
       "      <td>2207</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-09-06 12:46:34.440527872</th>\n",
       "      <td>15</td>\n",
       "      <td>42964</td>\n",
       "      <td>12898</td>\n",
       "      <td>666</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>120</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               id.orig_h  id.orig_p  id.resp_h  id.resp_p  \\\n",
       "ts                                                                          \n",
       "2019-01-10 19:27:37.433427968         10      52520       4937         45   \n",
       "2018-09-06 13:49:14.057720832         15      17832      29019      37215   \n",
       "2019-01-10 18:49:34.581686016         10      13486      24875         23   \n",
       "2019-01-09 21:24:05.414142976          6      33286      23351         22   \n",
       "2018-09-06 12:46:34.440527872         15      42964      12898        666   \n",
       "\n",
       "                               proto  resp_bytes  conn_state  history  \\\n",
       "ts                                                                      \n",
       "2019-01-10 19:27:37.433427968      1           0           5       10   \n",
       "2018-09-06 13:49:14.057720832      1           0           5       10   \n",
       "2019-01-10 18:49:34.581686016      1           0           5       10   \n",
       "2019-01-09 21:24:05.414142976      1        1419           9       28   \n",
       "2018-09-06 12:46:34.440527872      1           0           5       10   \n",
       "\n",
       "                               orig_pkts  orig_ip_bytes  resp_pkts  \\\n",
       "ts                                                                   \n",
       "2019-01-10 19:27:37.433427968          1             60          0   \n",
       "2018-09-06 13:49:14.057720832          2             80          0   \n",
       "2019-01-10 18:49:34.581686016          1             40          0   \n",
       "2019-01-09 21:24:05.414142976         14           2526         15   \n",
       "2018-09-06 12:46:34.440527872          2            120          0   \n",
       "\n",
       "                               resp_ip_bytes  label  \n",
       "ts                                                   \n",
       "2019-01-10 19:27:37.433427968              0      1  \n",
       "2018-09-06 13:49:14.057720832              0      7  \n",
       "2019-01-10 18:49:34.581686016              0      0  \n",
       "2019-01-09 21:24:05.414142976           2207      9  \n",
       "2018-09-06 12:46:34.440527872              0      2  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from autogluon.tabular import TabularDataset, TabularPredictor\n",
    "from tqdm import tqdm\n",
    "import numpy as np\n",
    "\n",
    "# train_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv')\n",
    "train_data = TabularDataset('/home/fuomag9/Dataset/train.parquet')\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3994ed21-29ee-4165-a5e4-96f21b8da4a6",
   "metadata": {},
   "source": [
    "In this case we have a preprocessed training parquet file, with the following features:\n",
    "- no tunnel-parents, `label` column is encoded\n",
    "- it is balanced (it contains 9000 samples for each label)\n",
    "- the following columns are dropped: `ts`, `local_orig`, `local_resp`, `service`, `uid`, `duration`, `orig_bytes`, `missed_bytes`.\n",
    "\n",
    "Our purpose is to predict the value of the malware label, based on the other columns.\n",
    "Each row represents which type of traffic was captured in a specific instant and other features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c5cbe9c8-1d64-4694-b430-9d7610cca886",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary of label variable: \n",
      " count    50478.000000\n",
      "mean         4.569456\n",
      "std          3.333549\n",
      "min          0.000000\n",
      "25%          1.000000\n",
      "50%          5.000000\n",
      "75%          8.000000\n",
      "max          9.000000\n",
      "Name: label, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "label = 'label'\n",
    "print(\"Summary of label variable: \\n\", train_data[label].describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7e52c7a-7b65-4749-b93f-236d0100d337",
   "metadata": {},
   "source": [
    "Now use AutoGluon to train multiple models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4c8ed251-4194-4be1-8908-a067aaaec4f7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning: path already exists! This predictor may overwrite an existing predictor! path=\"/home/fuomag9/Modelli/AAA/\"\n",
      "Presets specified: ['best_quality']\n",
      "Stack configuration (auto_stack=True): num_stack_levels=1, num_bag_folds=8, num_bag_sets=1\n",
      "Beginning AutoGluon training ...\n",
      "AutoGluon will save models to \"/home/fuomag9/Modelli/AAA/\"\n",
      "AutoGluon Version:  0.6.1\n",
      "Python Version:     3.8.15\n",
      "Operating System:   Linux\n",
      "Platform Machine:   x86_64\n",
      "Platform Version:   #1 SMP PREEMPT_DYNAMIC Wed, 21 Dec 2022 22:27:55 +0000\n",
      "Train Data Rows:    50478\n",
      "Train Data Columns: 12\n",
      "Label Column: label\n",
      "Preprocessing data ...\n",
      "AutoGluon infers your prediction problem is: 'multiclass' (because dtype of label-column == int, but few unique label-values observed).\n",
      "\t7 unique label values:  [1, 7, 0, 9, 2, 5, 8]\n",
      "\tIf 'multiclass' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "Train Data Class Count: 7\n",
      "Using Feature Generators to preprocess the data ...\n",
      "Fitting AutoMLPipelineFeatureGenerator...\n",
      "\tAvailable Memory:                    33077.71 MB\n",
      "\tTrain Data (Original)  Memory Usage: 3.28 MB (0.0% of available memory)\n",
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n",
      "\tStage 1 Generators:\n",
      "\t\tFitting AsTypeFeatureGenerator...\n",
      "\t\t\tNote: Converting 1 features to boolean dtype as they only contain 2 unique values.\n",
      "\tStage 2 Generators:\n",
      "\t\tFitting FillNaFeatureGenerator...\n",
      "\tStage 3 Generators:\n",
      "\t\tFitting IdentityFeatureGenerator...\n",
      "\tStage 4 Generators:\n",
      "\t\tFitting DropUniqueFeatureGenerator...\n",
      "\tTypes of features in original data (raw dtype, special dtypes):\n",
      "\t\t('int', []) : 12 | ['id.orig_h', 'id.orig_p', 'id.resp_h', 'id.resp_p', 'proto', ...]\n",
      "\tTypes of features in processed data (raw dtype, special dtypes):\n",
      "\t\t('int', [])       : 11 | ['id.orig_h', 'id.orig_p', 'id.resp_h', 'id.resp_p', 'resp_bytes', ...]\n",
      "\t\t('int', ['bool']) :  1 | ['proto']\n",
      "\t0.1s = Fit runtime\n",
      "\t12 features in original data used to generate 12 features in processed data.\n",
      "\tTrain Data (Processed) Memory Usage: 3.28 MB (0.0% of available memory)\n",
      "Data preprocessing and feature engineering runtime = 0.09s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n",
      "\tTo change this, specify the eval_metric parameter of Predictor()\n",
      "AutoGluon will fit 2 stack levels (L1 to L2) ...\n",
      "Fitting 13 L1 models ...\n",
      "Fitting model: KNeighborsUnif_BAG_L1 ...\n",
      "\t0.9135\t = Validation score   (accuracy)\n",
      "\t0.05s\t = Training   runtime\n",
      "\t0.55s\t = Validation runtime\n",
      "Fitting model: KNeighborsDist_BAG_L1 ...\n",
      "\t0.9216\t = Validation score   (accuracy)\n",
      "\t0.05s\t = Training   runtime\n",
      "\t0.54s\t = Validation runtime\n",
      "Fitting model: NeuralNetFastAI_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9956\t = Validation score   (accuracy)\n",
      "\t69.89s\t = Training   runtime\n",
      "\t0.46s\t = Validation runtime\n",
      "Fitting model: LightGBMXT_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.999\t = Validation score   (accuracy)\n",
      "\t13.37s\t = Training   runtime\n",
      "\t4.01s\t = Validation runtime\n",
      "Fitting model: LightGBM_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9995\t = Validation score   (accuracy)\n",
      "\t6.16s\t = Training   runtime\n",
      "\t0.39s\t = Validation runtime\n",
      "Fitting model: RandomForestGini_BAG_L1 ...\n",
      "\t0.9996\t = Validation score   (accuracy)\n",
      "\t1.49s\t = Training   runtime\n",
      "\t0.9s\t = Validation runtime\n",
      "Fitting model: RandomForestEntr_BAG_L1 ...\n",
      "\t0.9997\t = Validation score   (accuracy)\n",
      "\t1.31s\t = Training   runtime\n",
      "\t0.76s\t = Validation runtime\n",
      "Fitting model: CatBoost_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9992\t = Validation score   (accuracy)\n",
      "\t18.45s\t = Training   runtime\n",
      "\t0.08s\t = Validation runtime\n",
      "Fitting model: ExtraTreesGini_BAG_L1 ...\n",
      "\t0.9994\t = Validation score   (accuracy)\n",
      "\t0.97s\t = Training   runtime\n",
      "\t0.79s\t = Validation runtime\n",
      "Fitting model: ExtraTreesEntr_BAG_L1 ...\n",
      "\t0.9994\t = Validation score   (accuracy)\n",
      "\t0.97s\t = Training   runtime\n",
      "\t0.8s\t = Validation runtime\n",
      "Fitting model: XGBoost_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9996\t = Validation score   (accuracy)\n",
      "\t7.07s\t = Training   runtime\n",
      "\t0.26s\t = Validation runtime\n",
      "Fitting model: NeuralNetTorch_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9989\t = Validation score   (accuracy)\n",
      "\t105.4s\t = Training   runtime\n",
      "\t0.16s\t = Validation runtime\n",
      "Fitting model: LightGBMLarge_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9996\t = Validation score   (accuracy)\n",
      "\t6.66s\t = Training   runtime\n",
      "\t0.28s\t = Validation runtime\n",
      "Fitting model: WeightedEnsemble_L2 ...\n",
      "\t0.9998\t = Validation score   (accuracy)\n",
      "\t5.67s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Fitting 11 L2 models ...\n",
      "Fitting model: NeuralNetFastAI_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9998\t = Validation score   (accuracy)\n",
      "\t61.1s\t = Training   runtime\n",
      "\t0.95s\t = Validation runtime\n",
      "Fitting model: LightGBMXT_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9998\t = Validation score   (accuracy)\n",
      "\t12.87s\t = Training   runtime\n",
      "\t0.32s\t = Validation runtime\n",
      "Fitting model: LightGBM_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9998\t = Validation score   (accuracy)\n",
      "\t12.71s\t = Training   runtime\n",
      "\t0.07s\t = Validation runtime\n",
      "Fitting model: RandomForestGini_BAG_L2 ...\n",
      "\t0.9997\t = Validation score   (accuracy)\n",
      "\t3.05s\t = Training   runtime\n",
      "\t1.4s\t = Validation runtime\n",
      "Fitting model: RandomForestEntr_BAG_L2 ...\n",
      "\t0.9997\t = Validation score   (accuracy)\n",
      "\t3.12s\t = Training   runtime\n",
      "\t1.4s\t = Validation runtime\n",
      "Fitting model: CatBoost_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9998\t = Validation score   (accuracy)\n",
      "\t47.59s\t = Training   runtime\n",
      "\t0.09s\t = Validation runtime\n",
      "Fitting model: ExtraTreesGini_BAG_L2 ...\n",
      "\t0.9998\t = Validation score   (accuracy)\n",
      "\t1.22s\t = Training   runtime\n",
      "\t1.44s\t = Validation runtime\n",
      "Fitting model: ExtraTreesEntr_BAG_L2 ...\n",
      "\t0.9998\t = Validation score   (accuracy)\n",
      "\t1.12s\t = Training   runtime\n",
      "\t1.48s\t = Validation runtime\n",
      "Fitting model: XGBoost_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9997\t = Validation score   (accuracy)\n",
      "\t16.88s\t = Training   runtime\n",
      "\t0.16s\t = Validation runtime\n",
      "Fitting model: NeuralNetTorch_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9995\t = Validation score   (accuracy)\n",
      "\t55.84s\t = Training   runtime\n",
      "\t0.87s\t = Validation runtime\n",
      "Fitting model: LightGBMLarge_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.9998\t = Validation score   (accuracy)\n",
      "\t31.7s\t = Training   runtime\n",
      "\t0.17s\t = Validation runtime\n",
      "Fitting model: WeightedEnsemble_L3 ...\n",
      "\t0.9998\t = Validation score   (accuracy)\n",
      "\t4.81s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 541.14s ... Best model: \"WeightedEnsemble_L3\"\n",
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"/home/fuomag9/Modelli/AAA/\")\n"
     ]
    }
   ],
   "source": [
    "save_path = '/home/fuomag9/Modelli/AAA/'  # specifies folder to store trained models\n",
    "predictor = TabularPredictor(label=label, path=save_path).fit(train_data, presets='best_quality')\n",
    "#time_limit = 60  # for quick demonstration only, you should set this to longest time you are willing to wait (in seconds)\n",
    "#metric = 'roc_auc'  # specify your evaluation metric here\n",
    "#predictor = TabularPredictor(label, eval_metric=metric).fit(train_data, time_limit=time_limit, presets='best_quality')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f533e37a-e215-4bea-850c-b3486ca1a295",
   "metadata": {},
   "source": [
    "Next, load separate test data to demonstrate how to make predictions on new examples at inference time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fe9687b4-071e-4c05-b9b6-a9049acd5789",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loaded data from: /home/fuomag9/Dataset/test.parquet | Columns = 13 / 13 | Rows = 12620 -> 12620\n"
     ]
    }
   ],
   "source": [
    "test_data = TabularDataset('/home/fuomag9/Dataset/test.parquet')\n",
    "X_test = test_data.drop(columns=[label])\n",
    "y_test = test_data[label]  # values to predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0aacd6f6-d1e6-4606-ad86-4354f4e09ba2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: accuracy on test data: 0.9998415213946117\n",
      "Evaluations on test data:\n",
      "{\n",
      "    \"accuracy\": 0.9998415213946117,\n",
      "    \"balanced_accuracy\": 0.9998433463419997,\n",
      "    \"mcc\": 0.9998151270793555\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions:  \n",
      " ts\n",
      "2018-12-22 02:45:16.347652864    0\n",
      "2019-01-10 19:08:33.919929088    0\n",
      "2018-12-21 19:02:49.106920960    8\n",
      "2018-09-07 01:18:58.862360064    5\n",
      "2018-12-21 16:41:41.499401216    1\n",
      "                                ..\n",
      "2019-01-09 20:48:52.073498880    8\n",
      "2018-08-01 07:44:37.375854080    1\n",
      "2018-12-22 08:38:17.139638784    1\n",
      "2018-07-26 07:27:35.999921920    8\n",
      "2018-12-22 06:02:45.783879168    2\n",
      "Name: label, Length: 12620, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#predictor = TabularPredictor.load(save_path)  # unnecessary, just demonstrates how to load previously-trained predictor from file\n",
    "\n",
    "y_pred = predictor.predict(X_test)\n",
    "print(\"Predictions:  \\n\", y_pred)\n",
    "perf = predictor.evaluate_predictions(y_true=y_test, y_pred=y_pred, auxiliary_metrics=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "23e5b474-d1bc-48c3-b096-5187ae01fa57",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>score_test</th>\n",
       "      <th>score_val</th>\n",
       "      <th>pred_time_test</th>\n",
       "      <th>pred_time_val</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>pred_time_test_marginal</th>\n",
       "      <th>pred_time_val_marginal</th>\n",
       "      <th>fit_time_marginal</th>\n",
       "      <th>stack_level</th>\n",
       "      <th>can_infer</th>\n",
       "      <th>fit_order</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ExtraTreesEntr_BAG_L2</td>\n",
       "      <td>0.999921</td>\n",
       "      <td>0.999762</td>\n",
       "      <td>4.769219</td>\n",
       "      <td>11.467763</td>\n",
       "      <td>232.962383</td>\n",
       "      <td>0.109097</td>\n",
       "      <td>1.479923</td>\n",
       "      <td>1.124924</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ExtraTreesGini_BAG_L2</td>\n",
       "      <td>0.999921</td>\n",
       "      <td>0.999762</td>\n",
       "      <td>4.774835</td>\n",
       "      <td>11.432402</td>\n",
       "      <td>233.052523</td>\n",
       "      <td>0.114714</td>\n",
       "      <td>1.444561</td>\n",
       "      <td>1.215064</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RandomForestEntr_BAG_L2</td>\n",
       "      <td>0.999921</td>\n",
       "      <td>0.999723</td>\n",
       "      <td>4.776412</td>\n",
       "      <td>11.389512</td>\n",
       "      <td>234.959031</td>\n",
       "      <td>0.116291</td>\n",
       "      <td>1.401671</td>\n",
       "      <td>3.121572</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RandomForestGini_BAG_L2</td>\n",
       "      <td>0.999921</td>\n",
       "      <td>0.999703</td>\n",
       "      <td>4.780225</td>\n",
       "      <td>11.385758</td>\n",
       "      <td>234.891113</td>\n",
       "      <td>0.120103</td>\n",
       "      <td>1.397918</td>\n",
       "      <td>3.053654</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>WeightedEnsemble_L2</td>\n",
       "      <td>0.999842</td>\n",
       "      <td>0.999802</td>\n",
       "      <td>1.052884</td>\n",
       "      <td>1.688675</td>\n",
       "      <td>26.858392</td>\n",
       "      <td>0.004531</td>\n",
       "      <td>0.003320</td>\n",
       "      <td>5.666573</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NeuralNetFastAI_BAG_L2</td>\n",
       "      <td>0.999842</td>\n",
       "      <td>0.999822</td>\n",
       "      <td>5.876441</td>\n",
       "      <td>10.940377</td>\n",
       "      <td>292.935756</td>\n",
       "      <td>1.216320</td>\n",
       "      <td>0.952536</td>\n",
       "      <td>61.098296</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>WeightedEnsemble_L3</td>\n",
       "      <td>0.999842</td>\n",
       "      <td>0.999842</td>\n",
       "      <td>6.103206</td>\n",
       "      <td>11.267824</td>\n",
       "      <td>310.615211</td>\n",
       "      <td>0.002182</td>\n",
       "      <td>0.003327</td>\n",
       "      <td>4.812466</td>\n",
       "      <td>3</td>\n",
       "      <td>True</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>RandomForestGini_BAG_L1</td>\n",
       "      <td>0.999762</td>\n",
       "      <td>0.999624</td>\n",
       "      <td>0.109604</td>\n",
       "      <td>0.897619</td>\n",
       "      <td>1.492563</td>\n",
       "      <td>0.109604</td>\n",
       "      <td>0.897619</td>\n",
       "      <td>1.492563</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>LightGBM_BAG_L2</td>\n",
       "      <td>0.999762</td>\n",
       "      <td>0.999762</td>\n",
       "      <td>4.731534</td>\n",
       "      <td>10.060448</td>\n",
       "      <td>244.550537</td>\n",
       "      <td>0.071412</td>\n",
       "      <td>0.072607</td>\n",
       "      <td>12.713078</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>LightGBMLarge_BAG_L2</td>\n",
       "      <td>0.999762</td>\n",
       "      <td>0.999782</td>\n",
       "      <td>4.772942</td>\n",
       "      <td>10.157421</td>\n",
       "      <td>263.538974</td>\n",
       "      <td>0.112820</td>\n",
       "      <td>0.169580</td>\n",
       "      <td>31.701515</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>LightGBMXT_BAG_L2</td>\n",
       "      <td>0.999762</td>\n",
       "      <td>0.999762</td>\n",
       "      <td>4.884704</td>\n",
       "      <td>10.311961</td>\n",
       "      <td>244.704448</td>\n",
       "      <td>0.224582</td>\n",
       "      <td>0.324121</td>\n",
       "      <td>12.866989</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>XGBoost_BAG_L2</td>\n",
       "      <td>0.999762</td>\n",
       "      <td>0.999663</td>\n",
       "      <td>4.970886</td>\n",
       "      <td>10.147380</td>\n",
       "      <td>248.716908</td>\n",
       "      <td>0.310764</td>\n",
       "      <td>0.159539</td>\n",
       "      <td>16.879449</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>CatBoost_BAG_L2</td>\n",
       "      <td>0.999683</td>\n",
       "      <td>0.999822</td>\n",
       "      <td>4.749632</td>\n",
       "      <td>10.080818</td>\n",
       "      <td>279.429843</td>\n",
       "      <td>0.089510</td>\n",
       "      <td>0.092978</td>\n",
       "      <td>47.592384</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>NeuralNetTorch_BAG_L2</td>\n",
       "      <td>0.999683</td>\n",
       "      <td>0.999525</td>\n",
       "      <td>5.788403</td>\n",
       "      <td>10.862667</td>\n",
       "      <td>287.679106</td>\n",
       "      <td>1.128281</td>\n",
       "      <td>0.874826</td>\n",
       "      <td>55.841646</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>RandomForestEntr_BAG_L1</td>\n",
       "      <td>0.999604</td>\n",
       "      <td>0.999723</td>\n",
       "      <td>0.099203</td>\n",
       "      <td>0.755432</td>\n",
       "      <td>1.305125</td>\n",
       "      <td>0.099203</td>\n",
       "      <td>0.755432</td>\n",
       "      <td>1.305125</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>XGBoost_BAG_L1</td>\n",
       "      <td>0.999604</td>\n",
       "      <td>0.999564</td>\n",
       "      <td>0.439709</td>\n",
       "      <td>0.260700</td>\n",
       "      <td>7.068130</td>\n",
       "      <td>0.439709</td>\n",
       "      <td>0.260700</td>\n",
       "      <td>7.068130</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>LightGBMLarge_BAG_L1</td>\n",
       "      <td>0.999445</td>\n",
       "      <td>0.999604</td>\n",
       "      <td>0.190340</td>\n",
       "      <td>0.284174</td>\n",
       "      <td>6.659400</td>\n",
       "      <td>0.190340</td>\n",
       "      <td>0.284174</td>\n",
       "      <td>6.659400</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>LightGBM_BAG_L1</td>\n",
       "      <td>0.999445</td>\n",
       "      <td>0.999544</td>\n",
       "      <td>0.319102</td>\n",
       "      <td>0.385049</td>\n",
       "      <td>6.159164</td>\n",
       "      <td>0.319102</td>\n",
       "      <td>0.385049</td>\n",
       "      <td>6.159164</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>CatBoost_BAG_L1</td>\n",
       "      <td>0.999366</td>\n",
       "      <td>0.999227</td>\n",
       "      <td>0.076305</td>\n",
       "      <td>0.079099</td>\n",
       "      <td>18.450965</td>\n",
       "      <td>0.076305</td>\n",
       "      <td>0.079099</td>\n",
       "      <td>18.450965</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>ExtraTreesEntr_BAG_L1</td>\n",
       "      <td>0.999366</td>\n",
       "      <td>0.999386</td>\n",
       "      <td>0.118302</td>\n",
       "      <td>0.798300</td>\n",
       "      <td>0.973575</td>\n",
       "      <td>0.118302</td>\n",
       "      <td>0.798300</td>\n",
       "      <td>0.973575</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>ExtraTreesGini_BAG_L1</td>\n",
       "      <td>0.999208</td>\n",
       "      <td>0.999366</td>\n",
       "      <td>0.122299</td>\n",
       "      <td>0.794334</td>\n",
       "      <td>0.969857</td>\n",
       "      <td>0.122299</td>\n",
       "      <td>0.794334</td>\n",
       "      <td>0.969857</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>LightGBMXT_BAG_L1</td>\n",
       "      <td>0.999049</td>\n",
       "      <td>0.999049</td>\n",
       "      <td>2.024941</td>\n",
       "      <td>4.014495</td>\n",
       "      <td>13.374473</td>\n",
       "      <td>2.024941</td>\n",
       "      <td>4.014495</td>\n",
       "      <td>13.374473</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>NeuralNetTorch_BAG_L1</td>\n",
       "      <td>0.998653</td>\n",
       "      <td>0.998910</td>\n",
       "      <td>0.242557</td>\n",
       "      <td>0.160240</td>\n",
       "      <td>105.400009</td>\n",
       "      <td>0.242557</td>\n",
       "      <td>0.160240</td>\n",
       "      <td>105.400009</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>NeuralNetFastAI_BAG_L1</td>\n",
       "      <td>0.995959</td>\n",
       "      <td>0.995602</td>\n",
       "      <td>0.605527</td>\n",
       "      <td>0.462065</td>\n",
       "      <td>69.890762</td>\n",
       "      <td>0.605527</td>\n",
       "      <td>0.462065</td>\n",
       "      <td>69.890762</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>KNeighborsDist_BAG_L1</td>\n",
       "      <td>0.919889</td>\n",
       "      <td>0.921649</td>\n",
       "      <td>0.157709</td>\n",
       "      <td>0.541724</td>\n",
       "      <td>0.045432</td>\n",
       "      <td>0.157709</td>\n",
       "      <td>0.541724</td>\n",
       "      <td>0.045432</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>KNeighborsUnif_BAG_L1</td>\n",
       "      <td>0.912441</td>\n",
       "      <td>0.913546</td>\n",
       "      <td>0.154525</td>\n",
       "      <td>0.554609</td>\n",
       "      <td>0.048004</td>\n",
       "      <td>0.154525</td>\n",
       "      <td>0.554609</td>\n",
       "      <td>0.048004</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      model  score_test  score_val  pred_time_test  \\\n",
       "0     ExtraTreesEntr_BAG_L2    0.999921   0.999762        4.769219   \n",
       "1     ExtraTreesGini_BAG_L2    0.999921   0.999762        4.774835   \n",
       "2   RandomForestEntr_BAG_L2    0.999921   0.999723        4.776412   \n",
       "3   RandomForestGini_BAG_L2    0.999921   0.999703        4.780225   \n",
       "4       WeightedEnsemble_L2    0.999842   0.999802        1.052884   \n",
       "5    NeuralNetFastAI_BAG_L2    0.999842   0.999822        5.876441   \n",
       "6       WeightedEnsemble_L3    0.999842   0.999842        6.103206   \n",
       "7   RandomForestGini_BAG_L1    0.999762   0.999624        0.109604   \n",
       "8           LightGBM_BAG_L2    0.999762   0.999762        4.731534   \n",
       "9      LightGBMLarge_BAG_L2    0.999762   0.999782        4.772942   \n",
       "10        LightGBMXT_BAG_L2    0.999762   0.999762        4.884704   \n",
       "11           XGBoost_BAG_L2    0.999762   0.999663        4.970886   \n",
       "12          CatBoost_BAG_L2    0.999683   0.999822        4.749632   \n",
       "13    NeuralNetTorch_BAG_L2    0.999683   0.999525        5.788403   \n",
       "14  RandomForestEntr_BAG_L1    0.999604   0.999723        0.099203   \n",
       "15           XGBoost_BAG_L1    0.999604   0.999564        0.439709   \n",
       "16     LightGBMLarge_BAG_L1    0.999445   0.999604        0.190340   \n",
       "17          LightGBM_BAG_L1    0.999445   0.999544        0.319102   \n",
       "18          CatBoost_BAG_L1    0.999366   0.999227        0.076305   \n",
       "19    ExtraTreesEntr_BAG_L1    0.999366   0.999386        0.118302   \n",
       "20    ExtraTreesGini_BAG_L1    0.999208   0.999366        0.122299   \n",
       "21        LightGBMXT_BAG_L1    0.999049   0.999049        2.024941   \n",
       "22    NeuralNetTorch_BAG_L1    0.998653   0.998910        0.242557   \n",
       "23   NeuralNetFastAI_BAG_L1    0.995959   0.995602        0.605527   \n",
       "24    KNeighborsDist_BAG_L1    0.919889   0.921649        0.157709   \n",
       "25    KNeighborsUnif_BAG_L1    0.912441   0.913546        0.154525   \n",
       "\n",
       "    pred_time_val    fit_time  pred_time_test_marginal  \\\n",
       "0       11.467763  232.962383                 0.109097   \n",
       "1       11.432402  233.052523                 0.114714   \n",
       "2       11.389512  234.959031                 0.116291   \n",
       "3       11.385758  234.891113                 0.120103   \n",
       "4        1.688675   26.858392                 0.004531   \n",
       "5       10.940377  292.935756                 1.216320   \n",
       "6       11.267824  310.615211                 0.002182   \n",
       "7        0.897619    1.492563                 0.109604   \n",
       "8       10.060448  244.550537                 0.071412   \n",
       "9       10.157421  263.538974                 0.112820   \n",
       "10      10.311961  244.704448                 0.224582   \n",
       "11      10.147380  248.716908                 0.310764   \n",
       "12      10.080818  279.429843                 0.089510   \n",
       "13      10.862667  287.679106                 1.128281   \n",
       "14       0.755432    1.305125                 0.099203   \n",
       "15       0.260700    7.068130                 0.439709   \n",
       "16       0.284174    6.659400                 0.190340   \n",
       "17       0.385049    6.159164                 0.319102   \n",
       "18       0.079099   18.450965                 0.076305   \n",
       "19       0.798300    0.973575                 0.118302   \n",
       "20       0.794334    0.969857                 0.122299   \n",
       "21       4.014495   13.374473                 2.024941   \n",
       "22       0.160240  105.400009                 0.242557   \n",
       "23       0.462065   69.890762                 0.605527   \n",
       "24       0.541724    0.045432                 0.157709   \n",
       "25       0.554609    0.048004                 0.154525   \n",
       "\n",
       "    pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  \\\n",
       "0                 1.479923           1.124924            2       True   \n",
       "1                 1.444561           1.215064            2       True   \n",
       "2                 1.401671           3.121572            2       True   \n",
       "3                 1.397918           3.053654            2       True   \n",
       "4                 0.003320           5.666573            2       True   \n",
       "5                 0.952536          61.098296            2       True   \n",
       "6                 0.003327           4.812466            3       True   \n",
       "7                 0.897619           1.492563            1       True   \n",
       "8                 0.072607          12.713078            2       True   \n",
       "9                 0.169580          31.701515            2       True   \n",
       "10                0.324121          12.866989            2       True   \n",
       "11                0.159539          16.879449            2       True   \n",
       "12                0.092978          47.592384            2       True   \n",
       "13                0.874826          55.841646            2       True   \n",
       "14                0.755432           1.305125            1       True   \n",
       "15                0.260700           7.068130            1       True   \n",
       "16                0.284174           6.659400            1       True   \n",
       "17                0.385049           6.159164            1       True   \n",
       "18                0.079099          18.450965            1       True   \n",
       "19                0.798300           0.973575            1       True   \n",
       "20                0.794334           0.969857            1       True   \n",
       "21                4.014495          13.374473            1       True   \n",
       "22                0.160240         105.400009            1       True   \n",
       "23                0.462065          69.890762            1       True   \n",
       "24                0.541724           0.045432            1       True   \n",
       "25                0.554609           0.048004            1       True   \n",
       "\n",
       "    fit_order  \n",
       "0          22  \n",
       "1          21  \n",
       "2          19  \n",
       "3          18  \n",
       "4          14  \n",
       "5          15  \n",
       "6          26  \n",
       "7           6  \n",
       "8          17  \n",
       "9          25  \n",
       "10         16  \n",
       "11         23  \n",
       "12         20  \n",
       "13         24  \n",
       "14          7  \n",
       "15         11  \n",
       "16         13  \n",
       "17          5  \n",
       "18          8  \n",
       "19         10  \n",
       "20          9  \n",
       "21          4  \n",
       "22         12  \n",
       "23          3  \n",
       "24          2  \n",
       "25          1  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.leaderboard(test_data, silent=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04ee1cc2-17dd-434a-b8ea-d9dd094637fc",
   "metadata": {},
   "source": [
    "The best model, based on average accuracy on both validation and test set was `WeightedEnsemble_L3`, yielding an accuracy of 99.98% on the two subsets. There are models, like `ExtraTrees` family,\n",
    "which have a higher test accuracy (99.99%), however the validation score is lower (99.97%). Of\n",
    "course, the best model depends on the task and the time given. If we consider a compromise between\n",
    "inference time and accuracy, the best model is `RandomForestGini_Bag_L1` (one order of magnitude\n",
    "faster than `WeightedEnsemble_L3`, with 0.01s inference time). Neural networks perform better with\n",
    "two stack level, but machine learning models are generally faster and more accurate.\n",
    "Notice the best model in accuracy is from Ensemble family, as we know they make better predic-\n",
    "tions and achieve better performance than any single contributing model."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}