Chapter6Prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 6 Prediction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 General Prediction Framework\n",
    "\n",
    "Suppose that we are interested in predicting an <span style=\"color:red\"> outcome, $y$,</span> using a set of features ${\\bf  x}= (x_1, x_2, \\ldots, x_p)^T$. In other words, we want to learn a function $f\\left({\\bf x},\\boldsymbol{\\beta}\\right)$ that predicts $y$, where the function $f$ is indexed by a vector of parameters $\\boldsymbol{\\beta} \\equiv \\left(\\beta_1,\\ldots, \\beta_p\\right)$. The hope is that, for any new observation ${\\bf x}^*= (x_1^*, x_2^*,\\ldots, x_p^*)^T$, we can predict its outcome $y^*$ with $f({\\bf x}^*,\\boldsymbol{\\beta})$.  To this end, we need to find the **best** $\\boldsymbol{\\beta}$ for predicting our data. In other words, we search for the vector $\\boldsymbol{\\beta}$ that minimizes $\\mathbb{E}\\left[L\\left(y,  f\\left({\\bf x},\\boldsymbol{\\beta}\\right) \\right)\\right]$, where $L(\\cdot,\\cdot)$ is the loss function or criterion of interest. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can summarize the framework of a general prediction problem as follows. \n",
    "\n",
    "\n",
    "1. Obtain a data set $\\{(y_i,{\\bf x}_i)\\}_{i=1}^n$.\n",
    "2. Choose a criterion $L(y, \\hat{y})$ between the outcome and its predictor.\n",
    "3. Choose a model $\\hat{y}=f({\\bf x},\\boldsymbol{\\beta})$ to produce the prediction. \n",
    "4. Find the $\\hat{\\boldsymbol{\\beta}}$ that optimizes the criterion.\n",
    "5. For any given new observations ${\\bf x}^*$, predict its outcome with $f({\\bf x}^*,\\hat{\\boldsymbol{\\beta}})$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many moving pieces in this framework, in particular, one needs to choose the criterion $L$ and the model $f$. We will visit a few classic examples in this chapter. \n",
    "\n",
    "In general, there are lots of candidate methods for building predictive models. There is no best method **overall**, but there often is a best method **for your data**. There are two keys for a method to do well on your data:\n",
    "    \n",
    "- It does not <span style=\"color:red\"> completely</span> miss important structure\n",
    "- It admits a proper amount of <span style=\"color:blue\"> complexity</span> for the <span style=\"color:blue\"> number of observations</span> in your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example: Hotel booking demand data.** As a motivating example, we consider the Hotel booking demand data from the Kaggle Open Datasets ([link](https://www.kaggle.com/jessemostipak/hotel-booking-demand)). This is a publicly available data set with [many suggestions](https://www.kaggle.com/jessemostipak/hotel-booking-demand/notebooks?datasetId=511638&sortBy=voteCount) on how to analysis it. We consider two questions in this chapter. \n",
    "\n",
    "1. As a customer, predict the average daily rate (`adr`) using available features.\n",
    "2. As a hotel manager, predict the cancellation (`is_canceled`) using available features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[36m--\u001b[39m \u001b[1m\u001b[1mColumn specification\u001b[1m\u001b[22m \u001b[36m------------------------------------------------------------------------------------------------\u001b[39m\n",
      "cols(\n",
      "  .default = col_double(),\n",
      "  hotel = \u001b[31mcol_character()\u001b[39m,\n",
      "  arrival_date_month = \u001b[31mcol_character()\u001b[39m,\n",
      "  meal = \u001b[31mcol_character()\u001b[39m,\n",
      "  country = \u001b[31mcol_character()\u001b[39m,\n",
      "  market_segment = \u001b[31mcol_character()\u001b[39m,\n",
      "  distribution_channel = \u001b[31mcol_character()\u001b[39m,\n",
      "  reserved_room_type = \u001b[31mcol_character()\u001b[39m,\n",
      "  assigned_room_type = \u001b[31mcol_character()\u001b[39m,\n",
      "  deposit_type = \u001b[31mcol_character()\u001b[39m,\n",
      "  agent = \u001b[31mcol_character()\u001b[39m,\n",
      "  company = \u001b[31mcol_character()\u001b[39m,\n",
      "  customer_type = \u001b[31mcol_character()\u001b[39m,\n",
      "  reservation_status = \u001b[31mcol_character()\u001b[39m,\n",
      "  reservation_status_date = \u001b[34mcol_date(format = \"\")\u001b[39m\n",
      ")\n",
      "\u001b[36mi\u001b[39m Use \u001b[30m\u001b[47m\u001b[30m\u001b[47m`spec()`\u001b[47m\u001b[30m\u001b[49m\u001b[39m for the full column specifications.\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[90m# A tibble: 119,390 x 32\u001b[39m\n",
      "   hotel is_canceled lead_time arrival_date_ye~ arrival_date_mo~\n",
      "   \u001b[3m\u001b[90m<chr>\u001b[39m\u001b[23m       \u001b[3m\u001b[90m<dbl>\u001b[39m\u001b[23m     \u001b[3m\u001b[90m<dbl>\u001b[39m\u001b[23m            \u001b[3m\u001b[90m<dbl>\u001b[39m\u001b[23m \u001b[3m\u001b[90m<chr>\u001b[39m\u001b[23m           \n",
      "\u001b[90m 1\u001b[39m Reso~           0       342             \u001b[4m2\u001b[24m015 July            \n",
      "\u001b[90m 2\u001b[39m Reso~           0       737             \u001b[4m2\u001b[24m015 July            \n",
      "\u001b[90m 3\u001b[39m Reso~           0         7             \u001b[4m2\u001b[24m015 July            \n",
      "\u001b[90m 4\u001b[39m Reso~           0        13             \u001b[4m2\u001b[24m015 July            \n",
      "\u001b[90m 5\u001b[39m Reso~           0        14             \u001b[4m2\u001b[24m015 July            \n",
      "\u001b[90m 6\u001b[39m Reso~           0        14             \u001b[4m2\u001b[24m015 July            \n",
      "\u001b[90m 7\u001b[39m Reso~           0         0             \u001b[4m2\u001b[24m015 July            \n",
      "\u001b[90m 8\u001b[39m Reso~           0         9             \u001b[4m2\u001b[24m015 July            \n",
      "\u001b[90m 9\u001b[39m Reso~           1        85             \u001b[4m2\u001b[24m015 July            \n",
      "\u001b[90m10\u001b[39m Reso~           1        75             \u001b[4m2\u001b[24m015 July            \n",
      "\u001b[90m# ... with 119,380 more rows, and 27 more variables:\u001b[39m\n",
      "\u001b[90m#   arrival_date_week_number \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m, arrival_date_day_of_month \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m,\u001b[39m\n",
      "\u001b[90m#   stays_in_weekend_nights \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m, stays_in_week_nights \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m, adults \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m,\u001b[39m\n",
      "\u001b[90m#   children \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m, babies \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m, meal \u001b[3m\u001b[90m<chr>\u001b[90m\u001b[23m, country \u001b[3m\u001b[90m<chr>\u001b[90m\u001b[23m,\u001b[39m\n",
      "\u001b[90m#   market_segment \u001b[3m\u001b[90m<chr>\u001b[90m\u001b[23m, distribution_channel \u001b[3m\u001b[90m<chr>\u001b[90m\u001b[23m, is_repeated_guest \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m,\u001b[39m\n",
      "\u001b[90m#   previous_cancellations \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m, previous_bookings_not_canceled \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m,\u001b[39m\n",
      "\u001b[90m#   reserved_room_type \u001b[3m\u001b[90m<chr>\u001b[90m\u001b[23m, assigned_room_type \u001b[3m\u001b[90m<chr>\u001b[90m\u001b[23m, booking_changes \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m,\u001b[39m\n",
      "\u001b[90m#   deposit_type \u001b[3m\u001b[90m<chr>\u001b[90m\u001b[23m, agent \u001b[3m\u001b[90m<chr>\u001b[90m\u001b[23m, company \u001b[3m\u001b[90m<chr>\u001b[90m\u001b[23m, days_in_waiting_list \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m,\u001b[39m\n",
      "\u001b[90m#   customer_type \u001b[3m\u001b[90m<chr>\u001b[90m\u001b[23m, adr \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m, required_car_parking_spaces \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m,\u001b[39m\n",
      "\u001b[90m#   total_of_special_requests \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m, reservation_status \u001b[3m\u001b[90m<chr>\u001b[90m\u001b[23m,\u001b[39m\n",
      "\u001b[90m#   reservation_status_date \u001b[3m\u001b[90m<date>\u001b[90m\u001b[23m\u001b[39m\n"
     ]
    }
   ],
   "source": [
    "library(tidyverse)\n",
    "hotel <- read_csv(\"../Data/hotel_bookings.csv\") %>% print\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning message:\n",
      "\"Removed 2 rows containing non-finite values (stat_boxplot).\"\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAANlBMVEUAAAAzMzNNTU1oaGh8\nfHyMjIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enr6+vw8PD////agy6EAAAACXBIWXMA\nABJ0AAASdAHeZh94AAAgAElEQVR4nO2diXbbOBAEtUxiJ5usHf7/z65FURye4NUcj4Hq92Id\noFWA3BVKlCXfakLI6dw+ewKE5BBEIkQQRCJEEEQiRBBEIkQQRCJEEEQiRBBEIkQQb5HeV7K6\ngTKusIyXVi4MkfxhGS+tXBgi+cMyXlq5METyh2W8tHJhiOQPy3hp5cIQyR+W8dLKhSGSPyzj\npZULQyR/WMZLKxeGSP6wjJdWLgyR/GEZL61cGCL5wzJeWrkwRPKHZby0cmGI5A/LeGnlwhDJ\nH5bx0sqFlSTSP2einEioBgCTwIoSaXkyH6akZ4tIwNLjXRApFUQClh7vgkipIBKw9HgXREoF\nkYClx7sgUiqIBCw93gWRUkEkYOnxLoiUCiIBS493QaRUEAlYerwLIqWCSMDS410QKRVEApYe\n74JIqSASsPR4F0RKBZGApce7IFIqiAQsPd4FkVJBJGDp8S6IlAoiAUuPd0GkVBAJWHq8CyKl\ngkjA0uNdECkVRAKWHu+CSKkgErD0eJcNIlUf2WbJ8xsSY2dnvj+IBOwymFVlXaRqyY1FXxBp\nNoOl7f4UlsdHsdiF9/EXu8kOtuPzW0590EuobrvCrCqrIlWLciDSvvSXttejjaaZSbVBNt1J\n25VbWdnlCQWzqmwVqTn7eIxX1c1p1fvau84uDAYRqb80vUcjq+6wvldr99F25VZWdn1CwfaI\n1D1Dej7Ga+Woq8l11XSD7nJT27Xithtpk4S5z8Ylw8mv3efdpusbkqVsOmr33LHUQz9mrpts\n0J22OftfwP6wR9pwH23dcHVl1ycUbKdIrQ9V+2iuL9LoutGeqDeISDxHkiQU7JhIdna4R6pT\ne6Qakdpw1E6SUDCryo6jdoh0LqEaAEwC2yNS50jyYEO9sAHPkbqEagAwCWyHSL3jdsND3fXw\n8PfAqvHhb0R6D9YAYBLYLpHG2ff7QqOcnfn+IBKwy2BWFURKBZGApce7IFIqiAQsPd6Ft1Gk\ngkjA0uNdECkVRAKWHu+CSKkgErD0eBdESgWRgKXHuyBSKogELD3eBZFSQSRg6fEuiJQKIgFL\nj3dBpFQQCVh6vAsipYJIwNLjXRApFUQClh7vgkipIBKw9HgXREoFkYClx7sgUiqIBCw93gWR\nUkEkYOnxLgWIdCbKiYRqADAJrCSRosAyXlq5METyh2W8tHJhiOQPy3hp5cIQyR+W8dLKhSGS\nPyzjpZULQyR/WMZLKxeGSP6wjJdWLgyR/GEZL61cGCL5wzJeWrkwRPKHZby0cmGI5A/LeGnl\nwhDJH5bx0sqFIZI/LOOllQtDJH9YxksrF4ZI/rCMl1YuLG+RPv2dR/MJ1QBgEljmIi3BE++I\nRSRgB2BWIERqhwTclYRqADAJzAqESO2QgLuSUA0AJoFZgRCpHRJwVxKqAcAkMCsQIrVDAu5K\nQjUAmARmBUKkdkjAXUmoBgCTwKxAiNQOCbgrCdUAYBKYFQiR2iEBdyWhGgBMArMCIVI7JOCu\nJFQDgElgViBEaocE3JWEagAwCcwKhEjtkIC7klANACaBWYEQqR0ScFcSqgHAJDArECK1QwLu\nSkI1AJgEZgVCpHZIwF1JqAYAk8CsQIjUDgm4KwnVAGASmBUIkdohAXcloRoATAKzAiFSOyTg\nriRUA4BJYFYgRGqHBNyVhGoAMAnMCnRQpOojy4OJbzw78y1BJGBOMCvQIZEeEg1UWjo/ztmZ\nb8nni3Tir9aOPknCPk2iOTf+cInB5d72Lh9C0SVUt11hVqBjIg1OkufHOTvzLfl0kWQetTYN\nb3MMmlBH11+fUN12hVmBjohU9c88HuT1v35cXT33Wd3lkkQSe9QoMRGrB1qiCpeUTqhuu8J0\nIlW90+58NRqrnjVeLnKv6+eTuPFLuVdlNL/xpJdWsHZfE2EkItUjkeZO25z9L2BL2CONtrs6\noXYSrjAnkar2EV9hIvEcqRiYk0jj7etCROKoXSkwK9DZo3aItDmhGgBMArMCnX0dadPBBkS6\nJ1QDgElgJ0Xq/2bD8xB38vA3Ir0HawAwCey0SIdzduZbgkjAnGBWIERqhwTclYRqADAJzAqE\nSO2QgLuSUA0AJoFZgRCpHRJwVxKqAcAkMCsQIrVDAu5KQjUAmARmBUKkdkjAXUmoBgCTwKxA\niNQOCbgrCdUAYBKYFQiR2iEBdyWhGgBMArMCIVI7JOCuJFQDgElgViBEaocE3JWEagAwCcwK\nhEjtkIC7klANACaBWYEQqR0ScFcSqgHAJDArECK1QwLuSkI1AJgEZgVCpHZIwF1JqAYAk8Cs\nQIjUDgm4KwnVAGASmBUIkdohAXcloRoATAKzAiFSOyTgriRUA4BJYFagLEU69DEjlydUA4BJ\nYHmLtDmusIyXVi4MkfxhGS+tXBgi+cMyXlq5METyh2W8tHJhiOQPy3hp5cIQyR+W8dLKhSGS\nPyzjpZULQyR/WMZLKxeGSP6wjJdWLgyR/GEZL61cGCL5wzJeWrkwRPKHZby0cmGI5A/LeGnl\nwhDJH5bx0sqFIZI/LOOllQvLUKQjbz1CJGDnYDmKNIube08sIgFTwaxViOSWUA0AJoFZqxDJ\nLaEaAEwCs1YhkltCNQCYBGatQiS3hGoAMAnMWoVIbgnVAGASmLUKkdwSqgHAJDBrFSK5JVQD\ngElg1ipEckuoBgCTwKxViOSWUA0AJoFZqxDJLaEaAEwCs1YhkltCNQCYBGatQiS3hGoAMAnM\nWoVIbgnVAGASmLUKkdwSqgHAJDBrFSK5JVQDgElg1ipEckuoBgCTwKxViOSWUA0AJoFZqxDJ\nLaEaAEwCs1ZtE6kanGzcejZnZ74YRALmD7NWbRSpWjWkv3Vi7OzMF+Mk0uYPWJk5N7/V++PL\n++iDWR7XvPeu+mfPX4wOVbd8YdaqrXukas2QwdbLOTvzxfiItNWj3elueYbVu7h5oqHqli/M\nWrVbpKq67566PdTj4n2D6nHJLgwG8xDpMo/6Sk1ZvYtbZxqqbvnC9ov0kKh+CvUU6enXQ5iq\nf6EeDjalnqn1tPkHs12k2JmZZu/iph8X8c9BkeYuTi4MTtuc/S9gMeyRBgn1/3a+sAMitUaM\nRap6D/UmAvUHsxCJ50jAhuMHRHo8kLPnSk+X+hvUs3uijETiqB2wwTginYIdSqgGAJPADolU\nTx7DLT1HmnsChUihGgBMAjsmUu/wd0+uqtszmTjjw9+I5E0D5gLbK5IuZ2e+GEQC5g+zViGS\nW0I1AJgEZq1CJLeEagAwCcxahUhuCdUAYBKYtQqR3BKqAcAkMGsVIrklVAOASWDWKkRyS6gG\nAJPArFWI5JZQDQAmgVmrEMktoRoATAKzViGSW0I1AJgEZq1CJLeEagAwCcxahUhuCdUAYBKY\ntQqR3BKqAcAkMGsVIrklVAOASWDWKkRyS6gGAJPArFWI5JZQDQAmgVmrEMktoRoATAKzVuUj\n0vachx1KqAYAk8AyFOlIEAnYORgi+cMyXlq5METyh2W8tHJhiOQPy3hp5cIQyR+W8dLKhSGS\nPyzjpZULQyR/WMZLKxeGSP6wjJdWLgyR/GEZL61cGCL5wzJeWrkwRPKHZby0cmGI5A/LeGnl\nwhDJH5bx0sqFIZI/LOOllQtDJH9YxksrF1aeSDverrT07qWzCdUAYBJYgSKlJjX3PtpuTDaF\nUA0AJoFZTxCpRiRgR2HWE0SqEQnYUZj1BJFqRAJ2FGY9QaQakYAdhVlPEKlGJGBHYdYTRKoR\nCdhRmPUEkWpEAnYUZj1BpBqRgB2FWU8QqUYkYEdh1hNEqhEJ2FGY9QSRakQCdhRmPUGkGpGA\nHYVZTxCpRiRgR2HWE0SqEQnYUZj1BJFqRAJ2FGY9QaQakYAdhVlPEKlGJGBHYdYTRKoRCdhR\nmPVELFI1Op3k7MyPBpGAXQCznqhFqvonM1mZmPLDRoY3nJq1v0jpT1uZuS611fNu665tYd1n\nt/Rv5YnvnUhXdnkMdllZ5mAL411iiaT92J7BLadm7S5SyqOj6d9oA5sa97x7+9eLV3Z9Oth1\nZZmBLY13kT+0q+xrdfepNeu5QXJa9qOWJ5RIV3g0sqVehkzF0q3MIU+YbvobYIvjXa4TqbLL\njwvNupPffVGrVsGp4SvndF1SEx+N7PsBx0mw6V+4R3pcbq94Jul3e9/o/kPp3XRq1qmfB3uk\nPStzSCl7pP7uqHtst1EkniOdcWcgB8+RtLCl8S4XimT/eoceViZ22V0TSySO2ilg13sUQqT2\nOdI+kXgdCdhXgllPLnhBtv33OHBXI9IzoRoATAKznlwn0v0pku2Wnjk786NBJGAXwKwnHr9r\nh0hNQjUAmARmPUGkGpGAHYVZT64XafjrQmdnfjSIBOwCmPWEt1HUiATsKMx6gkg1IgE7CrOe\nIFKNSMCOwqwniFQjErCjMOsJItWIBOwozHqCSDUiATsKs54gUo1IwI7CrCeIVCMSsKMw6wki\n1YgE7CjMeoJINSIBOwqzniBSjUjAjsKsJ4hUIxKwozDrCSLViATsKMx6gkg1IgE7CrOeFCPS\n4cimEKoBwCSw8kT6fFjGSysXhkj+sIyXVi4MkfxhGS+tXBgi+cMyXlq5METyh2W8tHJhiOQP\ny3hp5cIQyR+W8dLKhSGSPyzjpZULQyR/WMZLKxeGSP6wjJdWLgyR/GEZL61cGCL5wzJeWrkw\nRPKHZby0cmGI5A/LeGnlwvIWafPbIxAJ2DlY5iJNqDPv3EMkYOdh1idEckuoBgCTwKxPiOSW\nUA0AJoFZnxDJLaEaAEwCsz4hkltCNQCYBGZ9QiS3hGoAMAnM+oRIbgnVAGASmPUJkdwSqgHA\nJDDrEyK5JVQDgElg1idEckuoBgCTwKxPiOSWUA0AJoFZnxDJLaEaAEwCsz4hkltCNQCYBGZ9\nQiS3hGoAMAnM+oRIbgnVAGASmPUJkdwSqgHAJDDrEyK5JVQDgElg1idEckuoBgCTwKxPiOSW\nUA0AJoFZn06LVI1Ol8bbnJ35liASMCeY9em8SFX/ZGZ8eHFlYpK/33qhSMf/FO3yR7D0Tppz\nvcHpPTG8U56XpndVqLrlC7M+xRJprmf7c51Ico86WVZvujeB4XQm1zYJVbd8YdYnwUO7yr5W\nVXuhGlzoyZSc1mJ/9uUykSTmHE1vAsPpjK59JFTd8oVdI1L1uFy3Ao0vND/t5G2pOjd3w1ex\n3NKb9HAJo2vJp0S6R3pc7hyqu1PbJSX9nqnPkf9G2CMF+387X5hUpP7uqHkYd1gkniMltrMJ\nDKczubZJqLrlC7tIpKo+t0fiqB1H7b4W7BqRuqdFx0XidSRgXwmmFam/Jxo+tBtbhUjAsoJd\nI1JzoPvpjh3+RqRHQjUAmAQmFGlnzs58SxAJmBPM+oRIbgnVAGASmPUJkdwSqgHAJDDrEyK5\nJVQDgElg1idEckuoBgCTwKxPiOSWUA0AJoFZnxDJLaEaAEwCsz4hkltCNQCYBGZ9QiS3hGoA\nMAnM+oRIbgnVAGASmPUJkdwSqgHAJDDrEyK5JVQDgElg1idEckuoBgCTwKxPiOSWUA0AJoFZ\nnxDJLaEaAEwCsz4hkltCNQCYBGZ9QiS3hGoAMAnM+pSlSFs/dgSRgJ2D5S3S5iASsHMwRPKH\nZby0cmGI5A/LeGnlwhDJH5bx0sqFIZI/LOOllQtDJH9YxksrF4ZI/rCMl1YuDJH8YRkvrVwY\nIvnDMl5auTBE8odlvLRyYYjkD8t4aeXCEMkflvHSyoUhkj8s46WVC0Mkf1jGSysXhkj+sIyX\nVi6scJEO/MVkQUI1AJgEVrpIC5ObeSNtc7WGGqoBwCQwKwki9YJIwPbBrCSI1AsiAdsHs5KM\nRXqtbo9ovJnk7MwlQSRgGpiVZCTM6+2GSJOrNdRQDQAmgVlJRsJUt58iYxZyduaSIBIwDcxK\nMhLpsj3RM2dnLgkiAdPArCSTh3Z/RcYs5OzMJUEkYBqYlWS8B/rx/U1jzELOzlwSRAKmgVlJ\n+iLd+tG5M8jZmUuCSMA0MCsJIvWNQSRgu2BWEl6Q7QWRgO2DWUkQqRdEArYPZiVZOPxdVQJp\n5nJ25pIgEjANzErSF6niORIiAdsDs5L0hfnV8+iXzp1Bzs5cEkQCpoFZSfjNhl4QCdg+mJWE\ngw29IBKwfTArCW+j6BuDSMB2wawk2rdRrB/rOztzSRAJmAZmJZm8jeLUUYbTIjWfNNJ95Mjo\no0dkH0PyKSLt/cSVPZ/NMjx7p9UddPBvNB/JwoJ12xVmJdEebDgr0lJRbFBx73yKSNd5NHuH\n1fPQ8XwUKwvWbVeYlWQkzo9zb6OoWpea0+rxqm5V9V/dTU4rUYzefup8PkGkC3RJpG3A7Mhw\nPoKlBeu2K8xKMhLprTr1Noq+SNXwclPT+aJaYxeSHDySZf6eacXN8sTHa9rxwyWpTB7anT3Y\nUC2ctkn6vdSL/qDi/xn2SOyRNDBPkarBY7v0vJZqYYOKe4fnSO88R9LAFkU6maU9kmVlYv9w\n1O6IN8OzdxpH7VxgVhKZSEsP6faJxOtIogBzgVlJxiL9ff12u3173X/sri9OdfA50oaZS4JI\nwDSwRZHe2t8QqvYfu6vscPdAoB2HvzfMXBJEAqaBLYr0crsf/n77fnvZLdK2nJ25JIgETAOz\nkiz8ZgO/tNq/WkMN1QBgEpiVBJF6QSRg+2BWEh7a9Y1BJGC7YFYS3cGGbTk7c0kQCZgGZiWR\nHf7emLMzlwSRgGlgVhLeat4LIgHbB7OSIFIviARsH8xKgki9IBKwfTAryUikvy98QOT0ag01\nVAOASWBWksk7ZBFperWGGqoBwCQwK8nkBdl/RcYs5OzMJUEkYBqYlWQk0jc+aXXmag01VAOA\nSWBWkvELshe+hNTk7MwlQSRgGpiVZLwH+pfnSNOrNdRQDQAmgVlJONjQNwaRgO2CWUk42NAL\nIgHbB7OSTPZIImGWcnbmkuz/dBFFQjUAmAS2KFL94+Wq3/sOJNKnwDJeWrmwRZHK+NOXnwLL\neGnlwhDJH5bx0sqFLYp0ec7OXBpEAnYOhkj+sIyXVi4MkfxhGS+tXBgi+cMyXlq5METyh2W8\ntHJhiOQPy3hp5cIQyR+W8dLKhc2LdOsHkb40DZgLDJH8YRkvrVzYvEgeOTtzaRAJ2DkYIvnD\nMl5aubBlkV55aJcFDZgLbFGk1wKeI+3/a8eadymFagAwCWxRpOr23/fb29/vtz8Zi5Sa38Kb\nZOe23MsN1QBgEpjVYfqHxn7eftd/b993CrI1Z2cuCCIBU8GsDlORft9+5f0X+xAJmApmdZh8\nZsO/b7dv9R9EWg0iAVsW6W7Q9/uxhoz/9CUiAVPBrA7jPc/vb/c/JHt73afH9pyduSCIBEwF\nszqMRPpvpxe7c3bmgiASMBXM6jA+2PDt904zdubszAVBJGAqmNVh8tcobtXPKz9G/+zMBUEk\nYCqY1WH8HOnttbrdflz1ciwieQWYC8zqMHOY+8/r7fbtqo8APztzQRAJmApmdZh9vegt79+1\nS80PkYDtgFkd5vZILx97pF875NiTszMXBJGAqWBWh9nnSC88R1oPIgFLHrX79oujdluCSMAS\nryP94HWkjStBJGDLIl37l5hrRPIKMBeY1WH4KUJF/FkXRAKmglkdEGmkByIB2w6zOlwhTDV7\n9pGzMxcEkYCpYFaHkUjfFe9DQqTZHF2a4nNaRp/aYh/dMv4sl/bc3Ce8zH/ey/3az+j2gU+f\nOQ5bHu8y+fCTjTVKBZFmc3BpAo/m5RrfuF1+H2/WDc3P7hNEmp/NRbDEeJfx+5G+v57/q+ZV\nK1BzWj2veuTszAX5WiJd5dFwx9NesbBZbx7zszu0soOpPbnHX0dSHGxYEKlZ+mo9HbLC37hK\nj5lemska5tc0XO78nbDxLtPls7jLuVyk5qw9wjv7X4Ag7JEemdw4e6R5WGq8y0VH7RBpGp4j\nSfI1niOpjtr1ROoe3jU5O3NBvphIHLUbpoyjdn2BEGmYUC+AAJPArA7io3YDgRBpkFANACaB\nWR3UBxuqqqra0/6eqcvZmQuCSMBUMKvDFUftxkGkJqEaAEwCszp4/MU+RGoSqgHAJDCrw/Ui\nVYNfEzo7c0EQCZgKZnWYF+nP6+SX5EQ5O3NBEAmYCmZ1mBHp90t1uyHSWhAJWOqvUbw0f9Tl\nsk9uODtzQRAJmApmdRiI9LDodrvwkxvOzlwQRAKmglkdhm81b/ZFl73N/J6zMxcEkYCpYFaH\noUg//tYX/v3Ye87OXBBEAqaCWR3YIw2DSMB2wKwOPEcaBpGA7YBZHRaO2l324d9nZy4IIgFT\nwawOvI40DCIB2wGzOvCbDcMgErAdMKuDxy+t9nN25oIgEjAVzOqASMMgErAdMKtDiSKJspcb\nqgHAJLCiRfokWMZLKxeGSP6wjJdWLgyR/GEZL61cGCL5wzJeWrkwRPKHZby0cmGI5A/LeGnl\nwhDJH5bx0sqFIZI/LOOllQtDJH9YxksrF4ZI/rCMl1YuDJH8YRkvrVwYIvnDMl5auTBE8odl\nvLRyYYjkD8t4aeXCEMkflvHSyoXlJ9KhtxAhErBzsAxFmoON3+6KSMC0MOsWIrklVAOASWDW\nLURyS6gGAJPArFuI5JZQDQAmgVm3EMktoRoATAKzbiGSW0I1AJgEZt1CJLeEagAwCcy6hUhu\nCdUAYBKYdQuR3BKqAcAkMOsWIrklVAOASWDWLURyS6gGAJPArFuI5JZQDQAmgVm3EMktoRoA\nTAKzbiGSW0I1AJgEZt1CJLeEagAwCcy6hUhuCdUAYBKYdQuR3BKqAcAkMOsWIrklVAOASWDW\nLURyS6gGAJPArFvHRaoOfdfZmS8FkYB9Asy6hUhbbvzcX2xuzy39/ebm+m7wwF95nvueUHXL\nF2bdQqQNt30mw29P3np3ae/aZ74nVN3yhVm3zolUfaQ5+zydv75n3dmZL+VCkU55NPUqcevd\npX1Ln/ueUHXLFyYS6SnO06Jq5noTqflxr93q4YYu3JjmxnVJLri7tOsHceR7iDrnH9rNCDS+\nvvcw8Ox/AUthj3RkaaKUC5OJ9HgMNxVpeP3XFonnSGspF6YSaeEh3eD6+zMl+66zM18KR+2O\nLU2ScmEnRZo8F0qdZiCSJqEaAEwCE4o0fmhXja9HpM+gAXOBnRTpeXi7Obo9eEj3uDy4fviK\n09mZLwWRgH0C7KxI+4JI/jRgLjDrFiK5JVQDgElg1q3rRaoGv0t0duZLQSRgnwCzbvE2CreE\nagAwCcy6hUhuCdUAYBKYdQuR3BKqAcAkMOsWIrklVAOASWDWLURyS6gGAJPArFuI5JZQDQAm\ngVm3EMktoRoATAKzbiGSW0I1AJgEZt1CJLeEagAwCcy6hUhuCdUAYBKYdQuR3BKqAcAkMOsW\nIrklVAOASWDWLURyS6gGAJPArFuI5JZQDQAmgVm3shFp80eRCGDHEqoBwCSw/EQ6FEQCdg6G\nSP6wjJdWLgyR/GEZL61cGCL5wzJeWrkwRPKHZby0cmGI5A/LeGnlwhDJH5bx0sqFIZI/LOOl\nlQtDJH9YxksrF4ZI/rCMl1YuDJH8YRkvrVwYIvnDMl5auTBE8odlvLRyYYjkD8t4aeXCEMkf\nlvHSyoXlKtLOtyQhErBzsGxFmgJH75JFJGBCmBULkdwSqgHAJDArFiK5JVQDgElgVixEckuo\nBgCTwKxYiOSWUA0AJoFZsRDJLaEaAEwCs2IhkltCNQCYBGbFQiS3hGoAMAnMioVIbgnVAGAS\nmBULkdwSqgHAJDArFiK5JVQDgElgVixEckuoBgCTwKxYiOSWUA0AJoFZsRDJLaEaAEwCs2Ih\nkltCNQCYBGbFQiS3hGoAMAnMioVIbgnVAGASmBULkdwSqgHAJDArFiK5JVQDgElgVqydIlWP\n9K7Y9/2IBCwnmBVrr0irV6xkZWKpP528IVeItO3PPA8nvjLU3OjMjb8/vyxMox0ygo2NNw9V\nt3xhVqxYIo1btzcXiHTAo41J3XhiGu+T7ea+I1Td8oVZsQ6L1D7Cq2o7fYxW08He9yWntd6n\nlehF0jizYEtqbOM0bGz4DaHqli/svEhPZ1phutPHhd6gidT8uJO3fr6b6yIFSmoyW+8ZG9v1\ncyTqHDrYUM84U9f9L6MrbUeW9HtSk73/gbBH2rE0XcqFHRdpbNSCSMPBrSLxHGlx5b2hyXZz\n3xGqbvnCBCL1TyciVb0r66p/RGJlYuk2rYajdruWpkq5sItFGpzuEonXkYB9JZhIpP7BhueV\nz4d0FSKNEqoBwCSw8yJND3+3Vz4Pf9seafBa09mZp4NIwFxhh0U6FETypwFzgVmxEMktoRoA\nTAKzYl0vUv9XXBEJWFYwKxZvo3BLqAYAk8CsWIjkllANACaBWbEQyS2hGgBMArNiIZJbQjUA\nmARmxUIkt4RqADAJzIqFSG4J1QBgEpgVC5HcEqoBwCQwKxYiuSVUA4BJYFYsRHJLqAYAk8Cs\nWIjkllANACaBWbEQyS2hGgBMArNiIZJbQjUAmARmxUIkt4RqADAJzIqFSG4J1QBgEpgVKzOR\nNnwwiQy2N6EaAEwCy1WknUEkYOdgiOQPy3hp5cIQyR+W8dLKhSGSPyzjpZULQyR/WMZLKxeG\nSP6wjJdWLgyR/GEZL61cGCL5wzJeWrkwRPKHZby0cmGI5A/LeGnlwhDJH5bx0sqFIZI/LOOl\nlQtDJH9YxksrF4ZI/rCMl1YuDJH8YRkvrVxYviJteEeSvTEJkYCdg2Us0hg4eoesXS+A7Uuo\nBgCTwKxPiOSWUA0AJoFZnxDJLaEaAEwCsz4hkltCNQCYBGZ9QiS3hGoAMAnM+oRIbgnVAGAS\nmPUJkdwSqgHAJDDrEyK5JVQDgElg1idEckuoBgCTwKxPiOSWUA0AJoFZnxDJLaEaAEwCsz4h\nkltCNQCYBGZ9QiS3hGoAMAnM+oRIbgnVAGASmPUJkdwSqgHAJDDrEyK5JVQDgElg1idEckuo\nBgCTwKLrDnwAAA00SURBVKxPiOSWUA0AJoFZn7aIVD2yWZbUlmdnvhZEAuYIsz5tEmndj7nt\nZ7Myscmfed0bJ5G2fzTE5o+QeK78n/4ZAzWXp/DnFf2Z9TbpTfd9eL5/a8PvHS90cGb2B3Tw\nfjz2o/76Im026bhIwx//kfiIpHVo4MX4zHh8PPA+LPfj/PAb3ntnn+eHt9b/3ulCe2f6t2o5\ndj/O3dKGZCLS8zHe6LSqm9PBhcHgJpGGP/5DcRFpsxf6TOGDcs99w2D3tHBrvaGZhc5ABzl0\nP87e0obkIVK1dPqQrH9hNNhUeqHUvdKfzmaRvmamE6+7kdlV9a6cblDX06GZO2kGej66WwqR\nMyLVU6GWbWuT9Lv7OR3/L4Q90uQb2CMdz3V7pPb4XXdV//J0NzXceFUkniPxHGk1mYg0vcpO\nl/ZE20XiqB1H7daCSFtE4nUkYF8JdkikasmVyUO78cEGRHr3pgFzge0UqfebDYuHvweWzQwi\nUqgGAJPA9omkzNmZrwWRgDnCrE+I5JZQDQAmgVmfEMktoRoATAKzPiGSW0I1AJgEZn1CJLeE\nagAwCcz6hEhuCdUAYBKY9QmR3BKqAcAkMOsTIrklVAOASWDWJ0RyS6gGAJPArE+I5JZQDQAm\ngVmfEMktoRoATAKzPiGSW0I1AJgEZn1CJLeEagAwCcz6hEhuCdUAYBKY9QmR3BKqAcAkMOsT\nIrklVAOASWDWJ0RyS6gGAJPArE/ZibT900YQCdhZWL4i7QoiATsHQyR/WMZLKxeGSP6wjJdW\nLgyR/GEZL61cGCL5wzJeWrkwRPKHZby0cmGI5A/LeGnlwhDJH5bx0sqFIZI/LOOllQtDJH9Y\nxksrF4ZI/rCMl1YuDJH8YRkvrVwYIvnDMl5auTBE8odlvLRyYYjkD8t4aeXCEGn7+5aO5FOX\nBswPhkjzb5sd55+Ft9eufdunLg2YH8x+5oiUCiIBS493QaRUEAlYerwLIqWCSMDS410QKRVE\nApYe74JIqSASsPR4F0RKBZGApce7IFIqiAQsPd4FkVJBJGDp8S6IlAoiAUuPd0GkVBAJWHq8\nCyKlgkjA0uNdECkVRAKWHu+CSKkgErD0eBdESgWRgKXHuyBSKogELD3eBZFSQSRg6fEuiJQK\nIgFLj3dBpFQQCVh6vMtJkaq933B25qogEjABzH7m50SqtpnU2+jszFUpRKRdH9jSnet9gEv/\n81yeQ+/dVk/Y0ue9vL/3R4ZbLX/PppX5BJHWUoZIOzyaM2twA7M39m6Y2Wn0R4ZbLX/PppU5\nxVmk7ktVNdf0L1fPr4+cnbkqRYh0yqP+Piq1TbfVzCz6I8Otlr9n08q84iRS1fv3FKiqhpe7\n07q551Y76JVNC9y63Wct4tpsWUhvuct3xOR86nu+aOQiTQXioZ3v0gx2UiT2SE57pOr5dUmk\nuqp6GzY5O3NVihCJ50hn4yTSIwmRWpUQyXNpfdgub4bKjG7gnaN2c+Ndrt0j9S8/cnbmqhQi\nErBrYfYzPyGSmdI/yDA+RSTvpQHzg9nPXCLS87D33OnzYpuzM1cFkYAJYPYz1/+uXfol2rMz\nVwWRgAlg9jOXijR4bjSfszNXBZGACWD2M9fukfq/wzCfszNXBZGACWD2M+dtFKkgErD0eBdE\nSgWRgKXHuyBSKogELD3eBZFSQSRg6fEuiJQKIgFLj3dBpFQQCVh6vAsipYJIwNLjXRApFUQC\nlh7vgkipIBKw9HgXREoFkYClx7sgUiqIBCw93gWRUkEkYOnxLoiUCiIBS493QaRUEAlYerxL\nsSJdmk9dGjA/GCL5wzJeWrkwRPKHZby0cmGI5A/LeGnlwhDJH5bx0sqFIZI/LOOllQtDJH9Y\nxksrF4ZI/rCMl1YuDJH8YRkvrVwYIvnDMl5auTBE8odlvLRyYYjkD8t4aeXCEMkflvHSyoUh\nkj8s46WVC0Mkf1jGSysXhkj+sIyXVi6sLJEUbyYSJFQDgElghYk0M49/Ft/7ikjANsOsNog0\nGblgBk1CNQCYBGa1QaTJyAUzaBKqAcAkMKsNIk1GLphBk1ANACaBWW0QaTJywQyahGoAMAnM\naoNIk5ELZtAkVAOASWBWG0SajFwwgyahGgBMArPaINJk5IIZNAnVAGASmNUGkSYjF8ygSagG\nAJPArDaINBm5YAZNQjUAmARmtUGkycgFM2gSqgHAJDCrDSJNRi6YQZNQDQAmgVltEGkycsEM\nmoRqADAJzGqDSJORC2bQJFQDgElgVhtEmoxcMIMmoRoATAKz2iDSZOSCGTQJ1QBgEpjVBpEm\nIxfMoEmoBgCTwKw2iDQZuWAGTUI1AJgEZrVBpMnIBTNoEqoBwCQwq41apGr2Krv27MyPBJGA\nXQSz2ohFquZM2iHS/bNH5J9E4i/SkT/g/E+79vnvnlz53Gy4+eMefB82YHA3Ll84nlDddoVZ\nbWKJNNeg83EX6ahHqrwPGvC8auXCiYTqtivManONSB8nVdWe7nhot9SKk/EWyUWWZPoN6K5K\nXziTUN12hVlttCJVz3/tme607e5ie9sOz2Xh6j2ZR+2ZxBfLzHrWLpBzuUik9sLgtEnS79lS\nnP9vhT3SP+yRroFZbaQiVc+vB0XiOZIm7zxHcoJZbbQiPXJcJI7acdTuS8GsNrH2SLyOBOxL\nwaw2SpGq7nRwkAGRHgnVAGASmNXmIpG6w981Ij0TqgHAJDCrzTW/azf7qmyTszM/EkQCdhHM\naoNIk5ELZtAkVAOASWBWG0SajFwwgyahGgBMArPa8DaKycgFM2gSqgHAJDCrDSJNRi6YQZNQ\nDQAmgVltEGkycsEMmoRqADAJzGqDSJORC2bQJFQDgElgVhtEmoxcMIMmoRoATAKz2iDSZOSC\nGTQJ1QBgEpjVBpEmIxfMoEmoBgCTwKw2iDQZuWAGTUI1AJgEZrVBpMnIBTNoEqoBwCQwqw0i\nTUYumEGTUA0AJoFZbRBpMnLBDJqEagAwCcxqg0iTkQtm0CRUA4BJYFYbRJqMXDCDJqEaAEwC\ns9og0mTkghk0CdUAYBKY1aYIkXZ+AslFCdUAYBJYWSLFgGW8tHJhiOQPy3hp5cIQyR+W8dLK\nhSGSPyzjpZULQyR/WMZLKxeGSP6wjJdWLgyR/GEZL61cGCL5wzJeWrkwRPKHZby0cmGI5A/L\neGnlwhDJH5bx0sqFIZI/LOOllQtDJH9YxksrF/Z5Iq0l47+xne/SWBkiOSbfpbEyRHJMvktj\nZYjkmHyXxsriiUTIlwwiESIIIhEiCCIRIggiESIIIhEiSCyRqo989hzEeS5pfJpFMl3ZkZ9Z\nKJGq7ks2eS5pfJpFmnrlt7JDPzNEujY5i1TViNQFkRySp0jtavJbWdU/RaRAQaQvlaqu9j9o\nRaTrk2vdsl1ZuyxECpYs69YtJLuV8RwpaKr+l2yWWD2S4coQKWYq+5pV3e5hj9QFkS5O1TvJ\nqm73IFKXUCJl9OL4M88HQBm+/l8/K5bdyr78bzYQ8lWDSIQIgkiECIJIhAiCSIQIgkiECIJI\nhAiCSIQIgkiECIJI8XK7qX4qwxu6X1q/6V+Z/HqCcxApXH5/iPRbc1NHRJJZXFa418Ll5fbj\n9qK5qakUiHRRuNfC5Xb7ey/z39u35uK323/135fb7eVvM/Zf9b2u//y43arX++jb99u33033\nu20e+Rj48ZCi2/ixR2pv93nz45u83eZujawFkaLl98fu6OX+2O7H7e3j4tu98NW93vfi327f\nP4bvD/4+8lH7v9XjbN3bpkkz8OM+YBu3D+1emweO/95+PrYc3eTsrZHVIFK03CW6y/Tx5b4b\nudf+5/3c6+3XvfX3677d/q3r/+6F/3n7Xv/9/jj33KbJazdgG7f//vsYumv657Hl+CYbj0a3\nRlaDSNHSFLn58u3xPqZ7y5uBH/fr35qN3n7/bC15u++0bv1tmtiAbfz89+PjwWJ9qzre8CYf\n6OGtkdUgUrC0j7HuD79+few0/twfgbVX2UG378PLj3Pdg7znVe3Xwcb3f/99CPK7O54xe5Oj\nWyOr4a4Klpe2wi/34wEvHw+v/k5Ferl9+/X7baNIw43b/c3ba3eEffYmEWlvuKuCpbqb8+HQ\n/ZHXy+2teXT1rfsp9XY0f+ce2nWxAdu4u/Tx7KtavMneQzuyPdxhsfKnfcj1cj8W8Odjn3A/\nJPB6f+r/7/0YwbP1f9ojCfdjCnV77rlNk5+373/r9jnPc2PbhX27dRtObrK5PLo1shpEipXX\n9mDa7/ZY2uM1n8dB7v+erX/tHnjZ4W/bpu6+5cfDsPFzpOaJ2L9P4vgmb/ed4ejWyGoQKVa6\nj6tpzvxq+/728czp+92w9llLc7E5f39B9t/Huec2j7z9eL4g223cO1rxPFQ3c5O/mkeVo1sj\na0GkDHLb+3umf3itVR1E+tJpnkO97v7VvO+q34olzyDSl0771OZtfctebjeOIsiDSF87v77d\nbi/7PKorfmNBH0QiRBBEIkQQRCJEEEQiRBBEIkQQRCJEEEQiRBBEIkSQ/wG9xrxa88UHBwAA\nAABJRU5ErkJggg==",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ggplot(data = hotel, aes(arrival_date_month, adr)) + \n",
    "    geom_boxplot() + ylim(0,600)+ ylab(\"Average daily rate\")+xlab(\"Arrival month\")+\n",
    "    coord_flip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAP1BMVEUAAAAAv8QzMzNNTU1o\naGh8fHyMjIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enr6+vw8PDy8vL4dm3///92l2KZ\nAAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO2di3baSrQtOVEeTnYeN+fy/996AjQg\n5EZYWo/ZLVWNsYOxrRkPuSpg2dk5HAHAzEH9AQBsAUICcICQABwgJAAHCAnAAUICcICQABwg\nJAAHskP6f1WevNoAi+0sJiumgZBYjF5MVkwDIbEYvZismAZCYjF6MVkxDYTEYvRismIaCInF\n6MVkxTQQEovRi8mKaSAkFqMXkxXTQEgsRi8mK6ahjZD+pw3+/9ZwiMA+QUgR1E+1uqCC2nt3\nHCKwTxBSBPVTrS6ooPbeHYcI7BOEFEH9VKsLKqi9d8chAvsEIUVQP9Xqggpq791xiMA+QUgR\n1E+1uqCC2nt3HCKwTxBSBPVTrS6ooPbeHYcI7BOEFEH9VKsLKqi9d8chAvsEIUVQP9Xqggpq\n791xiMA+QUgR1E+1uqCC2nt3HCKwTxBSBPVTrS6ooPbeHYcI7BOEFEH9VKsLKqi9d8chAvsE\nIUVQP9Xqggpq791xiMA+QUgR1E+1uqCC2nt3HCKwTxBSBPVTrS6ooPbeHYcI7BOEFEH9VKsL\nKqi9d8chAvsEIUVQP9Xqggpq791xiMA+QUgR1E+1uqCC2nt3HCKwTxBSBPVTrS6ooPbeHYcI\n7BOEFEH9VKsLKqi9d8chAvsEIUVQP9Xqggpq791xiMA+QUgR1E+1uqCC2nt3HCKwTxBSBPVT\nrS6ooPbeHYcI7BOEFEH9VKsLKqi9d8chAvsEIUVQP9Xqggpq791xiMA+QUiF4R+LNufeu36q\n1QUV1N674xCBfYKQLgy3X2pveHrEE+qnWl1QQe29Ow4R2CcI6czwcFN5y4I3HAkpF4cI7BOE\ndGZURXmONxzPt8Po19Hr7nce3njdqJ9qdUEFtffuOERgnyCkM/evkK7P8Uocx+Hd64b373C7\nfzx+OlH/TdQFFdTeu2NzAxbwoat21weW42Mflde9e4fbbaH+Z5a6oILae3ccHk3sEzwijSnP\n4oZpSJPXTR6JRm8s1E+1uqCC2nt3HCKwTxDSmFEP7x6RjnOPSEdC0uEQgX2CkM7cv0AipO5w\niMA+QUhnhlsjsxcbjk/ega+RlDhEYJ8gpMLtut3jpe7j4+Xv4zic6eXvO/VTrS6ooPbeHYcI\n7BOE9IRlPy80oX6q1QUV1N674xCBfYKQnkBI3eAQgX2CkJ5ASN3gEIF9gpAiqJ9qdUEFtffu\nOERgnyCkCOqnWl1QQe29Ow4R2CcIKYL6qVYXVFB7745DBPYJQoqgfqrVBRXU3rvjEIF9gpAi\nqJ9qdUEFtffuOERgnyCkCOqnWl1QQe29Ow4R2CcIKYL6qVYXVFB7745DBPYJQoqgfqrVBRXU\n3rvjEIF9gpAiqJ9qdUEFtffuOERgnyCkCOqnWl1QQe29Ow4R2CcIKYL6qVYXVFB7745DBPYJ\nQoqgfqrVBRXU3rvjEIF9gpAiqJ9qdUEFtffuOERgnyCkCOqnWl1QQe29Ow4R2CcIKYL6qVYX\nVFB7745DBPYJQoqgfqrVBRXU3rvjEIF9gpAiqJ9qdUEFtffuOERgnyCkCOqnWl1QQe29Ow4R\n2CcIKYK4TxeLrS4mK6aBkFiMXkxWTAMhsRi9mKyYBkJiMXoxWTENhMRi9GKyYhoIicXoxWTF\nNBASi9GLyYppaCMk9TeQRqi/9ZMAIQVASBPUlidASAEQ0gS15QkQUgCENEFteQKEFAAhTVBb\nngAhBUBIE9SWJ0BIARDSBLXlCRBSAIQ0QW15AoQUACFNUFueACEFQEgT1JYnQEgBENIEteUJ\nEFIAhDRBbXkChBQAIU1QW54AIQVASBPUlidASAEQ0gS15QkQUgCENEFteQKEFAAhTVBbngAh\nBUBIE9SWJ0BIARDSBLXlCRBSAIQ0QW15AoQUACFNUFueACEFQEgT1JYnQEgBENIEteUJEFIA\nhDRBbXkChBTAypCGfzx/48yBhKSHkAJYFdIlooeUnr08hZD0EFIA60J6uJl9eQoh6SGkANaE\nNIxfuDzJG//679XD9THrdv96CCHpIaQArCENo9vby8PkbZcDPp2oL6rrGaG2PIEVn3J4hUtI\nx0lItdsCj0h6eEQKIDSkoTzjGx1LSHoIKYDwR6QjIbUGIQVgvWpHSP1BSAFYv4/0oYsNo0MJ\nSQ8hBWD+yYbrJe7Zy993CEkPIQXAz9pNUFueACEFQEgT1JYnQEgBENIEteUJEFIAhDRBbXkC\nhBQAIU1QW54AIQVASBPUlidASAEQ0gS15QkQUgCENEFteQKEFAAhTVBbngAhBUBIE9SWJ0BI\nARDSBLXlCRBSAIQ0QW15AoQUACFNUFueACEFQEgT1JYnQEgBENIEteUJEFIAhDRBbXkChBQA\nIU1QW54AIQVASBPUlidASAEQ0gS15QkQUgCENEFteQKEFAAhTVBbngAhBUBIE9SWJ0BIAbQR\nkseni8VWF5MV00BILEYvJiumgZBYjF5MVkwDIbEYvZismAZCYjF6MVkxDYTEYvRismIa2ghp\n41euAyXtYTFZMQ2ElECgpD0sJiumgZASCJS0h8VkxTQQUgKBkvawmKyYBkJKIFDSHhaTFdNA\nSAkEStrDYrJiGggpgUBJe1hMVkwDISUQKGkPi8mKaSCkBAIl7WExWTENhJRAoKQ9LCYrpoGQ\nEgiUtIfFZMU0EFICgZL2sJismAZCSiBQ0h4WkxXTQEgJBEraw2KyYhoIKYFASXtYTFZMAyEl\nEChpD4vJimkgpAQCJe1hMVkxDYSUQKCkPSwmK6aBkBIIlLSHxWTFNBBSAoGS9rCYrJgGQkog\nUNIeFpMV00BICQRK2sNismIaCCmBQEl7WExWTAMhJRAoaQ+LyYppIKQEAiXtYTFZMQ0fC2l4\nuPnge1epn2pC8qHNxY9Z0zkfDOnSBiGtI1DSHhY/Zk3nfPQRabjcfPC9n1M/1YTkQ5uLH7Om\ncxaHNAynh6fbI9Tl7ukdhsu9+52HN16X6qeakHxoc9HX2Eb58NdIw/3m2tWor0sww/jO8fGN\nx+OnE/X9jYdk+PxAJ6wMqXb33Z2H20L9z6yNhxT4p30Pi16uNs3Hr9oN1ZCG0VO9dwGN31io\nn2pC8qHNRU9fm2XB5e/h4aFnuLY0fodj9ZGIkOIk7WHR6mgXEFICgZL2sGh1tAuWfEP23XO4\nZ18j1b6AKtRPNSH50Oaih6fNs/gnG8ol7Xtcw+2R6R7O9PL3nfqpJiQf2lx00LR9+Fm7BAIl\n7WExWTENhJRAoKQ9LCYrpoGQEgiUtIfFZMU0EFICgZL2sJismAZCSiBQ0h4WkxXTQEgJBEra\nw2KyYhoIKYFASXtYTFZMAyElEChpD4vJimkgpAQCJe1hMVkxDYSUQKCkPSwmK6aBkBIIlLSH\nxWTFNBBSAoGS9rCYrJgGQkogUNIeFpMV00BICQRK2sNismIaCCmBQEl7WExWTAMhJRAoaQ+L\nyYppIKQEAiXtYTFZMQ2ElECgpD0sJiumgZASCJS0h8VkxTQQUgKBkvawmKyYBkJKIFDSHhaT\nFdNASAkEStrDYrJiGtoIqU0BWCSkD0NILEYvJiumgZBYjF5MVkwDIbEYvZismAZCYjF6MVkx\nDYTEYvRismIa2gjJ5/J3BOor51kQkhFCmkcteBaEZISQ5lELngUhGSGkedSCZ0FIRghpHrXg\nWRCSEUKaRy14FoRkhJDmUQueBSEZIaR51IJnQUhGCGketeBZEJIRQppHLXgWhGSEkOZRC54F\nIRkhpHnUgmdBSEYIaR614FkQkhFCmkcteBaEZISQ5lELngUhGSGkedSCZ0FIRghpHrXgWRCS\nEUKaRy14FoRkhJDmUQueBSEZIaR51IJnQUhGCGketeBZEJIRQppHLXgWhGSEkOZRC54FIRkh\npHnUgmdBSEYIaR614FkQkhHnkIbJ7TsIqU0IyYh3SMP4pgIhtQkhGSGkedSCZ0FIRtyf2g33\nX4dTT6Ws6zsQUpsQkpG4kIb7/cudTyfqh6lzeY5a8Cx8NdghgY9Il/vlFVd4RGoTHpGM+F+1\nGz0c3Z7bEVLrEJKRwJDu/40uPRBSmxCSkbiQytdIhNQFhGQk4Buy5b/LhbsjIXUBIRmJC+n0\nJdL9YekKIbUJIRnJ+Fk7QmofQjJCSPOoBc+CkIzEh/T440KE1CaEZIS/RjGPWvAsCMkIIc2j\nFjwLQjJCSPOoBc+CkIwQ0jxqwbMgJCOENI9a8CwIyQghzaMWPAtCMkJI86gFz4KQjBDSPGrB\nsyAkI4Q0j1rwLAjJCCHNoxY8C0IyQkjzqAXPgpCMENI8asGzICQjhDSPWvAsCMkIIc2jFjwL\nQjJCSPOoBc+CkIwQ0jxqwbMgJCOENI9a8CwIyQghzaMWPAtCMtJGSB6fLhZbXUxWTAMhsRi9\nmKyYBkJiMXoxWTENhMRi9GKyYhoIicXoxWTFNBASi9GLyYppICQWoxeTFdPQRkh8V8iIo/aT\nCDw29gAhbQJH7ScReGzsAULaBI7aTyLw2NgDhLQJHLWfROCxsQcIaRM4aj+JwGNjDxDSJnDU\nfhKBx8YeIKRN4Kj9JAKPjT1ASJvAUftJBB4be4CQNoGj9pMIPDb2ACFtAkftJxF4bOwBQtoE\njtpPIvDY2AOEtAkctZ9E4LGxBwhpEzhqP4nAY2MPENImcNR+EoHHxh4gpE3gqP0kAo+NPUBI\nm8BR+0kEHht7gJA2gaP2kwg8NvYAIW0CR+0nEXhs7AFC2gSO2k8i8NjYA4S0CRy1n0TgsbEH\nCGkTOGo/icBjYw8Q0iZw1H4SgcfGHiCkTeCo/SQCj409QEibwFH7SQQeG3vAHNIwuX329kL9\nVBOSEUftJxF4bOwBe0jD+Kby9se79VNNSEYctZ9E4LGxBwhpEzhqP4nAY2MPODy1G+6/DkO5\nMzzcGcVUP9WEZMRR+0kEHht7wDOk4XL/WAKa3vl0oj5CSEasn0Ww4vqIdLl/a+h4u70/JNX/\nzCIkI46PH5NHE4+NPeBx1W70cHR+GkdI6ThqP4nAY2MPuIY0HHlE0uCo/SQCj4094BnS7cui\ncnu83RJSMI7aTyLw2NgDLt+QHT0SPT61m1ZFSEE4aj+JwGNjD3iGdL7QfW3nfvn7SEjhOGo/\nicBjYw/ws3abwFH7SQQeG3uAkDaBo/aTCDw29gAhbQJH7ScReGzsAULaBI7aTyLw2NgDhLQJ\nHLWfROCxsQcIaRM4aj+JwGNjDxDSJnDUfhKBx8YeIKRN4Kj9JAKPjT1ASJvAUftJBB4be4CQ\nNoGj9pMIPDb2ACFtAkftJxF4bOwBQtoEjtpPIvDY2AOEtAkctZ9E4LGxBwhpEzhqP4nAY2MP\nENImcNR+EoHHxh4gpE3gqP0kAo+NPUBIm8BR+0kEHht7gJA2gaP2kwg8NvYAIW0CR+0nEXhs\n7AFC2gSO2k8i8NjYA4S0CRy1n0TgsbEHCGkTOGo/icBjYw+0EVKbArBISB+GkFiMXkxWTAMh\nsRi9mKyYBkJiMXoxWTENhMRi9GKyYhoIicXoxWTFNLQRUtbl70WoL2n7syYCU0JlYw8Q0lPU\n2vuzJgJTQmVjDxDSU9Ta+7MmAlNCZWMPENJT1Nr7syYCU0JlYw8Q0lPU2vuzJgJTQmVjDxDS\nU9Ta+7MmAlNCZWMPENJT1Nr7syYCU0JlYw8Q0lPU2vuzJgJTQmVjDxDSU9Ta+7MmAlNCZWMP\nTEN6Gw4Xgn6/+qlWN1NFrb0/ayIwJVQ29sAkmLfDgZAKau39WROBKaGysQcmwQyH77G/X/1U\nq5upotbenzURmBIqG3tgElLYI9GV+qlWN1NFrb0/ayIwJVQ29sC7p3Z/Y3+/+qlWN1NFrb0/\nayIwJVQ29sD0Eejrlz+hv1/9VKubqaLW3p81EZgSKht7YBzSYUzQ71c/1epmqqi192dNBKaE\nysYeIKSnqLX3Z00EpoTKxh7gG7JPUWvvz5oITAmVjT1ASE9Ra+/PmghMCZWNPfDk8vcwBP1+\n9VOtbqaKWnt/1kRgSqhs7IFxSANfI41Ra+/PmghMCZWNPTAO5seoox9Bv1/9VKubqaLW3p81\nEZgSKht7gJ9seIpae3/WRGBKqGys4M+Xw+Fz5fU/nnzN8fObl7u1mX+v+/bzxWEuv/nHqZ9q\ndTNV1Nr7syYCU0JlYwXDky8vntTya3B7EHgS0nH4NX/Y5D5/jeKGWnt/1kRgSqhsrOCZgE9e\nP/h9KfIspGePhdd3ebxr/GsUr6/11U+1upkqau39WROBKaGyMcevr4fD8HZ66ftw+HztoSh4\nf+Ofr+cXipl/vh0O3/6c3+/38OX0x/+xFDAeKZRDR2uHw+11txf+nib/Xmdu907PMb9eXnd5\nv2e8+2sUprQJqWnWRGBKqGzM8PPyx/bb9c/wot8lpPsb/56fKH0tIV3uDX9P7/fl8O3493DJ\nYzJy4XroaO3fsZPd8lzyc5m53zu/w9fz6+Z/oNv3YgMhNc2aCEwJlY0ZPh/+Ox5/n7T79zBx\n/HW4GnQW8f7Gt3+5/Lq81/F079+j0JdLEqeEvh9+lkMeR87cDh3/Vl/+Hn+c3u32xu+Xln9c\nZu73Tr/V3y/n3/Xn7N/Vm4Tz1fbXKIbS0vl2uHxXdxjG392tn2p1M1XU2vuzJgJTQmVjlj8/\nv59FHQ7jC2PlT/TrGz9fvSyB/Xta9+f0iHEK52Ttn/Kmx5Ezn+9KX9cOt/e/vfHz+fc7PTZd\nXn29V36r8/PJ8yPXMyYh/RlMf41iHNLweP94/HSifpy6mSpq7f0xfGbD+HL9kvznvydRn2/y\nXUK6vfH2TOn8wuXe/fX3Xx9HRkuVtcnu+I2P9x5/fcK7p3bWiw3Dk9tC/c8sdTNV1Nr7s+bR\nxPRYVDZm+Hb4/OPn5U/84+/Ph9s15vNr7m/8aEiPI/cjjpW13kIaHp7b1U+1upkqau39WROB\nKaGyMeff+RrZVbcflWD+zj+1u73y+H7kzOOhfx/rmDy1u7/6fvTtqd2ikIw8e0S6Uz/V6maq\nqLX3Z00EpoTKxpx/h1/li/nh30u/Hy823N/49u+r/9/Viw2ndx5/jTQeOTM69Lp27+L2xtML\nx/9Ou4/3vp+uS3xZ/jXSep49pSOkdlgTgSmhsjHD2+0J0OWl04WxUTDXN/65Xo8+nCoZX/4+\njdyv2t1HbtwOva/dQ7q98TJ5+H159f3e6PL3oqt2x79vn/8Nvy2/djcOZ+BrpCZZE4EpobIx\nx7fD4cuvSzbDYTirenf9/sbfXy7fgj1ftB5/Q/Ys7e37SPeRO9dD72ujZ2q3N/45v/X66tu9\n03dsv16zXvB9pD/lJ4SG5dfuhvvl7oeAuPzdDGsiMCVUNsJ58/0Kpc5hyU82fDucLn//OX27\nOIb6qVY3U0WtvT9rIjAlVDbicfxZu2cs+1m764UJ4084PKd+qtXNVFFr78+aCEwJlY14fk0s\nN159rrHsp78J6Y5ae3/WRGBKqGwk8PPxKZR/SAv/PhJP7e6otfdnTQSmhMrGHvC72PAx6qda\n3UwVtfb+rInAlFDZ2ANul78/SP1Uq5upotbenzURmBIqG3sg47rhmPqpVjdTRa29P2siMCVU\nNvYAIT1Frb0/ayIwJVQ29gAhPUWtvT9rIjAlVDb2wCSk099Vd77+/kj9VKubqaLW3p81EZgS\nKht7YBLMV/9vZD1SP9XqZqqotfdnTQSmhMrGHnj3Ddn/Yn+/+qlWN1NFrb0/ayIwJVQ29sAk\npM/RXzPVT7W6mSpq7f1ZE4EpobKxB6bfkA38FtKZ+qlWN1NFrb0/ayIwJVQ29sD0Eeg/vka6\notbenzURmBIqG8/53zmCFIyBiw1PUWvvz5oITAmVjef878xH23VIXGy4k+Z3GmsiMCVUNp6z\n2ZC+crHhRprfaayJwJRQ2XjOZkM6fv0W9XPfF+qnWt1MlTS/01gTgSmhsvGczYYU8DcLH6mf\nanUzVdL8TmNNBKaEysZzCGkt9VOtbqZKmt9prInAlFDZeM5mQwqnfqrVzVRJ8zuNNRGYEiob\nzyGktcR9ulhsdXFGB0JaS9yni8VWF2d0IKS1xH26WGx1cUaHj4YU+FW7E4TEYvTijA4fDOlw\n+6VZCInF6MUZHQhpLXGfLhZbXZzRYZshHcYE/X71U62+0j3HzKd6LxDSawjpFWm6tgshvaaN\np3bqWOZI07VdCOk1hPSKNF3bhZBeM/3o3nhqNyFN13YhpNdMPro3vkaakqZruxDSayYf3XD4\n/eXw5++Xw/y/qrQeQuoQeUj9/WTDvw/39E9E/z39y+ghEFKH6ENqn/ch/Tz84F/sG5Gma7sQ\n0mve/T8b/vtz+Hz8RUg30nRtF0J6zSSYU0FfTk9I+acvr6Tp2i6E9JrpI8/Pz6d/SPbwFvX7\nEVKHENJrJiH9jv79CKlDCOk104sNn1/8K+hWCKlDAkPa6v+y+PPhMHyP/N/oE1KHENJrpl8j\n/XkbDoevUd+OJaQuIaTXVC5z/3o7HD5H/S/ACalDAkOaOfHdh/TvYYmftbujtrgBCOk1tUek\nb/8ekX4E/X6E1CGE9Jrq10jf+BrpjtriBiCk17y/avf5B1ftxqgtbgBCes30+0hf+T7SBLXF\nDUBIr5mEFPsvMR8JqUsI6TWP/xch/lmX96gtbgBCeg0hvUJtcQMQ0msighmqL14gpA5pIaTG\n/6b59OP74vH3kAhpYzQQUuv/y4b3//MTh01C2hj6kA69PSL9/vJm/1fNhxLQ+Xa4vuoCIXWI\nPqTuntq5XGx4EtKnE/Uj0qpYgdriBjC48AJCmuMhpPOL92d4PCJ1CI9Irwm6akdIW4KQXhN0\n1W4U0u3p3RlC6hBCeo3zVbtxQIS0FQjpNc5X7R4CIqSNQEiv8b7YMAzDUG7Hj0w3CKlDCOk1\nEVftphBS57QQUutkhE5InUNIr4kPaXj4MSFC6hBCek09pF9v735IzglC6hBCek0lpJ/fhsOB\nkK6oLW4AQnrNu3+N4tv5H3UJ+z83EFKHENJrHkK6VHQ4BP6fGwipQwJD2uT/srg8FoX+HSpC\n6pC4kLbDY0hf/x6D/zIiIXUIIb2GR6RXqC1uAEJ6DV8jvUJtcQMQ0mueXLUL+59/E1KHENJr\n+D7SK9QWNwAhvYafbHiF2uIGIKTXZP90OiF1CCG9hpBeoba4AQjpNYT0CrXFDUBIryGkV6gt\nbgBCeg0hvUJtcQMQ0msI6RVqixuAkF5DSK9QW9wAhPSaNkJ68mrLJ4/FZhaTFdNASCxGLyYr\npoGQWIxeTFZMAyGxGL2YrJgGQmIxejFZMQ2ExGL0YrJiGgiJxejFZMU0tBHSBr7Ps0Ywk579\nLCYrpoGQnFgjmEnPfhaTFdNASE6sEcykZz+LyYppICQn1ghm0rOfxWTFNBCSE2sEM+nZz2Ky\nYhoIyYk1gpn07GcxWTENhOTEGsFMevazmKyYBkJyYo1gJj37WUxWTAMhObFGMJOe/SwmK6aB\nkJxYI5hJz34WkxXTQEhOrBHMpGc/i8mKaSAkJ9YIZtKzn8VkxTQQkhNrBDPp2c9ismIaCMmJ\nNYKZ9OxnMVkxDYTkxBrBTHr2s5ismAZCcmKNYCY9+1lMVkwDITmxRjCTnv0sJiumgZCcWCOY\nSc9+FpMV00BITqwRzKRnP4vJimkgJCfWCGbSs5/FZMU0EJITawQz6dnPYrJiGgjJiTWCmfTs\nZzFZMQ2E5MQawUx69rOYrJgGQnJijWAmPftZTFZMw/qQhlVH1U81IfnQ5uJqxXqCkJxYI5hJ\nz34WVyvWE4TkxBrBTHr2s7hasZ6whTT84/zi9bb++lF19VNNSD60uWjQsx9MIV3DuVY0VF5/\nD+nTifrWBkJafR5hE9if2lUCmr5+9DSw/mfWBkJa8ye16c/5fhZXK9YTxpAuz+Heh/T4ekKq\nC2bSs5/F1Yr1hPFrpGP9EWn8+tNXSvej6qeakHxoc3G1Yj2xKqR3XwvN3RLSU8FMevazaNCz\nH8whTZ/aDdPXE9JTwUx69rNoM7QT1j21K5e3z1e3H57SXe4/vP7xO071U01IPrS5aPGzGzJ+\n1o6Qnghm0rOfxQTF9BCSE2sEM+nZz2KCYnriQxoefpaofqoJyYc2F8MVawH+GoUTawQz6dnP\nYrJiGgjJiTWCmfTsZzFZMQ2E5MQawUx69rOYrJgGQnJijWAmPftZTFZMAyE5sUYwk579LCYr\npoGQnFgjmEnPfhaTFdNASE6sEcykZz+LyYppICQn1ghm0rOfxWTFNBCSE2sEM+nZz2KyYhoI\nyYk1gpn07GcxWTENhOTEGsFMevazmKyYBkJyYo1gJj37WUxWTAMhObFGMJOe/SwmK6aBkJxY\nI5hJz34WkxXTQEhOrBHMpGc/i8mKaSAkJ9YIZtKzn8VkxTQQkhNrBDPp2c9ismIaCMmJNYKZ\n9OxnMVkxDYTkxBrBTHr2s5ismIY2QmpTABYJ6cMQEovRi8mKaSAkFqMXkxXTQEgsRi8mK6aB\nkFiMXkxWTAMhsRi9mKyYhjZCslz+7uHa+IexKms8PmYxWTENhNQUVmWNx8csJiumgZCawqqs\n8fiYxWTFNBBSU1iVNR4fs5ismAZCagqrssbjYxaTFdNASE1hVdZ4fMxismIaCKkprMoaj49Z\nTFZMAyE1hVVZ4/Exi8mKaSCkprAqazw+ZjFZMQ2E1BRWZY3HxywmK6aBkJrCqqzx+JjFZMU0\nEFJTWJU1Hh+zmKyYBkJqCquyxuNjFpMV00BITWFV1nh8zGKyYhoIqSmsyhqPj1lMVkwDITWF\nVVnj8TGLyYppIKSmsCprPD5mMVkxDYTUFFZljcfHLCYrpoGQmsKqrPH4mMVkxTQQUlNYlTUe\nH7OYrJgGQmoKq7LG42MWkxXTQEhNYVXWeHzMYrJiGgipKazKGo+PWUxWTAMhNYVVWePxMYvJ\nimkgpKawKms8PmYxWTENC0MaLoxesfD3q59qQipYlTUeH7O4UJE+WRrSy1e8oH6qCalgVdZ4\nfMziQkX6hJCawqqs8QDgeusAAA0mSURBVPiYxYWK9MnqkMozvOF4v728dXj/xtFx9VNNSAWr\nssbjYxatjnbB2pCuzZRgbreXO6M33kP6dKI+S0iFpZ8+aIVVFxuOlWaOx/Evk1feH8jqf2YR\nUsH6Z7/x+JhFm6GdsPoRqRT1JKTHNxLSR7Eqazw+ZtFmaCeYntodn4U0jF55HMZXJOqnmpAK\nVmWNx8csrnWzK0JCerglpAVYlTUeH7O41s2ucLvYcH3l9SndQEgrsCprPD5m0WZoJ/hd/i6v\nvF7+vj8iPXyvqX6qCalgVdZ4fMyixc9uyPhZO0L6MFZljcfHLCYopoeQmsKqrPH4mMUExfTE\nhzT+EVdCeoFVWePxMYvhirUAf42iKazKGo+PWUxWTAMhNYVVWePxMYvJimkgpKawKms8PmYx\nWTENhNQUVmWNx8csJiumgZCawqqs8fiYxWTFNBBSU1iVNR4fs5ismAZCagqrssbjYxaTFdNA\nSE1hVdZ4fMxismIaCKkprMoaj49ZTFZMAyE1hVVZ4/Exi8mKaSCkprAqazw+ZjFZMQ2E1BRW\nZY3HxywmK6aBkJrCqqzx+JjFZMU0EFJTWJU1Hh+zmKyYBkJqCquyxuNjFpMV00BITWFV1nh8\nzGKyYhoIqSmsyhqPj1lMVkwDITWFVVnj8TGLyYppaCOkNgVgkZA+DCGxGL2YrJgGQmIxejFZ\nMQ2ExGL0YrJiGgiJxejFZMU0EBKL0YvJimkgJBajF5MV09BGSNv+5lGkpD0sJiumgZDiiZS0\nh8VkxTQQUjyRkvawmKyYBkKKJ1LSHhaTFdNASPFEStrDYrJiGggpnkhJe1hMVkwDIcUTKWkP\ni8mKaSCkeCIl7WExWTENhBRPpKQ9LCYrpoGQ4omUtIfFZMU0EFI8kZL2sJismAZCiidS0h4W\nkxXTQEjxREraw2KyYhoIKZ5ISXtYTFZMAyHFEylpD4vJimkgpHgiJe1hMVkxDYQUT6SkPSwm\nK6aBkOKJlLSHxWTFNBBSPJGS9rCYrJgGQoonUtIeFpMV00BI8URK2sNismIaCCmeSEl7WExW\nTAMhxRMpaQ+LyYppIKR4IiXtYTFZMQ0fCWm48OHNufesn2pC8qHNxQ+L0zMfCunh5sPvX6V+\nqgnJhzYXP+pN1ywI6cMlEdIjkZL2sPhBbfpmaUjX53iT2+F4vn248/DG61b9VBOSD20uevra\nLAtDGp7dXiIb35m88Xj8dKL+O2w7pPWfHegGS0jH90E9r61Q/zNr2yFF/mnfw6Ld0g5YGlK5\nfnd71fj++4epx3c+UT/VhORDm4terjbNmkek45PbZ49EhBQnaQ+Lq+XsCUKKJ1LSHhZXy9kT\nC7+PNPtl0OOVB75GuhIpaQ+LPqY2ztKfbHh6+fs4DqfyxkL9VBOSD20uusnaMvysXTyRkvaw\nmKyYBkKKJ1LSHhaTFdNASPFEStrDYrJiGggpnkhJe1hMVkwDIcUTKWkPi8mKaSCkeCIl7WEx\nWTENhBRPpKQ9LCYrpoGQ4omUtIfFZMU0EFI8kZL2sJismAZCiidS0h4WkxXTQEjxREraw2Ky\nYhoIKZ5ISXtYTFZMAyHFEylpD4vJimkgpHgiJe1hMVkxDYQUT6SkPSwmK6aBkOKJlLSHxWTF\nNBBSPJGS9rCYrJgGQoonUtIeFpMV00BI8URK2sNismIaCCmeSEl7WExWTAMhxRMpaQ+LyYpp\nIKR4IiXtYTFZMQ1thNSmACwS0ochJBajF5MV00BILEYvJiumgZBYjF5MVkwDIbEYvZismAZC\nYjF6MVkxDW2E5Hv5Oxb1xfSGIaQ8CGnDEFIehLRhCCkPQtowhJQHIW0YQsqDkDYMIeVBSBuG\nkPIgpA1DSHkQ0oYhpDwIacMQUh6EtGEIKQ9C2jCElAchbRhCyoOQNgwh5UFIG4aQ8iCkDUNI\neRDShiGkPAhpwxBSHoS0YQgpD0LaMISUByFtGELKg5A2DCHlQUgbhpDyIKQNQ0h5ENKGIaTV\nDEsPIKQNQ0hrGT5W0uidCGnDENJaCAlGENJaSki3X4bh/Jrx/eH66wVC2jCEtJJh9N81oGF4\nvH+7PX46UR9Sx7EEta0NY3Gpd9xDeh8QT+32Ao9I6xiuvz4L6TgMo3c8Q0gbhpDWMVyYCamk\nREj7gJDW8foRaXz/AiFtGEJaxb2U8UWG6S0h7QdCWsX4IWe41vTu9nq3QEgbhpA8mf8WLSFt\nGEJy4uFrozqEtGEIyYvxzzDUIaQNQ0h5ENKGIaQ8CGnDEFIehLRhCCkPQtowhJQHIW0YQsqD\nkDYMIeVBSBuGkPIgpA1DSHkQ0oYhpDwIacMQUh6EtGEIKQ9C2jCElAchbRhCyoOQNgwh5UFI\nG4aQ8iCkDUNIeRDShiGkPAhpwxBSHoS0YQgpjyen+smnYD0strOYrJgGQmIxejFZMQ2ExGL0\nYrJiGgiJxejFZMU0EBKL0YvJimkgJBajF5MV09BGSOpL2lXUF5ODWBiBvSNCiqB+qtXNVFEb\nH8TCCOwdEVIE9VOtbqaK2vggFkZg74iQIqifanUzVdTGB7EwAntHhBRB/VSrm6miNj6IhRHY\nOyKkCOqnWt1MFbXxQSyMwN4RIUVQP9XqZqqojQ9iYQT2jggpgvqpVjdTRW18EAsjsHdESBHU\nT7W6mSpq44NYGIG9I0KKoH6q1c1UURsfxMII7B0RUgT1U61upora+CAWRmDviJAiqJ9qdTNV\n1MYHsTACe0eEFEH9VKubqaI2PoiFEdg7IqQI6qda3UwVtfFBLIzA3hEhRVA/1epmqqiND2Jh\nBPaOCCmC+qlWN1NFbXwQCyOwd0RIEdRPtbqZKmrjg1gYgb0jQoqgfqrVzVRRGx/EwgjsHRFS\nBPVTrW6mitr4IBZGYO+IkCKon2p1M1XUxgexMAJ7R4QUQf1Uq5upojY+iIUR2DsipAjqp1rd\nTBW18UEsjMDeESFFUD/V6maqqI0PYmEE9o4IKYL6qVY3U0VtfBALI7B3REgR1E+1upkqauOD\nWBiBvSNCWsNQfdX9tfVTrW6mitr4IBZGYO+IkFYw1EoipJZYGIG9I0JaASE1z8II7B0R0gpK\nSP9uhqHc8tSuLRZGYO+IkJYzXP8rL9xuT2/8dKJ+nLqZKmrjg3D9hMOVoJDKnYfbM/U/s9TN\nVFEbH8TCRxP7AxKPSIsZrr8SUrssjMDeESEtZrhASC2zMAJ7R4S0GB6ROmBhBPaOCGkpw+32\n4SIDITXFwgjsHRHSUkYh3S5/HwmpMRZGYO+IkNZT/a7smfqpVjdTRW18EAsjsHdESOshpHZZ\nGIG9I0JaDyG1y8II7B0RUgT1U61upora+CAWRmDviJAiqJ9qdTNV1MYHsTACe0eEFEH9VKub\nqaI2PoiFEdg7IqQI6qda3UwVtfFBLIzA3hEhRVA/1epmqqiND2JhBPaOCCmC+qlWN1NFbXwQ\nCyOwd0RIEdRPtbqZKmrjg1gYgb0jQoqgfqrVzVRRGx/EwgjsHRFSBPVTrW6mitr4IBZGYO+I\nkCKon2p1M1XUxgexMAJ7R4QUQf1Uq5upojY+iIUR2DsipAjqp1rdTBW18UEsjMDeESFFUD/V\n6maqqI0PYmEE9o4IKYL6qVY3U0VtfBALI7B3REgR1E+1upkqauODWBiBvSNCiqB+qtXNVFEb\nH8TCCOwdEVIE9VOtbqaK2vggFkZg74iQIqifanUzVdTGB7EwAntHhBRB3KeLxVYXkxXTQEgs\nRi8mK6aBkFiMXkxWTAMhsRi9mKyYBkJiMXoxWTENhMRi9GKyYhoIicXoxWTFNBASi9GLyYpp\nICQWoxeTFdNASCxGLyYrpiE7pDpP/rHzpuBjhBkI6aPwMcIMhPRR+BhhBkL6KHyMMEMbIQF0\nDiEBOEBIAA4QEoADhATgACEBONBCSMM/1B9DjaF8YNePb3qr5/JhPPv42vk4d0ADIQ23Xxpj\nGN0M72/1DPcPpfLxtfNx7gFCek7rIQ1HQmoGQnrKML5tU1BCagZCesrtS6TjsVVBCakZCOkp\nHQhKSM1ASPO0LSghNQMhzdO2oITUDIT0lA4EJaRmIKSnDKP/GhWUkJqhgZCa/Q58+z8xcPkw\n2v84d0ALIQF0DyEBOEBIAA4QEoADhATgACEBOEBIAA4QEoADhATgACHp+MHPHWwHQtJx4ORv\nBz6XOghpQ/C5jODP18Pwdn7h2+Hw7c/xGs3p18OhvPVwoKTtwKcygL/DKZKv1xeGv48hnV/5\nRkibgk9lAG+Hb8dfp0reDl+Oxy+Ht8eQvvw9/jgMPLXbFHwuA/h8+Ht94d/Tuj+Hz5OndreX\nhB8j+MLnMoBbIeN8nr0Em4DPZQCEtD/4XAbw7KndH0LaLHwuA3g7vB1/P15sGA7/Hf9+IaTN\nwucygD/nC9yfx5e/304vfJ+GxM8IbQZCiuD3l/J92Ns3ZI9vw+H75GukH4S0HQgJwAFCAnCA\nkAAcICQABwgJwAFCAnCAkAAcICQABwgJwAFCAnCAkAAcICQAB/4Pel0dsuPG/HgAAAAASUVO\nRK5CYII=",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ggplot(data = hotel, aes(arrival_date_month, ..count..)) + \n",
    "    geom_bar(mapping = aes(fill=as.factor(is_canceled))) +xlab(\"Arrival month\")+\n",
    "    coord_flip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[90m# A tibble: 1,000 x 7\u001b[39m\n",
      "   is_canceled   adr lead_time total_of_specia~ stays_in_week_n~\n",
      "   \u001b[3m\u001b[90m<lgl>\u001b[39m\u001b[23m       \u001b[3m\u001b[90m<dbl>\u001b[39m\u001b[23m     \u001b[3m\u001b[90m<dbl>\u001b[39m\u001b[23m            \u001b[3m\u001b[90m<dbl>\u001b[39m\u001b[23m            \u001b[3m\u001b[90m<dbl>\u001b[39m\u001b[23m\n",
      "\u001b[90m 1\u001b[39m FALSE          0        342                0                0\n",
      "\u001b[90m 2\u001b[39m FALSE          0        737                0                0\n",
      "\u001b[90m 3\u001b[39m FALSE         75          7                0                1\n",
      "\u001b[90m 4\u001b[39m FALSE         75         13                0                1\n",
      "\u001b[90m 5\u001b[39m FALSE         98         14                1                2\n",
      "\u001b[90m 6\u001b[39m FALSE         98         14                1                2\n",
      "\u001b[90m 7\u001b[39m FALSE        107          0                0                2\n",
      "\u001b[90m 8\u001b[39m FALSE        103          9                1                2\n",
      "\u001b[90m 9\u001b[39m TRUE          82         85                1                3\n",
      "\u001b[90m10\u001b[39m TRUE         106.        75                0                3\n",
      "\u001b[90m# ... with 990 more rows, and 2 more variables: stays_in_weekend_nights \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m,\u001b[39m\n",
      "\u001b[90m#   previous_cancellations \u001b[3m\u001b[90m<dbl>\u001b[90m\u001b[23m\u001b[39m\n"
     ]
    }
   ],
   "source": [
    "# Consider a smaller set:\n",
    "hotel<- hotel %>% select(is_canceled, adr, lead_time, total_of_special_requests, stays_in_week_nights, stays_in_weekend_nights, previous_cancellations)\n",
    "hotel<- hotel %>% mutate(is_canceled = (is_canceled==1));\n",
    "hotel<-hotel %>% filter(adr<1000)\n",
    "hotel<-hotel[1:1000,]\n",
    "hotel %>% print\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2 Criteria\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2.1 Continuous outcomes\n",
    "\n",
    "With continuous outcome, we often use the squared error, or the $\\ell_2$-norm,  \n",
    "$$\n",
    "L\\left( y, \\hat{y}\\right) = \\|y-\\hat{y}\\|_2^2\\equiv  \\left( y - \\hat{y}\\right)^2.\n",
    "$$\n",
    "\n",
    "A famous example that uses this loss function is the least squares estimation for linear regression. In a linear regression, the function $f(X,\\beta)$ takes the form \n",
    "$$\n",
    "f(X,\\beta)= \\beta_1 x_1 + \\ldots + \\beta_p x_p.  \n",
    "$$\n",
    "The linear model is usually the place to start for building a predictive model. However, it is rare to have a truly linear relationship between the outcomes and features. The linear form is also very restrictive in modeling. Therefore, the linear model is almost always outperformed by more flexible models. Nevertheless, it is a simply model for us to start exploring predictive modeling. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAM1BMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD/AAD///89ODILAAAACXBIWXMAABJ0\nAAASdAHeZh94AAAgAElEQVR4nO2di3aqOhBA4+NYa20v//+1t2JVHkkIMJlMcO9117kWCDOh\nsw0EtK4BgNW40gkAbAFEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQA\nARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAA\nkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJ\nQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAE\nQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABE\nAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQA\nARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQABEAhAAkQAEQCQAARAJQAAFkRxA\nZSyocnlxCoQAkASRAARAJAABEAlAAEQCEACRAARAJAABEAlAAEQCEACRAARAJAABEAlAAEQC\nEEBVpK+PY/ug7PH0lSsEQBEURfrZdx46P2QJAVAIRZFObvd5bV99X3bulCMEQCEURdq56/P1\n1e1yhAAohKJIvQ8Rxj9RiEhQGYxIYJBFH90uiu410uW7fcU1EsRY/C0IBdGc/j50Zu32P1lC\nwBZApAm+Tu19pN3xg/tIEGTFF/OUgycbwBqIJEZVhxCEQaQkznvnjpesIaBuKvRI/z7S34xD\ndNIOkd4bRIq3uzU8udNP03yf3DlHCNgItWmkL9LOtfPeP26fIwRAIbRFerzTjN9xVn4jOUBJ\ntEX69xCJR4RgS6iKdPw4X9zn78ufE48IQQL1nJyoivQ8bXNuxyNCMEVN5/ma95Gu1/P5eGyn\nHE5RjxAJbiDSWuo4dpCXqqaeEAmsgkirqeLQQWYQaTVVHDrITUUeIRLYBZH87frkCAEboxaN\nVEU6IxJEqUebMar3kXbx71cVCAH1UtOJ3BjVa6TrxMeQBEJAtSBSOufOV9tlCgGVUtVk9xhm\n7SCGXmEjUgYqPZibQ7O0ESkDlR7MzaFa2lV7hEgQRneQQCR5aj2aG0P7bKtejRAJIlR+2aIK\nIkEYPEoGkSAMIiWDSBADjRJBJAABEAlAAEQCEACRAARAJNgIZedFEAk2QemZekSCTYBIhULA\npij+NBMiwRZApFIhYFMgUqkQsC3SvuctY3yVJgZDwLZwrqxKiAQbAZGKhIDtUdIkRILNgEgF\nQsD2QKQCIWCDcI2kHwI2CCLph4BNwn0k7RCwGUx8rwQiQd2Ufuz7kYZKE4MhYCMgUtkQsA2K\nP636yEOlicEQYIZVFiBS4RBghJUeIFLhEGCEoAeJctjwCJGgLKERJdkPRCobAmywWiTuI5UN\nATYIiGTl0icZRIKyRAckRFpHNYcPVoNIGanm8IEAoTm7mjxCJLAJIklQz/GDbNSkESIBiIBI\nYJtKBiZEAstUc6mESGAZRFpFBQcONKjndhIigWEQaR32jxuogEjrsH/cQIeBR3aVQiSwTE8k\ny6MTIoFt+uMRIlkLAdVh+oIJkaAWEMliCKgORLIYAmzj88WwR4gEFol+bLZIRlMgEhgkpIxV\njRAJLGL6asgPIoE9EEmIig4gZACRhKjoAEIOqvMIkcAiiCRDTUcQslCXRogERahNk2kQCdSp\n78RtGkQCdRaJZNw8RAJtlkxumx/EVEX6+ji2h+N4+soVAuyDSIubtPzs3YtDlhBQAwtEsn+H\nVlGkk9t9XttX35edO+UIAVWweEBCpBs7d32+vrpdjhBQBQoiqTunKFKva/F+ItLGWTJnN8Oj\nAsMXIxLUACK9+L1Guny3r7hGgtnMO69TN0lz+vvQmbXb/2QJAdvDJ0Rckq2L1Hyd2vtIu+MH\n95EgDZ8Sj2UhVTYvkqUQUAcRkcK2IJJeCKgCny5uSFKr3ImqNPmDR4RgJog0hkeEYDZJIo3K\nZdsi8YgQzGf6GulvbW+jTV8jcUMW5pMo0mCzTYs08YhQ/KwX3pbARVDTq5ZR3WgXESMSVMto\nQCr4DswjQlAZvoHnrUTiESFYj1+Y9xKJR4RgNQFhSnvEkw1QFaGhB5FKhYBJchfmkv2Hz+EK\nz/UiEvjJ/R4/e/+R6TkDN0wQCfwUFsnjiuvcPZqxJx0QCbzkngeL73+8DpFe7frkCAFiGBOp\ns3lgrCpskqJIZ0SqiFzl+dhjdP/jlcHNEysqO5qndtdd/MMTAiFAjCzV2dlpwoA0LVLyW3N2\nVK+RrvEHgyRCgBS2RJp6oqG0R8qTDefOc6uZQoAYWc7rOkUf3r/HDeMDErN2oEhi0fc2e27t\naWZGI0QCTeaLFN/ejEaIBKqk1X2ySDZuIbUgEijiK/zwKVvoJC++vzIgEqji0Sg4iZAyNtnQ\nCJGgMHE/0k/yEuKsSXN6/ypNDIYAE0xNF/hP8paGWZ7ndASVJgZDgAkQKSuI9C5M+zE+ycsR\nZTWIBEVJrnCBAQmRYLMgUk4Q6Y1Iru8VKnCNBFtnWN8JEw+LgiASbJhhhccqfpUNeTVCJCjL\nDJEULnRWgEhQkKEcMVkQyWQIsEBYpLEwiGQyBFggKJJPJ89poG62MRAJShK4Rurr1PS+ZTXQ\nsjCIBCVJEum5kWfLqb1ny3wYSqWJwRBghGGxd8wZj0y9zaZMUh2zEAnM4ffI+z3FiFQ6BNhF\nRqQE1QRBJDBHikjT4w0iIdK7k+ARIhkJAYbp6zMU4vFyyhGukRDp7XGvu7GDkSVdD0RCJHgx\nOLebo4eaRogE5vGf5wU/aaGc3TOyShODIaAaUqYeuhuqJ9jGVmliMARUQ1yk/ktE0g4BFRH2\nyHP1VMYkRIIKQKRlIBL0CZ/XvRYgUokQYJ2eEYEhyftTiWQRCWwyksJ7nYRIxUOAOrNq3COF\nT6XB6VwpjRAJtJg3XASudzwuBbbUBpFABwmRYh4lPn03O+9EEAlUmDlwpIuUvuesIxcigQqJ\n5e46VzvzBqTp7+lCJKifJJE6m/S3do8v5UoQKRQneeRaBCKBDilV3Kv1vlGudxIXG5AQSTUE\nKJM+IHmGl79F3XUTA9I4EiLBNpis4QmRXF9G/3gU8SWnR4gEdhibMZKlJwIiGQgB9pgSKXy2\n5rmoCgTIlrtKE4MhIBORYp2oY481bkioRcefrANPOHmVJgZDvAP65TQxHMQLfGSNB//2g90X\n6DcibZci78y5RfJcC7n+PSbpHiWCSJulRGFFyjmh0iPyxERK3X1WEGmrFKmsHCL5Ww8XIFKh\nEJunQpFCjwJ1mod3V9YjRNosZd6iIzFnpeMfnCIjEiIVCrF9ilSWlEiRmYfg7gYD1rIOLAWR\nNkuht+hIyDnZzBap+/NouMoPIm2Ygmc665k0adA9r0h6BwCRwCbTIvk3HzRVS1elicEQMEH5\n4cxNfPBouLFXJK1OIBL4UL/GiOcxbQYiFQoBUayINPnc6mi7YZvAtuJpqjQxGAJiqF9iTKbS\ndFXxJdZVJ+5Rlq4hEnhQE2kcYrDklcnkyOQdk/wxEQlUUBJpHGQkSrJI4zWB9PP0DZHAh86A\nFBVpOLQMRfIPXNNZIxLooTkgBT5k1L+5OlqJSDZCwAT5x6NJkQZyRESa4RHXSLA10kUKrPS3\nSwwr3BeVJgZDgAEmhpaBHGkiJcaV7QgiQUnWiuR/9rsIiAQlGRd/WCTfkPRYi0ilQoBl/CNP\naLh6qFQ0Y5UmBkNAYaYL33MKFxGpMKoifX0c214fT1+5QkAdhOv/tTTkiUmTFEX62Xf6fcgS\nAmZTqAZD5R84lQvu4B1FOrnd57V99X3ZuVOOEDCTUlUYrP+ASLGHvd9OpJ27Pl9f3S5HCJiJ\nNZF6y6OeGPNIU6TB/YIcIWAexeoQkZY1aWFEMke5OhzHdb1P5PWyi32qyIZG2tdIl+/2FddI\nRrAj0uPn3vJEkXQynkBz+vvQeQ/Z/2QJAfMoWIqj8SgsUngP7ylS83Vq7yPtjh/cR7JBsVIc\nBO2MLr010dxMjUk82fDmlNLIDYXxKDGpCSL5dtslTwgwgrRIBupFX6Tz3rnjJWsIsM24/r1v\nooHzvc5WgzYljVK/j/Q34xCdtEOkbRMXyfesnccdN9xR2eFJW6STO/00zffJnXOEgCrwlPwy\nkRr/mhLloy3SzrXz3j9unyME1MG43n1D0lAk51s2HpDeQ6TXQcoRAupgYkS6r/L6EtIoabIq\np2LaIv17dJtHhN6ZaZECxgQsGq5JiinbI5Um93bu+HG+uM/flz8nHhF6ZzwFH/NnvkmRoLm6\npNLk3q57NHY8IvTGzBKpmS9SYkzRLqk0uXO9ns/HYzvlcIp6hEgbx1vUzn9hNJzmDoo04cqG\nRDIVAkoSq2mPJ6OfRrvoLouGRCTYEN7RZLAyKNJ9Y+/6mChZPUIkKIP3pKz3w98i1znjG6/3\ntQhHRCSom6kCjokR1ca3eTBqPo0QCRSYHAs8JgyXhCQLLEqJKgoiQXbmiuSdvvNuh0jFQ4Aa\nwwqPbDFhTUCk+COv2ft3D6fSxGAIUCOhpBP8iHiESAVDwELml6a3pPs/+iVJFqlzHykeNR+I\nBHNYVJzjRsElySTPBM5LdTGIBHOQFSntoVWvRt4o61NdDCLBDKKVHG/n3c3UjaOISP5dxqIm\nrVkKIsEMlorU38VakRqPN3PyWt0H3z5VmhgMAUtYLZLXiimPxhsMn7SblxgiQWnW1uAikfwf\nRPI7NieHxb3w7VSlicEQsIiVJeh3pAn91N0MkRaASHZZU4ABjwZ+euXxtx75lZIrIkHVBIaU\nzqr+huPPU3haR0QK6pLBI0QCNTyDSW9laMMIg+HLG28cA5GgYsYSRLdN0qmZHne8T4NLa4RI\noEan/F1/+BluNtg8g0gZeqfSxGAIUMY/FA0XjQSZEilsxyCeP75c91SaGAwByvjreJ1I8VGm\nvxKRYAv463i4sPvzPJE8goREylJdiAQq+Kt4lUjT8xe9RYgEGyBJpEGtT3k0ayIQkWAT+Is4\n4E1/bbJIE3WT0SNEAiWmRPIaMdekJSnIgEighb+GPeNRokiem7bLUpAAkcAEISMSxqTOVnND\nCuav0sRgiDrI9w5qjeDIMmVSf6PZAcXyX9nkvG+a773bf0klNA7xvqz8XVdlYfAc7T1Eutwy\n2d0SEjWpogLIyarftXChSDPKLeRIVCXfJs8lk/ElD9BKkQ7us7m6ffPpDkIJjUL0ltstjAzM\n+10HCjNHYuvx9CwoiW/dcJPh8unOGxPplsf19oeVNT5uaLo0MjDndz3aUrpQZPElF/Fk0qQp\n0cIZGBLp6C6IlIPtiuQ1ITYmTYmUZpJvl2JdWtfk4K4Xt2tUTu1Ml0YWZnvkq6x86S1nWOmT\nnkyakiDSc5GLf45pYZfWNbnccvm4ZXWRymgY4rXQcmlkYY1IpsfvBSL9bTaeJ+9pkSDSa43o\n0Vk9/b27XSE1+0+hfDwhngvfTqT033VlIg2ukUIeedOPbBxtPrnjlT1SaSITwnRpFMZ3bAwf\nq2SR7ltFlUjzaPbTRHN7pNJEJgQihanu2HSzHWsTsOKvzcZEypgY95HmU/OxSbOiU2gxc1zg\nAgiRYOv4fAkLNWyRWIRZParq1A42Sr/Cp0SKXVjFB+bHBogEm2S+SBPbTYbL0AehJl/HtZlM\nhoDKCRXwpDlDTVJU02atSKcsySOSMVb/fmNlMtejhAYiOc/r4LomL4/yP9kApVj9VhkfLOaK\nlLKN+v2AlSLt3GdzcN/fBz6PtGHSijKyxXCwiKyeVmSWSGourRTplubH72h01fk8EpQgqSRj\nmwxNiK5/bDbHrkmt8iMg0sWdG5WPUUAZsovkm4VLuRBKWFKLSMffU7tvt2++EGm7pBRkdJu0\noh5V/7RKHmNKmbRSpPY7Gw63XP+JpdQgkjESyjFes4k1vUKkZvD/2kT6vUD6/eefaz9LIQci\nmUJOpMRIEScShapOpDwgkjGmi3GiZNOreYk+HpO05+0QCUQQq1kpkbrLFEAkEEKqYj0mzBHp\nbwfaH6hePf2dJV9EemOe9bTQo8yflwhlva4JIoE8XiGC5mxCpD++DqIPfyNSDcgX6ajuE4eg\n/mmcb4P8CF0j/XAf6c2QL1Nf5ScNSp0LonETwQTj2Qs14dTuzZCq09c+/KWfqNKrad0inW9f\ntyoHIllHoFA7I0lvj3/DyjBQmkbd3ShqJDjZ8CGWUoNI9lktUmQkSfm6oKBJA69S8ljahf5+\n1jV5JL0/i2TjCwEWkRWpf22zWKTeHN6cLJZ2orsnlSYGQ8A6RD3qP4fg1ytRpHmJIRKU4Vl1\nSiLFTPKtnZdYL9I6Vog07pYYiGST3q96+e98XP5BU+aLtOgB2YUd6e5qeRNEejtEftURNbwl\nFV3rbzI7lXU9ane1sslxd/v6oK+d6P1YRLKJTN31Cz9BpdUihTMW82j913Fd2/9fZT/Zh0gm\nERGpX/ZxjzonP35lQg08AWPJrOnPY0/rmnS6KpCMNwQoMV1Q60Ryg4ue/i7jJvk2DDYPZL20\n14m9W9dk9xyReLKhbpIUWeHRs2lApMh8Q3/D3kp/o27DoF/CrD61292+GfKy48mGylETabCT\n0fIpH7orPa/6Tb37ysLayYbDX5J8jKJuUqttaT129u8XyWtKfEDqbVe7SM3n8aaR6Dd/I5I+\nuautu/9BJBcej9zE6ohGbixnpq799UKlicEQ0CNztQ3VGEWKaBRRZbhZlDw9e3ZApYnBENAn\nb7WFin68waD200RK8ciuSIMeFM4K1pG12npV4i3vSQsWrlPSCJHgScZq6xWJv8CXi5QwIOXq\nV7eLKk0MhgBF+hXtLfNJGYL2eOYUBktVuqjSxGAI0KRX0YERY5ZGnQcWvDtrBueT+Xu4ssl5\n3zTfe7cX/YN9iLQ1gjW+TKXukLNUJGHBVorU/lmX3S1Z/vQlxBjW7SqROjqFpOzsL5BOULFl\n/VvX5OA+m6vbN5/86UuYhaf6Z3oUPFH0PH4UCi/Yn3VNbqm0H6FIyunr49imfzxNjF+ItHkE\nRAoPY90oje+H8Zar+7OuyS2To7skifSz73Q1PoAh0uYZ1X66PgNlwo75Ag6iy/VnXZODu15u\nn6BIObU7ud3n/UMX35dd/IOAiLR5kr2Z5VW1Il1uyXzcEpt+bPXx2aUbE59f+g+gMtaJ1Jzv\nY8v+M6Hd8JxwuPpF6YMCMJeVIs2AEQk2zHwhlor0e410+W5fcY30NkQuReJXPgnb+BqFL36k\nL4qGu1/b5HJsZ+6+ExoeOn3e/whnBSbRFCke0LhIh3tybpdi0tepvY+0O35wH+k9CA8QU5Kk\nbBNpEUgmUzeb1SKd3eHnlt6Zv9gHHrwiDQaPsBhzNBp+dH2UR+6ermuycz/N37mpVEbDEFAx\nHpFeCybNmKNUQhZ5e7quyb2TDSKBn3ENd5d0bUkxar5DgSTkWSnS/m9Euj24KggibYVRDXvL\nP0Gm5RpleIzBF2Rdk79rpMvOif7JPkTaDiFnPIsymVSDSM3xL0fRT1Eg0nYZVfVKjZ4fLZ8R\nMku/1ja5tF8QmfCE0PIQUAdppdopajd/ai4uVCAhBY/4zgYQIrVan9uJGRSI/FqMSFAP6dXq\nfF/LJW9Sd2lujeREuop+iz4iVUdwYJhs0JdBzKXZCa1jjUhfB+cO7RPd1yP3kd6cYN2GSjmL\nSI9vRIkllIcVIn3d87w237f5Bv705XsTqNtgMa8VZkKnmkQ63OQ5ucPtU7LH6MPcGllBYfxl\nW04klSmGTn+WN3mku3PHa2TzJSBSfXjrNjws5BVJa66u05/lTR4iCX/LajcE1ISnasMixUya\nY1nkWkthrq7TneVNHiIJZjMMAZXTEWlY1TE5kk2K70ezjBAJcjIYN4brQt+llSrSxKaaHV3e\nBJFgkphIzy0GNjRiIinW0SqRsuWMSNtgXO2+TTpbLhDJyJCESJCPSZEeSxdqNPmokWJXVZoY\nDAH5iVV2SJlAu6goBjxCJMhHuLKjbszUKLJDzb6qNDEYAvKzSKQ5xHf3XKfSV5UmBkOAAs+K\nzuORi5wJDjPI3lWVJgZDvCdKb8+dcLlFCq/pZ5C9qypNDIZ4R9Sqqhdy8KKTSFb6gfJ3VKWJ\nwRDvSAGRAmiYpPtZCkR6H/SqKj2VeWIgkr0Qb4ghkaae7Al6NMulhmuk/CHeEFMiNQrndw0i\n5Q/xjryhSNxHAnkGJVYeBY8QCTKwTZMi92TVuotIb8Xo/bo47nkps1irxt/ysXtEAnHsidQn\npktEpGnB8meu0sRgiPdk9I5tjEUiBb8ABZFM/pI3QUmRPDGHiyIizXesQSREykU5kTxRw4sk\nTAqFyNQ9lSYGQ7wthTyaEOn1PzFCUfPMiCPS21FGpH59DxYJWOPV6C+KNxPp/qk0MRjindHX\naGCNZ5EoKZlI90+licEQoIun1KX1meWRtEmIBDp4ah2RcoNI28NT7AU0QiSoHte9GnouErKn\nt+OoJlk8QiRQpVf5/SVrNPLtZiIF6Y6pNDEYAsqQYMACjXz7ieYg3y+VJgZDQBnGxe6XY71I\nwyLKoU9n7ypNDIaAEow0aUJDUvJQNdh1SCS/XII9U2liMASUwCfSqjFpvG9E0g0BBQiJ4Fdk\npUj+yNm6ptLEYAgowKucJz1KHJL8u0ckvRBQgHkiJZjkCdD5nzdyrq6pNDEYAkrQqeZXUQcl\nmRYpercoFDlPz1SaGAwBJfCWc1iRFJO8tTJehUhQP50KHhdzUJAkjVJF4j4S1M7EYBD0I1Gk\n14XWeJfZ+/YKqdLEYAhQY5lI6XR3MtilTgfbkCpNDIYALSar2m/HPJX8WiFS/hCgxTKRFtPd\npVofEQmykzA8iJs0HVIaRILcpBS1sEj6QxIiQW6mi3qmKDPW63VSpYnBEKDIVEVLitRfq9RB\nRIIumpXXDzvLJEQyFAJGaNfeMLAETSGPEAmemBTJ+yEmRLITAobkqL7EvfmM6CQTEy2GYEem\nu6DSxGAIGCJff8n7m5AhtMrzqpRHiAQPMosU3/Eik+IiiXUjCUSCB5k86n6yKGnzBSJ5Tw5V\nQSR4UFKkmCTR1cFNxLqRBiLBC9n6CwwYsS3D48qkSMNtBDuSAiJBNl4lPTFYPJYvMmm8nwIe\nIRLkI1Wk2GjjOpMVwdVufHqn2M17aJUmBkOABgENxltNmOQSPun3DFhAI0QCHdJESvzaoIhI\nxUAk0CFW892RZ7FIZYsGkWAKuRoNVfxreZJJiGQoBCQiWaXhfbnRdMF8p0RS7GUzp8mCKPOb\n/PH1cWx7fDx95QoB0shWaXdPvb260e2mYh4t2p+iSD/7Tq8PWUKAOBne73s7HkTpeoFIfk5u\n93ltX31fdu6UIwSIU16kVKOEE5u5Q0WRdu76fH11uxwhQJxcIvUE8HqRvDAhxTn5mxdp8DBH\njhAgTx6Ppi+HAg90R00KpDmvC+ZFYkSqEgWR/EosEin2TOzM1Gb2Z97mC5u0/F4jXb7bV1wj\nVUUGjZpFJiW1i0Sal9nM7szbfGGTO4dOd/c/WUJANUyKFHDEuyJq0lyRKriPdGrvI+2OH9xH\nAs9H8eKOhNrJi7SkMypNDIYAA/jqf9Kj0UaDBcEwebui0sRgCChEt6KnRQoIMloc82ijIp33\nzh0vWUOAWYY17av/GSZ5WoXCSvbCF0Glyb1d2/BvxiE6aYdImyUowPOH7sJkkVzvg7Il0Bbp\n5E4/TfN9cufx6sAxg+3g/Q0Pz8+aBJG61pgoGm2Rdq6d9/5x+xwhwDjxt8rOygSTknbp2XpF\n+rEdqzS5t3OPfzr/lw0Bxola4ZVnhkjTVZNx6NIW6d9DJB4RektSRfLfd43u8I1EOn6cL+7z\n9+XPiUeE3pM1I9LEDqcFSTduPqoiPXvh3I5HhN6UxNElVaUFA1LtIjXX6/l8PLZTDqeoR4i0\nYdaIFB/HUkNXL5KpEFCMcCVHBOor0NnFHDnyeYRIFZKpFEwQGod6487Ah/TjkSDS0oNbQqSE\nq8K1ITZMxnfVUvS6Ex2QfKPT8lDe1Qv3jEi1sTmRPB3yj0uegcqzL+lcUluqNBnsAJGWE6mi\nSpETafWRWXFwEakyNieSv0OIlLIDRFrO24oU+AqHyT2tzyWtqUqTwQ4QaQUb8yhYvG7wGPjr\nh9AhmNAg5aAtP7hMf9fG1kSamszurO4pFdxR9GbvvFxmgEj1sRWNRnoMfxgviXc9pkGqIksP\nLiJBGfojjWdpd8vZuwysyvcWhEhQBm9hx+s96dQsGg2RYGP4K9sFCt6te54BkWCrTIkUfGpo\nXbzlCU/tX6WJwRBQlvDQMyHSuiFpecJT+1dpYjAEFMZf2T5j+h4tnFTj1A62SaiwEUkQRHoH\nAmU9IdLiWIgE78UMk9LN4BoJ3o2hNMFTuzlyIBK8HxlEyv1kFSKBQbraTHlk48lDRAKDGBUp\nEguRwCBdcXqvvNvoJhVYuWB/69IxEgLM0rNnNCR1N5v2SFIzRIKKGJzJBUVKGZBER6x4PEQC\nWwzO6B7LRnWcMiAhUv4QYJPxgDRc4f/ZV+BJJ3/zM/OvXbDDlQnZCAE26Z/S+Vf0FoxeR5qI\npBZYuWB/69IxEgJs4vOlt2a4pXddYF8iqQVWLtjfunSMhACjeE/gxiv+lvS2CpkkmVtw1YK9\nrUnFTAgwSlCkSB1riRQBkcAa4+ugKRfCmylphEhgGZcokubQE8pApYnBEFABS0Qq5BMigVmc\nXySvKsEZCSUQCczinEeluCqIpB0C7DNfpNRTQXkQCeziMSmuCiKph4AKQKSVIBLcmSkS10jq\nIaAWmGxYDiLBk1kicR9JOwTUg0u4j1QaRAIQAJGgeiwMUYgElVNsfqGfhUoTgyFgKyBS0RCw\nETtbKaoAAAk3SURBVMrdg+2nodLEYAjYCIhUNgRUS18aRCobAiplpI0JjxAJKgORbIWAOvGd\nyZXXCJGgMoxcEo1AJKiKFJH4PJJiCKiUSY/KjFiIBHWBSLZCQLVMnteVMAmRYFsgkm4I2CiI\npBsCtgrXSKohYKsgkmoI2C6uwNfpIxJsE+WBCZFgmyCSTgjYNtqTd4gEmwSRlELAtkEkpRCw\ncbhG0gkBGweRdELA5uE+EiJBbaiK9PVxbIfb4+krVwiAIiiK9LN3Lw5ZQgAUQlGkk9t9XttX\n35edO+UIAVAIRZF27vp8fXW7HCEACqEo0uCryHKEACgEIxKAALrXSJfv9hXXSLA1NKe/D51Z\nu/1PlhAAZdC9j3Rq7yPtjh/cR4JtwZMNAAIgEoAAPCIEIACPCAEIwCNCAAJwQxZAAB4RAhCA\nEQlAAB4RAhCAR4QABLDziJDrsjgEQBF4sgFAAEQCEACRAARAJAABEAlAANUnG5In5hAJKkNR\npDMiwWbRPLW77uIfnhAIAVAG1Wuka/zBIIkQAEXQnWw4d55bzRQCoATM2gEIgEgAAiASgAAl\nRJp+uBuRoDIQCUAARAIQAJEABEAkAAEQCUAApr8BBEAkAAEQCVTY+ldDIRIosP1vWUMkUGD7\nX1iISJCfN/jqT0SC/LzBt+giEuQHkYSaGAwBmmzfJEQCBRBJponBEKBLX6Tt6YRIoMTLoy2O\nTIgEWgxF2tRvGZFAj+d5HSIta2IwBBRjk/MOiATaINLiJgZDQDm2aBIigTo9kTZiEyKBPm5I\n6YTWg0hQgodAiJSVDRxYiLK5IQmRoAy9W7P1u4RIUJDtDEuIBCXZzBkeIkFJNjN/h0hQFkTK\nSLWHE5awBZcQCcqzgfM7RILyIFImajySsIbqTUIksEHlKiESWCFNJaOSIRIYYlIks+MVIoEh\nJockRLIWAmzSFWlsjN1LKEQCY8QulRDJXAgwCyKJYe84gR5uyHhdsdyCIBLYw70+hD4QB5Gs\nhQDjhAYlkxohEpglNulgD0QCo8QuleyBSGAWRFqL7WMGelRjEiKBZRBpFaYPGWiCSGswfchA\nlyo8QiSwDyItxfYxA3Wsa4RIUCEWtUIkqAybJ3qIBJVhc/IBkaAujM6GIxLUhdH7SogEdYFI\ntkJArSCSqRBQKyY9QiSoD0QyFAJqxppGiAQgAiIBCIBIAAIgEoAAiAQgACIBCIBIAAIgEoAA\niAQgACIBCIBIAAIgEoAAiAQgACIBCIBIAAIgEoAAiAQgACIBCGBUJIDKWFDl8uKUxHh3SG8N\nptMzndx8jHeH9NZgOj3Tyc3HeHdIbw2m0zOd3HyMd4f01mA6PdPJzcd4d0hvDabTM53cfIx3\nh/TWYDo908nNx3h3SG8NptMzndx8jHeH9NZgOj3Tyc3HeHdIbw2m0zOd3HyMd4f01mA6PdPJ\nzcd4d0hvDabTM53cfIx3h/TWYDo908kB1AIiAQiASAACIBKAAIgEIAAiAQiASAACIBKAAIgE\nIAAiAQiASAACIBKAAIgEIAAiAQiASAACIBKAAFsS6bRzu9NP6Sw8dL+Z3VqS50cFdBIzlOMj\nPcuHsGVDIh3aQ70vncaYa6cKrCV5ffzlhU5ihnJ8pGf5EN7Zjkhfbndtrjv3VTqREVd3fLy0\nluRvKvcK6CRmKMdneoYP4R/bEenkLr//frqP0omMOL9yMpbk2R0ep0uvxOzk+ErP7iF8sB2R\nju676b11meHszo+XxpJ0p+avUjuJ2cnxlZ7dQ/hgOyL9HfIlf20tM0d3+fd7eXx7aSzJ6zCj\n2//s5PhKz+4hfGAsnRUYPcDNrQpaDo3FJC2L1HREMnwIW4ylswKjB7i5pfTZND+n29mJvSTr\nEMn0IWwxls4KjB7gJz+3GVt7SdYh0h2jh7DFWDor2Nk8wC9umdlL8i+VTmKmcuxnYS69F8bS\nWcF9Nufb2mzOi9uv3l6SvVm779esnZEcxyKZSu/FdkT6aO8vXNypdCIjdu72PEv7q7eX5F+l\ndhIzleNzwDR8CFu2I5LRO97N7Rbiqb1SvlhM0vSTDc/0TB/Clu2I1OyfM6TG+Nm1mbVvoeaS\nfJw7dRKzlONfeqYPYcuGRPppnwounYWPW2b78/OlqSQfInUSs5RjNz2rh7BlQyIBlAORAARA\nJAABEAlAAEQCEACRAARAJAABEAlAAEQCEACRAARAJAABEAlAAEQCEACRAARAJAABEAlAAEQC\nEACRAARAJAABEAlAAEQCEACRAARAJAABEAlAAEQCEACRAARAJAABEAlAAEQCEACRAARAJAAB\nEAlAAEQCEACRLOAe/P2Juku79BLaOL4r6eQgBQ67BQYi7dvfyj7wu0Eki3DYLTCo/vuPISUQ\nySIcdgsgUvVw2C3Qqf7fl/eTvL9TvaY5793u/ve8m9POnV5/6Nvt2//v3U9zObq/v/T9vM5q\nhq0vB+cOgesuWAsiWSAm0rF9cbitO9xeHZ8bH9z377/fv+s+7ldYp2Yk0qv1+b7NWbVj7wMi\nWeA11zAaUS7u8NP8HNzvUPLpdtfmunuK9Ok+fv/9+F3l3Oftx3H7Tuudu9622Zfo3xuASBaI\niHT8PXG7ncYdby+/mpsbz99Za8Vrcs/TvtPaOU7rMoJIFuif2vVEGt5i6m787/fc7rs9oWu+\nLx8Hj0id1qffs8LrVa1L7wYiWWChSF+/53andpg6BEa0Tuvm4/ek0O2+1Tr1XiCSBaIiDbfq\nLNntb//dhqb9+fLtFakb5nLac42UCUSyQESk4+vS5v7yq7PxyZ3bCYd20UCkr/s10uDCiNtM\nmeC4WmAk0nfz+LedqmvOt+mCy2DWrnWnnU1wv+d319c10t6db1N1rtd6f5/ZY0TKAyJZYCDS\n/vda5vHv3+VPe23T3hT61x1U9vcbTKe/K6Gve/vz637Tq/XncxPIACJZYCDS1/6m0P3f9tkE\n9+8+R/DRfbLhxuffqduvXYevy32W+77dv9eTDX+t2ycb8CgTiAQgACIBCIBIAAIgEoAAiAQg\nACIBCIBIAAIgEoAAiAQgACIBCIBIAAIgEoAAiAQgACIBCIBIAAIgEoAAiAQgACIBCIBIAAIg\nEoAAiAQgACIBCIBIAAIgEoAAiAQgACIBCIBIAAIgEoAAiAQgACIBCIBIAAL8D+ArZ8HztZBG\nAAAAAElFTkSuQmCC",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Predicting ADR using a linear model \n",
    "\n",
    "mod.adr.lm<-lm(adr~lead_time+ total_of_special_requests+ stays_in_week_nights+ stays_in_weekend_nights+ previous_cancellations,data=hotel)\n",
    "\n",
    "plot(mod.adr.lm$residuals~mod.adr.lm$fitted.values,pch=16,xlab=\"Fitted values\", ylab=\"Residuals\")\n",
    "abline(h=0,lwd=3,col=\"red\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2.2 Binary outcomes \n",
    "\n",
    "With a binary outcome, we can adapt the squared error loss in the previous section by treating the outcome as $0$ and $1$. Alternatively, we can also take the the <span style=\"color:red\"> likelihood </span> approach. The likelihood for observing $y \\in \\{0,1\\}$ when $y$ is a Bernoulli random variable with probability $p$ is \n",
    "$$\n",
    "L_1(y,p)= p^y (1-p)^{1-y}.\n",
    "$$\n",
    "For a binary outcome $y$, we will often model the probability $y = 1$ as $f(x,\\beta)$, i.e., $p=\\hat{y}=f(x,\\beta)$. In addition, we generally consider the \"log-likelihood\" or the \"negative-log-likelihood\". These lead us to the negative-log-likelihood\n",
    "$$\n",
    "L_2\\big( y,f(x,\\beta)\\big) = - y \\cdot \\log\\left(\\frac{ f(x,\\beta)}{1-f(x,\\beta) }\\right) -\\log\\big(1- f(x,\\beta) \\big).\n",
    "$$\n",
    "\n",
    "It is slightly more traditional to model $g(x,\\beta) = \\log\\left(\\frac{f(x,\\beta)}{1-f(x,\\beta)}\\right)$, since $g(x,\\beta)$ takes values in the whole real line while $f(x,\\beta)$ is restricted to live in $[0,1]$. For instance, we can assume that $g(x,\\beta)= \\beta_1 x_1 + \\ldots + \\beta_p x_p.$ This is equivalent to \n",
    "$$\n",
    "    f(x) = \\frac{e^{x_1\\beta_1 + \\ldots + x_p\\beta_p}}{1+ e^{x_1\\beta_1 + \\ldots + x_p\\beta_p}},\n",
    "$$\n",
    "which is known as the <span style=\"color:blue\"> logistic</span> regression.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Predicting cancellations with a logistic regression\n",
    "\n",
    "\n",
    "mod.can.lg<-glm(is_canceled~adr+lead_time+ total_of_special_requests+ stays_in_week_nights+ stays_in_weekend_nights+  previous_cancellations,data=hotel,\n",
    "               family=\"binomial\")\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "0.0900621118012422"
      ],
      "text/latex": [
       "0.0900621118012422"
      ],
      "text/markdown": [
       "0.0900621118012422"
      ],
      "text/plain": [
       "[1] 0.09006211"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "0.0530973451327433"
      ],
      "text/latex": [
       "0.0530973451327433"
      ],
      "text/markdown": [
       "0.0530973451327433"
      ],
      "text/plain": [
       "[1] 0.05309735"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAMFBMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD////QFLu4AAAACXBIWXMAABJ0AAAS\ndAHeZh94AAAgAElEQVR4nO2di3biOgxFHaC0Q3n8/99OgRZs62HZVkISzl53zYXEluxEmzyA\nEi4AgG7CqwcAwBqASAA4AJEAcAAiAeAARALAAYgEgAMQCQAHIBIADkAkAByASAA4AJEAcAAi\nAeAARALAAYgEgAMQCQAHIBIADkAkAByASAA4AJEAcAAiAeAARALAAYgEgAMQCQAHIBIADkAk\nAByASAA4AJEAcAAiAeAARALAAYgEgAMQCQAHIBIADkAkAByASAA4AJEAcAAiAeAARALAAYgE\ngAMQCQAHIBIADkAkAByASAA4AJEAcAAiAeAARALAAYgEgAMQCQAHIBIADkAkAByASAA4AJEA\ncAAiAeAARALAAYgEgAMQCQAHIBIADkAkAByASAA4AJEAcAAiAeAARALAAYgEgAMQCQAHIBIA\nDkAkAByASAA4AJEAcAAiAeAARALAAYgEgAMQCQAHIBIADkAkAByASAA4AJEAcAAiAeAARALA\nAYgEgAMQCQAHIBIADkAkAByASAA4AJEAcAAiAeAARALAAYgEgAMQCQAHIBIADkAkAByASAA4\nAJEAcAAiAeAARALAAYgEgAMQCQAHIBIADkAkAByYQKQAwMJoqHJ/cV6QAgBPIBIADkAkAByA\nSAA4AJEAcAAiAeAARALAAYgEgAMQCQAHphTp/BHC9vAbRI0CkcDCmFCk83D7KMXuHgQigTUx\noUj78PVj09ewvQWBSGBNTCjScO94GjYniARWxoQi/blz3m4hElgZE4q0Cee/R1uIBNbFhCJ9\nhY/fR6ewhUhgVUx5+3v/sOdQ+CIURAILY9I3ZI+7v0enD4gE1gQ+2SDyPGo2fJHY1kVv1fT9\nZfAaIJI0hMc38Ru+k2/rordq/lMAZiCqIxBJGoKDSIVOeqM2kew9xhf1rXiVSHO/2fBUoeHP\nxPB/WiYPoQeO1o4jB0RyZT4idf5tI2daRHq04SZCg+izDRk1o65oOoNNvQ5waieMoFqkqFVa\n/0E4QdQ1aRGppjFE8gUiSUOIRbAUXNwsqn9FCF2TBpP4toXwhrigDESShlApUlKY0RObSMZD\nUmEYShJtgsABiCTyrLJy/ZIi/nuS65DffhBE+gvJOamPRMhhagvagUi98Kb8PdQ8uvDHkKzG\nn09aRJIS/66rmilQgEi9ZMeeZFn5SoddzIukORF35UJBmbGZUKRCSXmkeAGsKdHjwgGF2x75\nojRS41vDC9qki2RCkb7WLRL3PtKlIFJIrqKYkEKmitMyV5Hgo8iUp3bH+59rGDPF9HCFmi4T\nDyixgQaR2j615CiSo5HrY9JrpGPYj51icrg6ZYtXFkk6JAmp5AbaACunNW6kFTLtzYavcBw7\nxRho1cO+4CvLmAOPVSR6cCu98xppWpiiAcdj2wrBXbsiavmIJU27JAeUiyhSSC639KSBOb7R\nhE61D5E0IFKRBpFYE6K6zvyJQphqtVIkJyCSBkQqoddPVXUF4a2lWpE4k0SzTEMrtEoGbgg3\nS8YdOkQqUajH6urKPYpO9szF7yxSod3f6kWLNPbgIVKJ8UVqKf5XiLRokyDSyyntgsrd0ycS\nMUcZZqVH5cvABYtkf1VpTTBJlzmlqN6Y3nvgGU82yTCU6UUavRhHBCJ5R27ZnHyH1r0SDaFH\nJDoA5rn1TgNE6k4wSZf5pHDbnO2BVJFUP6uqwdg0lE/Z/lYvWSRcIzkHdqsFJY4ePhnC84ll\nTP4imQ40j9UL9ggiOQf2EkkKVE6QVG5IcRy9ra1R41R7Q/Y5Mu7IIZJnIIsVaRtZJC5CxeBN\nM63eHMvVaGzeTCS3F9W0BKP/lw8vikih8OL/cpGABETqDxRLQEwSbdBQRaofYKkNROrn3URy\nOzuxiaT48LuGF4mt8LqiN7WGR168n0huhPx8ziyS0oc6RTpVD1BvApFcgEjdRDUveKTfNtB1\nYvK4bh1o5ANE6kYSSX0Lk9NEOFCRPKj9GQKR+omLu3hciXspoZjeEGnOQKR+xHKvr/tne6Y7\nTJoxEMkD4QTMWvZxC+UAlIUcUaeZmDqTYZiASPUU9m9ILo5MGnGycdZ0HOuUwRoHNDUzGYYR\niFSLaf/WFLrYkrOGLOoqNT71TCp4JsMwApFqse1f+wmY5lxkiyBSe7GlcY0DmpCZDMMKRKpE\n3r/JMrnQ864GkeImTiIp/WdSwTMZhhWIVEm0fzlzkgXkzjh3POMK5vks7pR8IKnTJKNIL6xj\niOTAjDeeVMLsXk+tE2rXvMBRJLV7Puba4E4syiOIVM3f/k33M1+VikiKgmKcPJOXSMIMraU8\nUrVDpH7mvPX4GuYL87lELX7WGkGtNISh2NjVBQ1Dejlm2R5qm0aWoxFEquTvhZpUmaxICPwB\nSSwSdr2QqFjFiip/AUp9XyfSkoBIFcQ1o4lEDkn9ItG1BhMuZZEMfU0J3t4kiFQBp07yhCkq\nIl5RpEKR22LkjdlVhc5+tr0BEMlOWjNlkaKW5B5Fs0hWG7kh1wKR7EAkO3nNJI9IcWfmkCv4\n8pWNPogJRHI7bL0BEMkOV5XEDlkkJYqUShmDycaRy3z8DMsBIlVAaoaW9WO1UPHGY4lZJLW7\nLVcl+VTEw9Z72QWRKjCIJK8iR65yIvluG5+QaeNdz1FafRr+Bs8biFRFWhraCz8VKXtbtpBG\napRYyS2269pCkl7LAJHG6DLDFB4oHrG36bTmfGBxlbLYmqSFKKyeYSyRZwtE6qBTpPKHCqR1\n8kjIqLomKOcpZYBIo3SZYQoua/VuN3hEWsV2lQLXDyRPVzkhc55SBog0SpcZpqA5G3a8TSR6\n5DIUet1QkngjisTfQ1EaeuefLRDpkbNpzz9rl43Gm1RZ6pZGfLbKyRgwagSRxukywxQkZXPx\ncX2ySusRydYsbTVmBZuH/k4aQaRnykaRol6PzsQT8jJeYZJ1WNPVbftrzoqBSH8pSXXYKiV3\nI4nFvkea3xCoHVcfHpHgEQUiPXKm1WEslmeds0cbSaTSCRLTs3d+NbOaJMq6gEiPnH0iEaOy\n3iH5NILuUbymT6S8n5cC0CgHIkVZ0/M6U81RkQrHGtJNXss8qYN07bMSKEAkYQTWkmOOQ4au\napt0JURaBBBJGEGtSIxRrfHzda2FT3PYZgXRGoBI0hDML90huX1g7KkduLyOG0wcu+LR085h\nvAcQSRpCfS3nRulFqOji45GHSE4jUXKNFXpqIJJI+15+aKQWYfGQ1Jhdj0PiZgvSUY0q0siW\nTgtEGo12ke7rxh8B1yQZlTbESYa3HCDSWBSLcPxzO8NR9YUijWvp1ECksSjVSbyeOf/yrDDb\nIJgzUohkZlKRvj93ty2323+PlWI+6HUSntAb5r4lZhtGfvu+3NdpXBCplvMmqp7tKCmmpFgB\naplkx6PUpVeIlI7HNIcxBzYqYySdUKR9GP4db49OhyHsx0gxHYYiYJswqtBSdhVJD8aaxAx4\nDF4k0jhpJxRpCMfH42MYxkgxFcZK5zSiqvDHhPp9LXToFGlUXqDRCkRKxl54MW9MMRWtZWcS\nKRj29V8rJrQ41uJcphfpFYw0URyRGmgtO0EVrpRLGpEByAMqjTRLvnaXli/SzzXS4XR7tPRr\npAlEsuYnN/q4i5xC1Dz/ulVavkiXbbSnNudRUkyDqeCYtUm3eH1dFXOtiyKV411CfCu+NIjl\nMs4Ep30faX97H2nYfS78faRyuZWLmqw1q1QQiX1vVzjnyx/ZbfbhFcauQKQ5pejC/iovLVXX\nN4gkaiiawS42y+zCpM6mef1jTtJlhik6KewLrXzjBrXdsmbRYl4BRYxWkYLfDQl2vB6BXwBE\nGgXx6KCuLyymzZKWXN+iR8wQVZEMppkh04geL86oV4m07PeRinDVlpZJTRULr9xCdzagfPAT\nepQmZip0vVUI2SAej7xUnZD5iJRv1WXDzCMvE0UkNpbtvjjfUxqMPAg2Xt1OKjXLR/d8uMAq\nwKldlNVj14XkmJOuSMskXp8JRkKyJpgGYxHpecuOO26K3YzZzcMLhPIMZwNEeuT02HdpGXKr\nhArW6odZXhisEFkcjbpEnIFtW9gPSdLmWQQQ6ZHTY98pQWiZcHfc2O6iB2wW/mJMtYJpSkRg\nQ2kbzKIDm7HiXHY+QKS/lB47Tw0ilolY7HnPYh4mDFv/l7yEuUEUGhTvCagbg52dMfAsgUh/\nKY27vT2IWCZCqWd9LXm4OGJ03ZPcyvxJMiN1a2gbLG9mCzxHINJfSqWMa0NIQYQySYs3j1cx\nWFaZ3BBy5cY3yxYpF2qlQ5KwlhuEumjeTCiStj+dUvTQ7ZF2103PVLDCPFh+C6siaZ3zjrUi\nLVCHdiYU6WtdIimVXzgNM1RkaUvxa/gNLJugdA75MvmOR50sa3VrylO746D/yROHFF1UalQ2\nJl1MTqLEcJbSZxcLvUJ6Ka/MibTSRGo4iDeYtxAmvUY66l/n80gxFUaROCH4Smdb1NZdVPSm\nIUtt+MMn6SxHlPJApL4uf3xF3zYfKcU0sDUu+5G0LorEdzMPrHElP4XyLW5TlHTFGk3CXbsm\n2kVi9UiW9XjUiTQtqbEexhh/HUCkJmSRuEYlkdJFZpGe4o06LbVpXRSI1Nllhik6oQVBC1/1\niAnGH5AKI/CrzLpwLSKt2COI1IgmkvZWqC5SdNusWG7OIlWG01pCpNG6zDBFN7IM+W3s30dp\nszwWI5K2FThJe6fjJZJ0tFqzRxDJC64QFWfEzqZTLKJRd3Ey4dSg6Xj5SHKOzsHOEYjkBF/W\ntGjKr9bzEKkUVminDQgidXeZYQp3zGUduNf7pGcpkL9H3iKVkvQOd35AJC9KZZ0spu3SJZOL\nlB9RDYFDcvisG3f3cGcHRPJDq5PiC3ezSE7vI9WLxA200A0idXaZYYqxEKqk6EmNSPbTyJph\nhvj93XFEKp4wLheINAnFesuXmOqxuSrFntYBiKEK3SBSX5cZppgWZ5Gs7zYVOgvLTQPg+iTj\nktvXD3gBQKRJKHrCqmWI13NAKtyat8bO2nWNbLlApNEJTNXTUqMNjDF7BiQsrw/JBuoNsygg\n0siEhGQ52yx+ZojYNyRheUvQEeIsCog0MsaqZ0XSPx6gRKzJVV5eC0QarcsMU0yEQaNA3tjU\n+hhEKpax1Nup/iHSaF1mmGIiiiL9rWQPSMpn8rLV2WOTSObltbyhRxBpZHpFet4vJ/HS1eRJ\n6eyuarm+SprTOwGR2jDXic2j37O7+z/ElSiCk0jVVIZ8N40gUhsVZWUVKVyoJYxI7Lmd9syJ\nEUKuC4jUglBWfKWpFRgpwWuUW8N4JHxuwmGedJieQVcFRGqAL6u2UivrI2oDkWYERGrgtSIx\nA2DWN06tNErfsCsCIjXAlhVT3HXBYmNCcmlEspVE8gceFYBILTAHiZoP+cR9OJPyJDQYl91l\nZspIIZIGRGrBRyR6FIqPQdyhKu86yuS4cV7GV3XhQKQW2MME45HpI9xRpT6f86SdR5iYNkqg\nAZFa6BQpFYfeIxBFYsLZarzDBIhkAiI1IJX286km0t8asY3skdTUNl7z/NjBNPXuYlH6QqQG\n1ENE2kJeZxBJvUxKYj2fKqOxT9A4k1F5mb9tQKQGukQimqiddZHSpZZ4ZI15rqWG3kAkB+a+\n+YrFpahWFkn7kKr23lWtSLZSfa1HizEJIrVQ3sdS6ZPjjRY+DlAWyaBu7SzsrdyBSB7Mf+uV\n9rAskvUtpuzNKj5avEwuPXUcprO7UhN/IJIHc9x6tftUNEk0TAwQP2XbpO2lSFL8mllNx6wH\nR4FINlp2q1ipJpUiEeNDEtMs78EHsw9vHsx6cBSIZMNVJFO8WKTHIac5X9rMOobXMuexESAS\nk91+vV4MpHukxQsp9nTGzyXNX6RFAZFIbq68XiASufNtTWcWyfKaD9GMQCSS208kuQ5HEcly\niKmbCI5ZZiBSnpqvNO+SSrMIkRsPSZa8EMkZiJSnHlOkZ4T8cFM8dNmSF1tViVRn3XsDkfLU\nUvG4aPQIPJJIbABxEMYBQyQDEInkHq12CiIJKc3VzDWhXSHSOEAkkrvxgFR5VsWKxMcwVTPf\niFtaIQY8MgORmOzi+0hqn1LJFUWSYliPR7RZwyElaQyRzEAkE/4iMbe39YOKHp1Xplok0hwa\nGYFIFooFaalYro3hkERXcWnGEgkYgUgWXETiqjR2JI4RhOMWH0UeQJtHMKkeiGRhNJH4mw5x\nw3xdnTJBaN0zDcABkUwU6ytyoBBG6Bn7o4mUWlUeYZVKEKkZiGTCKpJU4Be5oqlIaUOjSMp9\ngcLg41XwqBWIZKRYXrEK8UIK7fi3NFOOhlCM1QZGWj+fpesgUisQqRPm9ZxbotS/bF+U4rnE\nQSTGUzIf6FQHROqCfT1/liDvEXeZxAQVztzUSMoYeZGsctt5UwOnFOn0EYbPy+VrE4b9SCmm\nplEkrdjKltSqJHjEnEaKXZ6LK3O9DROKdB6um/jr87alt6OkmJDA3YoWS5bVSQxcqMXxRVKX\nVuV6GyYUaR9+jkP7IXycL+fbY/8UkyFIQcqoXqSyITYf8y7R/5OubJRGkSpHtSomFGn43XHn\n2/+GMVJMhihFVkOcP3qxlUrR6KPSVb/ZIA+CWxi4Ax1EGq3LvV94/nvRN/Xc90NFMT8aPBrq\nBVmIyYhZO2rmyk5uyqUWWkGk8bvcGCKRzss+IrUX80UpSOVcS8xc79Gf1Za2tnEzI7OPai28\n4Bppf/597J9iKqKCaagaUmxJBaqlSDxqFImu4hfxw2SfQ6QJutxY0127jnoRX9gfC5XA7Rop\nIml+xSvMItWMai3gfSQDtDZqK4aWIw1mCNmhkTJoH5He9873jSlFmlUKO3yBVGpEX7jJ6lqR\nzOm5YXBBiyvydqxX1cNaBxCpSH+BFEQi53aFOO2DYXt6idR0vbgaIFKJ3uLlb8pJIo08lKqo\nzIq81Turk/IqkZbzPpKzSDUfJfAfiha2YgUgzEekEOORwolJRPpdYYzUPJLKqPPbGfMFp3ZF\n+sspi9AebKzKFmNCIysQqYi7SJ2D8QgDvIFIBvqrF/W/diYV6ftzd3tp3u2/x0oBwEuY8iNC\nm+huwsI/IgRAyoQi7cPw73h7dDoMy/7QKgAZE4o0hOPj8XHZX6MAIGNCkbI388dIAcCLwBEJ\nAAemvUY6nG6PcI0E1saUt7+30V27zXmUFAC8hmnfR9rf3kcadp94HwmsC3yyAQAHIBIADkAk\nAByASAA4AJEAcAAiAeAARALAAYgEgAMQCQAHIBIADkAkAByASAA4AJEAcAAiAeAARALAAYgE\ngAMQCQAHIBIADkAkAByASAA40CvS9/76t4G2pb+K35MCgPnTJ9K/59/F3xz8BgWRwNLoEem0\nDduv4/UP1J2/P38en145KgBeSYdIh7CP/8rjaR/cDkoQCSyMDpF2+R9LPX/0jiZPAcBCwF07\nAByASAA40CFSSHnxqAB4JRAJAAd6T+12w/VW3ffgdp+BpgBg/nSKtP/9Fb6j/sNhPSkAWACd\nIoWQP3ABIoGF0SnS8Dgiqb8J25MCgAXQfWo3XD+uehjCp9eI8hQALIDemw1/vwu78xoQTQHA\n/Ol+Q/bf9Wdhd54f/SYpAJg9+GQDAA5AJAAc6BbpsLve+d75fReJpgBg9rjcbPhZNriaBJHA\nwugU6Stsz1eRvoLrZ4QgElgY3W/Inu8fasAnG8Bb4/ARIYgEQKdIm98j0jFs3IZ0gUhgcfhc\nIx2G8OU2pAtEAouj+/tIvx8R2noNiKYAYP64vI8Udv+chsOmAGD24JMNADjQKdLO9YuxbAoA\nFoDXN2R9gUhgYTjc/h4BiAQWRqdI593W9wddaAoAFkD3qR3+rh0AEAkAF3D7GwAHIBIADkz5\nDdnz/vrH7z43IWwLn4SASGBhTPgN2dPw0/I8WD6bB5HAwpjwG7IfYXf++efjR7nTh/63wiES\nWBgTfkM2XN+8Dfd3cM/6nziGSGBhTPgN2VuTIURPPEcFwCuZ8BuyH9c/uP95/6v7Z/0iCSKB\nhTHhN2SPYdgfL7vhx6TDJqh/5BgigYUx5TdkD8PzcxD6j1dAJLAwpv2G7L+Pze2nKz4LN8sh\nElgY+GQDAA5AJAAc8PqG7ICfvgTvjJNIp9qvUeB9JLAqOkQ6hJjKv7RKRUqi1Y8KgFfSc0Ta\nxB65fuMcIoGFgb8iBIADuGsHgAOT3rX7/rx/EGK3L5wIQiSwMCa8a3eOr6nwxT6wKia8a7cP\nw7/bR78vp8OAL/aBVTHhXbvh/g2KG0d8sQ+signv2gV7R4gEFsaEd+1wRALrpUOk+9fM7R9G\n+LlGOty/PoFrJLA2JhTp/qe7/q6p1F+xgEhgYUz6huz3/vY+0rD7xPtIYF3gkw0AOOB3ahdu\nf/rxZaMC4JV4imT6CyhjjQqAV+J4alf95b76FADMFM9rpP2uZySmFADMkw6Rdvkd7HP5D+lX\npgBgIXR9aHUfq3Ta6389tSUFAAuh59TutA3br+NVpvP3589jt5t2EAksjb5rpH/PD4Bv3A5H\nF4gEFkfvzYbv/fWDP9vSV157UgAwf/DJBgAcgEgAOACRAHAAIgHgAEQCwAGIBIADEAkAByAS\nAA50fR9ptF9igUhgYUAkABzoPbXbDdfP2H0Pbt+goCkAmD+dIu1//+jjUf87dT0pAFgAXn+y\nGKd24K3pFGl4HJHwq+bgnek+tRuuX6A4DOHTa0R5CgAWQO/Nhr8/Q+z2d09oCgDmT/cbsv+u\nf4V45/n1WJICgNmDTzYA4ABEAsCBbpEOu+ud753fXxCiKQCYPS43G36WDa4mQSSwMDpF+grb\n81Wkr+D6GSGIBBZG9xuy5/uHGvDJBvDWOHxECCIB0CnS5veIdAwbtyFdIBJYHD7XSIchfLkN\n6QKRwOLo/j6S94/10RQAzB+X95HC7p/TcNgUAMwefLIBAAcgEgAO4I+fAOAARALAAZzaAeAA\nRALAAXyNAgAH8DUKABzA1ygAcABfowDAAXyNAgAH8DUKABzA1ygAcABfowDAAXyNAgAH8MkG\nAByASAA40CvSeX/9YaRhf3YaD5MCgPnTKdJp+H0XqeojQsU3nSASWBidIm3Dx/VYdN5X/UAS\nRAJrw+GTDekDpZ/9i4AQCSwMh8/aXTkbRPoeIBJYK50i7cP2+huy39uwL3c878L2dimFUzuw\nNly+j2T+ZMO/EK5v3UIksDa630e6/Ybs1vpJu9M27M4QCayOyd+Q/QzDASKBtTH9JxuOm/Lf\n7oJIYGG84iNCHxAJrI1ekT43+AORAPSK9Im/tArAxeEN2cZvxuINWbAqvD4iVB2EdBztD4kD\nMD6dIu2C7/cnmBQALIDur1HcPiLkDUQCC6P71A43GwCYWKTvz/sfHdrtC4cxiAQWxoRvyJ43\nkXb6h1whElgYE4q0D8O/4+3R6TDoX7uASGBheIn0Xf6q+RCOj8fHMFSnAGC+9Iq0t18jBfsb\nUBAJLIzub8j+cSj2wxEJrJfujwj9u2zD6bQN5beTfq6RDvc/2oVrJLA2HD4i9PlzNDpavmu+\nje7abdRPREAksDAcRDpcP7hqex9pf3sfadh94n0ksC66P2v373IKm8s3PtkA3ppOkW5/fuF2\nyoYfYwbvTPc3ZK/PPoLlz9q1pgBg/uBnXQBwACIB4AB+HwkAB17y+0hVKQBYAC/5faSqFAAs\ngAl/H6kxBQALYMLfR2pMAcACmPL3kdpSALAApv19pKYUAMyfiX8fqSkFALMHb8gC4ABEAsAB\niASAAxAJAAcgEgAOQCQAHIBIADgAkQBwoFekr83lctqEje+vJEEksDA8/vjJcP2MkKtJEAks\njO7vI/27HMPm8s/3w3YQCSwMh+8jHa+f/MbXKMBb4yDS7voH9CESeGu6T+2Oh+sPS+DUDrw3\n/TcbQvi8HpDKP+vSmAKABdB9+/v+Ay2bf07jYVIAMH/whiwADkAkAByASAA40CFSSHnxqAB4\nJRAJAAdwageAAxAJAAcgEgAOQCQAHIBIADgAkQBwACIB4ABEAsABiASAAxAJAAcgEgAOQCQA\nHIBIADgAkQBwACIB4ABEAsABiASAAxAJAAcgEgAOQCQAHIBIADgAkQBwYEqRzh8hbH//2L7+\n57sgElgYE4p0vv1EZtjdg0AksCYmFGkfvn5s+hpuv6QEkcCqmFCk4d7xNGxOEAmsjAlF+nPn\nvN1CJLAyJhRpE85/j7YQCayLCUX6Ch+/j05hC5HAqpjy9vf+Yc+h8OsVEAksjEnfkD3u/h6d\nPiASWBP4ZIOB7PiZPHX+ZShD+ilTAysQqUj2W2rJ76qZf2StveizFI9nPr/vBhmdgEhFsp8l\nrBep70cNuXzX5x4idcaAhU9eJdJybjakv+/JSpVPJj/3qxKJnkfSw2GoDarP7QWdV8d8RHKr\nDmeY2tVFyhbVTYs0TRYIw+ieWk/vjvxrAqd2pVoQRQomkZii1zJyjd1FCvnxrSFGp4Wr491F\nMhSDbJLmEb2qSWUo5/qt97h9MXkUhzyiU4ZIfkCkuMK1Jg4i0bVxL+GAk3RgljPDJumk4eUj\nbdhy/Ezej0lF+v7c3Tb9bv89VopKSF2ydcAVeFrtbEjSV7xeUkPH7ZnlzICfi2mDpFe0WghV\n3naGhe/AhCKdN1F9bEdJUQ1f40x6pdSDUaQsij06E0+9N8NMisv09zxZUb/tDAvfgQlF2ofh\n3/H26HQYwn6MFHEI094Ua1auzvB32BIqmS38QDzijxIa6dykWJc6kdg+hg33GIeU+t2YUKQh\nHB+Pj2EYI8UzgHV3kpJTildcpInEnD6RoQnJyVgC1ZGb53MxbVDsw6200x1gsUwoUlZuY6SI\nwht3Jyk5qUaZEmELT+7IxP19lK1JRkSGyAT8DcXMngxPKnVh1rV0B1gs6zwi1ezPZyHKBR81\nYPPoqaXIzCOlmdIoH9zzmTg+eZMJ67nNxrSzdl8d014jHU63R6NfI9nrgXahVZo0YHtxUdpX\n3z8AAB9gSURBVJ5LhNhWkZR7449ESc58DZ1pef72raVvkLdhytvf22jzb85ay7FEUnexUNGF\n3oHXiLn5bYL24IKLadTZF0iyl9tKLavzroMpRbp872/vIw27z7HfR2J3slQjgdiSl3ZdVq7C\nTXA92OD5eu1ZHbae/AjfmklFmi4Fu4/5Hc8tFSvWkvTeOORCWhBuJKQDZdZnrc0jFudgnylE\nurNSkSre4+DLgS3zcsq/hgVhTORTeI4ikJOwdIRdFW7pWLVZ3oPVisQEZfe8Ug8hyB/qUcLn\n3ZphB8TfFsh7jF3f5q3yNkAkvSCymlWK1cUdqgh1gm11iR+PD82frpxsILPhDUXil1pEilvm\nvZpMKaxm0oj3qdVZjECQbsNMPZCZ8PYilXa8UOCqSMarpYJ9wui4VnnDVxZyYXuulfcRiZRe\nurzYLS1eqZBTQ8QIj7WyR+lqZjzJQYEZm89mq4WO+D2ASMXXb7bElQqPnkgBnk3bRYrTsGNT\n5mPeaPXIm3ndvL1I5V3OlrhY4Zfkbp8U4bHaupQOhn8YSqXMrZPatsBm90wwTsBu3kck7Z1X\nU8fEjqRekv8x9c/ZIjgjdCBjkQNWi6S1bmD0BPO8DINI5V3CVvXfY1K2rAFy3TONlQNSt0jM\nSqV1CzSccwKINFmKrPLYx38LTPvk0SzIagjLpebPgNYYoXifLz2esbOmM6ZLihQaZ6sbEhTT\n+wb0YIUixVu5sMWtu4Rtl3uQNWWGUSRPd18WSgZl0+TGI8+ELrFtDGtzv7onW7M3oCcQybJL\npHaMBbTKuYUCbFxDt3x0NBS7daq2gRzB1r637p9BnAL6sj6R4s1c3ORyA7nYHqFZkbhDklz6\nTOMshtAyvmdHhqtNkM5X2gLVm6zQI1lg7sxFqR3AFEAktkG+lAlJSjsNyA7juUwOUkjz17Aw\nMWWCeS9jpNImK/ZozMekbYswLm8uktQiX8iEjMuZXfkMQRcyDUkv5drouVqclT4/Ols9THmL\nqYRM5MruNG19/7FZn0g110hSWTBLJSeS6lZqvtiQ9mK6lqZDJ23oIjSRej6Wl0eiZKvt29ht\nMiAS10Tea6Sm4/IWSp5zoYScUZ0OnbShi9DEtgkKYxGztR6SqvNNxApFSl8qS9ue3T98PSdr\nuOpmC0zuocEMMBiPAnkM4xYwbINoTXthy3FN/arzdWFPuEqRqlKx+0ew4aJ4cRlJpOpTqNpK\nFdpLYbSR1uSr7jf9hVHNSN9eJH7/CDZcRC/iVklfqzm/fZR9Z96llSUutJfCsDOvobXf9NSM\nFCJJQ2CvU55rRhFJqc7niuKura1UoT2/2DBUS8KGXpNTNcX3EqllD4bA/t1EtpycRBKGEaKh\nFEZcVeFCe36xZawrASJJURt2vVI2gZ6KcS1dNKqs34ZZWhdDJKFxQ/z6LvNIYd0ssgbFmB4i\nxcHkIA3z9+GZf90aXXCNJAW11WDSyFC9gjTRuoI2IWSfdCiPo0Ulr7J/tcgT8s4iKfPuEslW\nh2mZR11p+fPaCCJxHtUVs2f5v4lGV+xTXZZIpXmp5WKrv7SVVrR0YVbjnEhSW5KJi2QQSZqe\np0iAYUkilYtBb2EqJqmehbdZLlzLkJ7RMUGoES0iWW605WMB47AqkQrlYqomqZ6zRfkKpsCf\nz4pSRO35cWh5tC3EjgCMwIJEKhdDUrpKAz131iarW86DvMDpeOlC3Yh8HHyewHXgBg+RRmed\nIgntTOWUFzB/B0GFi3RJ3ncSeojjeKoo9WLDPRcYJg56gEhsGC6cJgKb9vnc3ocbhzhJce50\nwqaJg3YWJFKpGMrlWSjcUjiLSLS/RT7DcOi4/p7R0bITVtJUjgAwrFAk5cBjqNzHSr7erUZk\nJWztZuLZQUhRNWF1s1aP7W1Zkkj6fn3WjFw9fKnRBpe0NONaSxYSR5TTQpnarcPMhgmXLS+G\nqVoDcpYlktqHlJXcyPA8K/WHJNmyZKF0hChRNUkyXzZUNiLTCxBJUj28NwYiMeXHqEDPnuKu\nWdla7i7k4SqmmI2VDUTSWLZblqRydEa8482EZYmk7oSssi0hhCJiroZ0kfITKcM9ifz5b8bS\nTYGkMRc3n0i2Qgw6kUj+EWfCkkQq7ISGfSQUke2WddI1pCJJdS624O675ZNJ1rKdmaZ0lPp2\n40ZUs0kL5NttNaxIpIazhnyv0kLOVaAk/fL6zVuKDahIdLZiSqNINCKzWfUh9JJPcj0sSCSX\nHZB1FwqQSSoitGKXyeFEscQNkD678E0vjEiBbMHAKugkUhA3Z1/cmfFeIpEA+QKxekUHkpGl\ny8vL5PDsbLmUdIbJWppKi0qe0gSV8MH5sS/6PsTaRVJrhjYQL0y0YxO/ynRIqhAprT9+U9DO\nUmp5o/CRW8mjSYNwTzw1CxKpYUtLtSg11peHKniR2CsvGp07c2Mb8/NlBiEeDdglFdtYhQTn\nx5AMfpksWaTyVn/24IpTDa6GCwYkkaJAyfCERVnqoJ3ZkdxpGHYoPpeehS3GXyYVmy6JJYnE\nXhWrYSTktqZwFpEuSfHL1ZSFI7JyM2kRiRlf3mkE1HFams4LZXTLEilpU97spHjkPpbdGLfJ\nQxMudpGS1dHjqIcYWxvgcyl736Ni5u0IsflFI47DA3V8ixXJst3ZcuW7lMMJFZ/EzVNYij6N\nS2KxrQ03G/gVTMfyzHuoiD3mMFxQB7hqkZjzGalDOZxQ8Gn8JMzzf1xorrsoEnOnIP0f00qa\nXT6S0sQ7MYceeRzd6CXybiKVmppCMZFFWkXKeumLk6jcTRl+CLaZT8c8RiGhbsTlimTc/0l5\nVoTT697uUa1I8tWM3poLzA1c2UhAR9+Iaxepoi0tWibE7wrGAgkaLUjHNJIn7satoK0uqkim\njQUE1I24XJHqXkjrqkipzvsKzRxS72m0fHFe6MwiZgQkBDNU2q1iEwCCuhEnFen7c3cbym7/\nPVYKF7Tq/F3MOsPy25uGltvShWxG6eonWcI3AU0oG3FCkc6baKduR0nhBFN6eWlrNc1pQOPQ\nIMJ9dXYIJALNIc4FjMCEIu3D8O94e3Q6DGE/RgonlOp81G98fyx9qImUrBHaSiG4xXKaPKXv\nFgI5E4o0hOPj8TEMY6SoQSsupvi46k2rmqtyIbBAMjS6mL184k3S52LeDsDMhCJlL/BjpLCj\nlVfpZV50QOjIRCbduWziMNLlFpW0dWpfYGRNR6SagpALqMmGx9le2jWPIoukZNFGGKJjlzxc\naSYXiOTGtNdIh9Ptkd81Ulq09pIoV7Chr4wQqcIj7e2fiiil2ah9K3lvH6e8/b2Ndvrm7JBC\nL1tDT/3lvtS35AUJVO7CtzUOQBls/XaoQh/DWzDt+0j72/tIw+7T532keOfV7UpDldLlTGet\nommkYhdhiEaTtHmWtoOw3nSYKYzhPZhUJOcUyd6r3JXlIuWWZl2ZWg6Mb79P9K7KCB933LlV\nehUXt4q+utCZG06h7WpZmEjJnsortWZPyrueXRovVGxI7npHC/8epG2zJcoYyaC0/EIIdTs8\nnhi2hZyj3Ha9LEqkbF+lO692T0q7nlmalElaNaSKou7cwrzm9BLM03BD0stYi/5oIU283JkO\nRW26Yl4lUtP7SPm+Sp7X7kqpPbM0KZO8brIqirrTJkxitQZDjrxGiKEHL0xc6SwNUm25ZuYj\nUnl/kLXJ0+p9yTdnwqQjY0o3ekwf0pjJ08KkmWz8CimGebk2cdOGNLRcMUs6teN3tbI2bSCE\nMyWJK4WUbtyFLqYx9af8pMiokjRKyZuXcyGkzsIme1+NFi8Ss7qwKFttSZKVMqntuIu6NA+X\njzKk/pHUZFjkNoZ1lsw0mQzcubR5Q74ZSxKp4AWzVu+ghlEOSaJZSUqmA5+AyvJYKS2nM6yd\nqTzLVItAVTPneCsmFan3i32lPZmvk6pPz12oZyb4cw1dmCxmRsV0eK4NFGGwtUcFJp6WJ25Q\nDFwxjPUwoUgeX+zrrpaqblyFk2bCI16vEB3AsiicMfkyyxVd3STzZWTF86ElV+tols+EIk3/\nxb7WMstKV6jlTBqSkFrAxHk+01cLM2mdodyTzW/O1Tqa5TOhSC/4Yl/jfs1qV6xlEjprQ+WR\nTeGzFFNHK2onyovBDk9pbwj5FkwoUnY/aIwUNGXLbiXVK5az2FGKRcLoa+8N6DSeC/IxtkyU\nnwN5wqfgxgWRxupy4yVfNW/ZqaSkhRqXO0qhmDDM4qhw2csW7ayyaZ7cUqYF0zpbVN5Eq2Xa\nayT3L/aNAVfvJpGYFkWP1JO358I0q9a6ZaLcUrYFiS/Mt2IQa2FCkfy/2DcOpdIudxRjsXH5\nZBd+aTFRy0Tp8qyJOUKvSMuVcEqR3L/YNw7P6sh2a5VIIb3xLWpEe15oP6F179VJuUutSH0q\n9Gr4SiYVaU4ptOzi/izs5WfHSEWLStlCW9s6y+smmrQpBvCqfYjkzVxFMndUBGJrLxKP6800\njZp3DLq3bMfwaJkmQSQ2f+vODNltY6tJPS27B90BRPpjnSJpO2OKHVUtUvJ+UaNIdAwjzW6s\nNBDJnfHON6bZVaIRqgBW4SwmuUo2VW0v2COIZI1oypmn0RDHxrZ7dugXqXIbTFfeEMmbrhRa\ngaXrAn2vUwloSSneLugTKRt0WSRt3XxFwvtI3kwikliL+YJyJTGlLRskB3uuzVqmnQyhrNug\nTGHI4BeIlDcli8qVxIrC6cOEYjNF/ZkRiLHqt4FOOQ/4Y4UipW/nCOvUyn4uSZqWEqbxmIU0\nkvBcFclU3sra8nz4WRWavzlrFokrAGtpcy6UEtohQ5VCZQtpk9I2qF13eb76WGcPrqxRpL9a\n4CsgsG/YxKtL1c/mq4DpJ8XhWhlHJa9Tez5WVkzfmnjNrFOkexCxBEy1rbQxBGQ6M2Hy2Fw/\nOgbrqNTxFqbSfIHUM6wlA5HyJooKxWSyUX+Nov9lQ5T60UUVw7JtJm728k1Ny4aASON0eUkK\nuQzyQpVWtlRssW+8SNGFtYYuZAegPM2X5mHSDVM594u20VfOikWSbjYIa0g/UQVbWqlrvJSt\n2rSvMBhpXNni4hDyBlzmirlDpHG7vCYF3aV/S8o7WyheoWmhN7smfkLlo+VMB8NNIVvMt+JG\nwPXQNxELN+e3uP2wZpHE07aLWIekKVPabDu1M7v8oook3oUWJxCYowhb1voRV5hmlqe43Upb\naHWsW6Q87LNsLPXCIrTTu3JrlBzaUTQdv3i/j0y4MLV0dXGz1DQodlkFbyoSX2FCa7HkLlyp\nKn3IIiGFdl0XsmdC1nsvMZ4+qfJGKTTRt9AqgUiG5krN0cVRS63alQxJByFX1pL018asHDLt\nW9BMQ5cl8lYisZfwhvZKzZHFyQJpnX78km5N/y5LroaEMHUiFTaBNt0yDV2WyJuKRA8Oeo+4\nMVtatEO+MipvLX7UwGQBvzB5VJqWaYMxszNS32OJvJdI/L2wUge21LNAtLmcj7uEiP9PMysD\n0ocpSCRH52dPs6vbrBBjlaxEpIYdlVeSEEKqUKGjtE6q28dCupIXQlBFHSGT0KZEHCZk/mn9\naJiK1stkFSLV79q4W1xWjxVpM9v9tciHpOouSv1FSwMvZNZNHECSXd4k/IxLW4iZitrx/YBI\npCrjUlMbp2m1SMUDUkivZuQsskjplpA2SdyNWy+1ZqZS2rzvxRpEat61fL0aCzhvoQSSrsjE\nYHIWq0iSJny30haKu1VGeBcgUl6WUp0yjeMm2iJeJGuspJuy6tHAcVuRCUAkgbcWiX19F8uU\nLXy2+PML8wqRgrBG66ROr3dj5VN5l7twtaxBpJ5dS2tSLFO+8IN8FNGvkZRgUo4sEB2iYbOU\nOpVjNfVfPQsXKXS/RtKK5UtYbiUEShpQIzSRuMNb3u1SdoJdW+pkiVXd+w1YtEhJTbWmombQ\nwr5ktZ0uog2KjcX27Dp+ziUnhNVqH31DVfd5BS8a50pE6kpGyjJEYv024CvboBI/1ijkRWsu\nqEQaKVvHS6Rl8LLZLVkkuVKacnLFfOH1kHowzuVtL0JI6RROKwy9hbiKj+dffq8oaIjUkIKt\nlNYtKZW+/S41FyqPll6uR+Hl0ajzURvUbAr/AnxJSZe214iZJ+kyTgp2q8VFWpVSLH0OvgHb\nkY5VesyPhi8MNjA/pZrpm9ra8I9oTwqR6lIwG00vv3IsSQemrvml0hD4jsKel0JrwYRJVc7e\n1vwlEeeb9ZZ5ki4jpWA2mlZ+pmCiSOREjFY75xcz1iwoM1Q2MD/x+rlybSujGBo27YN+XpL0\nlniSLqOlINtML0AtDl/42XLuSdaMSELHGrfnBsqFZdeTIZlnKs2/PURPRD9ekvSWeJIuE6bQ\nCtDSJ18k61A4+yt/lSFulLRLljERki6VpSNsmZootravKulX5LxAJL4PeZa15Dtm/miD4JNx\na9XxPp7a5imEDpr0thByM/PI7LxIFZ3ViVRdF9zpoFgr7GJOJ+FYIwSqFMk0PTUvTTbCjYnR\nNBpJ0C5WKNItQIdIyrGEXUG6/1YQI5KgTt6wUKqlSg7KK4kskhmzSGRQLkCkaVNUbGurSIks\n+UKmU96c2EKyZ/30oVpmo6wWnhtoKGW/6m8Y7ySsVqSqdBaRhAJ9LpBWPRZG4YXUfD+2sXEy\n4lpxQZmGSvYr/obxTgJEusg3G8RG3FKmU9qFrfDCMmWcprk8zjPJanGBhdo6bkoyeihXINI9\nYVTpF1YkoYiTpVrlFw8+XHGXLsb4mTCQtaRD3faqxLP6pxhvAysUqWMrh/iokIYyFDcb77mW\nq22+u59IhvuBXDDvQoVITl0mTOG4nZNQUi0UKjsv/rSx2NUkUsMhSYp7yV5EvEu1MqTedIYa\nQSR7qFyrdLGYVBMp7xnIoUOPpY46JKeTf+GlnvHyV4s0Qv7xWZtIepFVRCGh0oqMUtgKm0qQ\nKyFlk0Npo896lO5U0HH4n91VNIVIPlhT0M1Nq78hu1CvUpXrhSeIRP9Gl5SNTrii0p5No7T5\nZNLNNo5Idl6dv40pRTp/hLA9/AZRo9hScNtbK00rkkhsDlKE5XgspH74xOLE9bnkg6aLkzm0\nbTcnXp2/jQlFOg+3zbO7BxlJJIeTfVrjUoP8ab5YiakgDT1bbZrIs6kQnkvettncELf7rJlQ\npH34+rHpa9jegvSLxG9wWhC1e4Sv7As5IKgi5ZmzJoHcDecRZlYxDTIrdopJvspE7rw6fxMT\nijTcO56GzWlEkeilTLdIpLz4GuUuNPKg8SOxmklq23T5abDPS+mkmNPAz3PmTCjS36Y5b7ej\nimRvUOqX9I8eZ2H/nkbteRHSR2I1M5UtzkacIWnOzEVO9mLmNBYjE4q0Cee/R9vxrpHKDYp7\niS2wtNqysCG/SrfVZtkjR5GSidOBLq9058WEIn2Fj99Hp7B9mUimqmGKmbNKztcmEquWZTb0\nOq4wAj6hvlGAxoQiXfaPXXUo7DVrCosRdImlZkwiKXcMWzySrpukkdM2zICV5CRKaaO8A62b\nYUqRLsfd36PTh4tI9ViLRtLotoSp8mc3e55KkbKdzLWpEImNIzUoBlkL7a8ok4o0gxTVIiVP\ntCpnL5HUTEyo5FmxvgWPoqd124Rrnm4GUzwbs/QTIpkjG0Wy3S2OAvEKMJmiJSSS8IwZm3ZA\nar9NqW2H9hpTUjpF86J1A17eTyTznbyo3XPzBoa0vdKCpk8C50FKV2JMguY6kLuRudeGrs74\nSpo34OtE8rnZ0JSYbil+6/H1LlQxt4bbK0LpX2gUq0jMcpeNIs2rOriS0PCCNikdk3w7kRru\n5D03r2QJs5wvvWyZ9DQo99DF4F21LvVSZ9OOOjWXDG10bMBJuswwRZytVCNMbad98kVi5WUL\nlTBpXi5EttK30sncfcPz0UaZQBXtI4BI5RKUDGF84JrJqXhvSFxpNOmqkUWa4BpppBnU0Zp+\nPiKJxTc6paxStfN33/SYfKP8pkbcXBpO7SyaSUfnF3SeIrUyqUjfn7vbhtrtv8dK0URh/+U7\nWBapHFIQSQpTM9rRq9A3uPgiA5EKnDfRK/B2lBSNlPZfvt6kjRSQCaSeyhVC2Gcxf5Y8gwlF\n2ofh3/H26HQYwn6MFM3ou69JJGNa5shWHSbq39x3DkAkE0M4Ph4fwzBGitEoHm5a93/Wb6ll\n5MVy5z+hSNk9pjFSAPAicEQCwIFpr5EOp9uj+V0jAdDHlLe/t9Fdu81ZawmRwMKY9n2k/e19\npGH3Oa/3kQDoZT6fbJg4BQCeQCQAHHiFSOX3CiASWBgQCQAHIBIADkAkAByASAA4AJEAcAC3\nvwFwACIB4ABEAsABiASAAxAJAAdmKhIAC6Ohyv3F8WC0YY033wVGxpD9IkOkN46MIftFhkhv\nHBlD9osMkd44MobsFxkivXFkDNkvMkR648gYsl9kiPTGkTFkv8gQ6Y0jY8h+kSHSG0fGkP0i\nQ6Q3jowh+0WGSG8cGUP2iwyR3jgyhuwXeaYiAbAsIBIADkAkAByASAA4AJEAcAAiAeAARALA\nAYgEgAMQCQAHIBIADkAkAByASAA4AJEAcAAiAeAARALAAYgEgANzEmk/hGF/ThZ9uYyPBP7a\nkEw+kc8fIXwcHQJzG+Py7bE18sDNfza+GPlyvG6Nk3vgjj91X4h8OXOb3cSMRNreNs4mXnR0\n2cEk8P62YOg3iUQebgscTGI2xuU8OGyNPPDRrSjJkA9OmzkP/OfR0BuYRD7d99/QIP98RPoO\nw/FyHML3c9HPM4fxkcDH8HG+Huw+3CPvrzH3YdcbmNsYl8vOYWswG6N/tHzky/Cz4LwLe/fA\nNw75AofIH7fB7lsqYz4i7cPh599/4fOx5CtsXQ7feeDdPWh/bBJ5CGeXwMzGuD4bI/BXksMz\n8r9bVZ67DxzctrgenvtfAEjk0F4Z8xFpF64H1PgF8mdHeIhEA/+G744tRHY452Ain1xeVkjg\nr/DVHZSP/OFxissF/l3af2ZOIv+eO7fswPmIRF8Nji4v7tLLzDlsx4m8dyhNJvI2nBy2Bgm8\nC4ePn+vr7sA08iZcPofbWbRz4CvH7jNGLvLn76ldw0F6ziKRZ46Bry/FhzEi/5yAjbGLf/bx\nP4+twYh0o/tFhUYOYedyS4Ddfx4HJCby1/Vuw9DyQvi2Ip0czrG5yF+7weGqg0S+nX+MIVL4\nEfRydjiKMpGvNxs+urcGt5WP/XeK2Mift1eVlhG/q0jnof81WDrWfYxQlZvrXeQxRLpzzu60\ne0S+vw1w6o7MDXnffzrBRf66nk2cm/bffEQaxhKJDbztLhwpsseNKhL541Y4DltDGLJDaBK5\n4xaYHvi5zDvy5nbC2PSqMh+R7rdQTunNGb+7dkng02br8IY7P+SL3/3AZ+SeH65XA//hP2S3\ndxmYITu9+UUir+L29+ftRfeQXqp7iEQDHxyurdnI9/eR+s9mSGQ3kcQhd1cmiXxfcOre1kxh\nON2zZzbG9d+mM4r5iMS+gT3KJxv6960U+fae+HnXv5v5d/PH+GTD/nZZ4HDNwWzmzfl6wfHP\nO/D1UOLyFhW3Mc6/m6SW+Yj0c4L6uA/7qBiXz9rlgT/cPvRIhjw43UtmN4bL1sgDn+9Ddrhl\nT4b86bQx6LbYeNz85iJvm4c8I5Hun7y9PfQVKQ/s9+lhOuSfBRuP0w5uY7hsDRL4PN6QD1uX\nt3ppYJeyYCM/F1QyI5EAWC4QCQAHIBIADkAkAByASAA4AJEAcAAiAeAARALAAYgEgAMQCQAH\nIBIADkAkAByASAA4AJEAcAAiAeAARALAAYgEgAMQCQAHIBIADkAkAByASAA4AJEAcAAiAeAA\nRALAAYgEgAMQCQAHIBIADkAkAByASAA4AJEAcAAiAeAARALAAYgEgAMQaQ5EvyF4++W4+w+6\nCj/rqv9cndeP2YE6sNnnQCbS5rZXNsK+gUhzBJt9DmTVf38qKQGR5gg2+xyASIsHm30ORNX/\n8/B+kvf43fWvTRh+f3V8P4T9o/E5bG7/34Tz5bALv7/G/bjOuuS9D9sQtsJ1F+gFIs0BTaTd\n7cH2um57fbR7NN6G08+/p591n/crrP2FiPTs/XVv8zXpxN4HiDQHnvcayBHlELbny3kbfg4l\n/8JwvByHh0j/wufPv58/q0L4d31K+0e9h3C8ttm8Yn5vAESaA4pIu58Tt+tp3O768PtydeOx\nz25WPG/uMf2j3iHgtG5EINIcSE/tEpHyt5jixh8/53an2wnd5XT43DIiRb33P2eFx+NkU3o3\nINIcaBTp++fcbn87TG2FI1rU+/L5c1IYhtNkk3ovINIcUEXKW0VLhs31v+uhafN1OLEixWkO\n+w2ukUYCIs0BRaTd89Lm/vA7arwPX7cbDrdFmUjf92uk7MIIbzONBLbrHCAinS5//95u1V2+\nrrcLDtldu5s7t7sJ4ef87vi8RtqEr+utupD03tzv7OGINA4QaQ5kIm1+rmX+/v29/Lld29ze\nFPqIDyqb+xtM+98roe97/6/n+03P3v8eTcAIQKQ5kIn0vbkqdP/39tmE8HG/R/AZf7Lhyr/f\nU7cfu7bfh/td7nu7j+cnG3573z7ZAI9GAiIB4ABEAsABiASAAxAJAAcgEgAOQCQAHIBIADgA\nkQBwACIB4ABEAsABiASAAxAJAAcgEgAOQCQAHIBIADgAkQBwACIB4ABEAsABiASAAxAJAAcg\nEgAOQCQAHIBIADgAkQBwACIB4ABEAsABiASAAxAJAAcgEgAOQCQAHIBIADjwH7JcwKo80bE8\nAAAAAElFTkSuQmCC",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "plot(jitter(mod.can.lg$y)~mod.can.lg$fitted.values,pch=16,xlab=\"Fitted values\", ylab=\"Is canceled (jittered)\")\n",
    "\n",
    "threshold = 0.5;\n",
    "TP= sum( (mod.can.lg$fitted.values>threshold) & mod.can.lg$y) ;\n",
    "TN= sum((mod.can.lg$fitted.values<threshold) & (!mod.can.lg$y));\n",
    "\n",
    "\n",
    "(TPR=TP/sum( mod.can.lg$y))\n",
    "(FPR=1- TN/sum(!mod.can.lg$y))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### 6.2.3 Categorical outcome\n",
    "\n",
    "If we observe categorical outcome, i.e., $y\\in\\{1,\\ldots, K\\}$. We can generalize the idea in the previous section by considering $K$ functions $f_1,\\ldots, f_k$  and set the probabilty of $y = j$ as $f_j(x)/\\sum_{k=1}^K f_k(x)$. \n",
    "\n",
    "We thus have the negative-log-likelihood\n",
    "$$\n",
    "L( y,  f_1(x),\\ldots, f_K(x)) = -\\log\\left( f_{ y } (x)\\right) + \\operatorname{log}\\left(\\sum_{k=1}^K  f_k(x)\\right)\n",
    "$$\n",
    "In this case, it is more traditional to model $g(x) = \\log(f(x))$ which lives in the whole real line. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Classification.** With categorical outcomes, a prediction problem is also known as _classification_. To be specific, we might be interested in classifying observations into different categories. Naturally, the general prediction framework and prediction methods are applicable on the classification problems.  Other notable methods for classification include \n",
    "- Support vector machine.\n",
    "- Linear discriminal analysis.\n",
    "- k-nearest neighbours. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 6.3 Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3.1 Meta-feature generation\n",
    "\t\n",
    "We can create a nonlinear model with even only one single feature $x$. \n",
    "\n",
    "- <span style=\"color:red\">Local smoothing:</span> For each $x$-value, estimate outcome with an average of observations with nearby $x$-values.\n",
    "\n",
    "- <span style=\"color:blue\"> Meta-feature construction:</span> Rather than just using $x$ as our one feature, we use e.g. $x, x^2, \\ldots, x^5$ as features; and then set he model to be \n",
    "$$\n",
    "    f(x,\\beta) =\\beta_1 x + \\beta_2 x^2 + \\ldots + \\beta_5 x^5\n",
    "$$\n",
    "This is like doing  multiple regression  on $5$ features, where all 5 features are generated by the one $x$. \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "For <span style=\"color:red\">local smoothing</span> we now need a **distance** or **kernel** for multivariate ${\\bf x}$ \n",
    "\t\n",
    "Often use Euclidean distance:\n",
    "\t$$\n",
    "\tD\\big[\\underbrace{\\left( x_{1},x_{2} \\right)}_{\\textrm{observation $1$}}, \\underbrace{\\left( x_{1},x_{2} \\right)}_{\\textrm{observation $2$}}\\big] \\equiv \\sqrt{\\left( x_{1} - x_{1} \\right)^2 + \\left( x_{2}  -  x_{2} \\right)^2}.\n",
    "\t$$\n",
    "The Euclidean distance is equivalent to a Gaussian kernel with equal variance. We can also include weights or choosing different kernels for interesting data-types.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For continuous features various expansions may be used:\n",
    "\n",
    "- Polynomial basis\n",
    "$$\n",
    "\t\t\\begin{pmatrix} \\,\\\\x\\\\ \\, \\end{pmatrix}\n",
    "\t\t\\rightarrow\n",
    "\t\t\\begin{pmatrix}\n",
    "\t\t\\, & \\, & \\, & \\, \\\\ x, & x^2, & \\cdots, &  x^m\\\\ \\, & \\, & \\, & \\,\n",
    "\t\t\\end{pmatrix}\n",
    "$$\n",
    "        \n",
    "- Spline basis\n",
    "$$\n",
    "\t\t\\begin{pmatrix} \\,\\\\x\\\\ \\, \\end{pmatrix}\n",
    "\t\t\\rightarrow\n",
    "\t\t\\begin{pmatrix}\n",
    "\t\t\\, & \\, & \\, & \\, \\\\  x, &  \\left(x - t_1\\right)_{+}, & \\cdots, & \\left(x - t_m\\right)_{+}\\\\ \\, & \\, & \\, & \\,\n",
    "\t\t\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "- Others include higher order spline, Fourier, wavelet, among others.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Same ideas can be employed with multiple features: \n",
    "\n",
    "- Additive: e.g. $( x_1 , x_2 ) \\rightarrow ( x_1,x_1^2,x_1^3, x_2,x_2^2,x_2^3)$\n",
    "- Interacting: e.g. $( x_1 ,x_2 ) \\rightarrow ( x_1,x_1^2 ,  x_2,x_2^2 , x_1 x_2 ,  x_1^2 x_2  , x_1 x_2^2 )$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAM1BMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD/AAD///89ODILAAAACXBIWXMAABJ0\nAAASdAHeZh94AAAd50lEQVR4nO3di1bqSrNA4URZiIro+z/tknAxQIBcqqqrquc3xtm/Z42N\njaxMknQ37uYHwGJN6ScAZEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCA\nkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCA\nkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCA\nkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCA\nkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCA\nkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCA\nkAABhAQIICRAACEBAggJEGAQUgMEM+Molw+nwBCAJEICBBASIICQAAGEBAggJEAAIQECCAkQ\nQEiAANOQPt9W3SLwav2pNQRQhGFIu5fehopXlSGAQgxDWjft+7b76uujbdYaQwCFGIbUNtvz\n19um1RgCKMQwpIsNso93yxISguGMBAiwvUf6+Oq+4h4J2VhOf7/2Zu1edipDAGXYriOtu3Wk\ndvXGOhJyYWeDlMOnjWd95hjxEdIc39/ff//T/fPq4/v0VBtCmu57gMyvw0BYpUKKuo60P/0M\nh3RbE+elivgJKcKb+VBCz05MpZ8zTHBpN1J3J0RIuIOQRnla0IOYuMKrASGNMiUkZh5qREhj\nTOuIkCpESE8NhHKM5fA/38c5u56f4xUdJVWDkJ54esY53AJdhXSMiZaqYfp5pNFHlaND7jKQ\nn9sLt8F/8d55qeRPAk2GIW0ihnSVxt7VuejJvx1ifQyLWV7abdvHv/JEYAhx12E8ruL6lHT9\nCEpKy/Qeafv443wSQ0h7EMW9f//qMc/iQwq2kw2b3qfNlYYQ9N3bzHD4k3Enl9vLux9Cyo5Z\nu3ueXKWNfuhP/6HscsiKkO7pT9Qd/2jseYUph/oQ0h3XF2d740sYvrr7IaS0COmOoQ4mhTBQ\nIqekvAjpjsVz2H/f4Pb6TutJoxhCGtKbxD78wTmACRXczlYQUl6EdOvBdN3sb3PxffjlKPkQ\n0q2bS7K5p5Lv69XZGVeIiIGQbsjOEtxOg0+bs0AMhHTjwcS3zHfjlJQQId0QXgC6WyUhZUJI\ntwbn62Yf92J3XPCMkG4NT3wv+n7i3xPOENKF7/4v9e6eidDZg5CSI6Se29sZqZCG7rtYTcqE\nkHrsQup962XfGE4Q0p+ho13ocCek7Ajpj2JITIJnR0h/hm9kZO5kmARPjpB6VKfWrifBCSkV\nQupRn6NmEjwtQrqge+V1ee1ISJkQ0r2noB6S7E0YyiKko6vJOpOQ+gMJDoMCCKljdIAPDENI\nORBSp1xITN7lQEh7g9dcKvcu31q7YlEWIe3dmQVQHu40FCElQEh7xvPSLCflQ0gd00Ob5aSE\nCKljerFlfCEJC4R0ZHj7Pzy1gdAI6XZw9dv/wY44L4VGSDd7GownG4zGhK7qQypzUNvHC12E\nVPAyi2XZPGoPqeCNP8uymRDS9RKS9dCncQkpNELqHc+mBzPLsqnUHtLQecF0YELKgZBuTkhG\nhzP7G1KpPqTe3JnxjQrbGzIhpN6oHkLixBQTIfWHNb5RGcyIW6WQCKk/bPHDuPwzwDyEdDlw\niYOYDQ4J1BySjzt91mVTqDckL5NmhJQCIZ2GLHT0si6bQ7UheTmAvTwPLENIhwGdhMQ6UlSE\n1I1X8N7Ey70aFqk2JDc3+YSUAiF14xWdLSOjBOoN6eIA9naT7+m5YIyaQ+oPWD4kx1njOUI6\nDVk8o787JUKKh5B88HPHhlkIyQU3k/GYqc6Q3M2TEVJ0NYbkZJddn5d9Fpir+pCcHLQenxMm\nqDAkl+/+Ds+SmKL2kPzcj7i7b8MUhOQlJIRWYUis2UBe7SF5uUc6u7jC8/TE8FCNITne1+a6\ncTxQZ0iXgzk6VgkpKkLyhHmQsAjJE0IKi5A8IaSwCMkV7pGiqjkkhwcpIUVVb0hOD1PWkWKq\nLCS3C0gIrqqQ2BsELYSkMxQqU1NIgSaX+UxFNNWG5PkeiU/5xUNIDjEJHk9NIUV5pw8TPP7U\nHJJXge7lcFJVSEFu4gkpoMpCioFp+ngIySEmG+KpMyT3ByZbmaKpMaRoh2ak51otQgIEVBgS\nt++QR0iehZitxx4h+RVk/Rh71YQUcB6MkAKpJKSIKzOXW+6YvPOtxpCCHJPsXY2kjpCu39xD\nIKRICMkvttwFQkh+EVIgdYQUdQLsYqaBkDwjpCDoyLdKQoq/SYCQfKsmpOM3DnwoRn7u+VUV\nUtB39egn0yoQknfxb++qUFNIMWe+CCkEQnIu5hJYfQjJOUKKoaaQQt4jEVIMhORdxI3rFaoq\npJCHYcSPUlUof0jxr4oCfri3PtlDynWDEXK6pA6EFAkhuZU8pGRzXoTkFiGFQkdeVRNSjsOP\nkLxKHtKppAwH4OG8Gv2nyIqQgsh1iZqPfUibl6ZZfagOceF4XRe9JELyzTCkw2H8ejii1ypD\n3B86ekjZZk3SsQ5p3ax3Pz9f62ajMcT9oQkJqqxDapvd/utd86IxxIOxY3dESN5Zh3Q6mB8f\n1IR0g458sw7p3ymkVmOIh6NHzohPU3hnGtLqbfPRvP9+uVs/nm3gIBnAJnDPTEM6//U3TbvT\nGKIWhOSO5TrSdrvZrFbdlMP6YUeE9Fj8Sch8su9sSImQ/CGkgAjJH0IKpptyoCN3SoVkvI6U\nRZo9uOn4Canpkxgi4+cOzqtJqX6qDPJe2l28eQt8Pw/YKOQWIUVCSG6lDel8zGUqiZDcMg3p\n823VHdWr9afWEGcpQ2LrqluGIe1eerMJrypD9BASLBmGtG7a92331ddHq79pNWNHPxl+A3NO\nhiG1zfb89Vb/YxRJQ4JP5r+zYej/ERviUsJ1pCuZf7Zg8p6R8mN/gyO290gfX91XJvdI+RGS\nI5bT36+9WbsXPti3FHvAPbFdR1p360jt6k1/HSm5v52rhORC2p0NqbEJ3B1CioipfXcIKaDL\nXRulnw32MoeU9iDrh1T6ueAgb0iJbyDYBO4PIUVER+6kDSn15DAhuUNIMZGRM4QECEgZ0je/\n+w3GEobEsj/s5Q0p8ToS/MkXUn2LLLxhOEBIke1/Si5hXSCkuNi66ki+kOpZrWRWxRFCCivn\nb+6LKmFItSz7E5InKUOqA59K8oSQ4uKE5EjKkCo5rk43g5X8uL4lDKmiWaw6bgZDICRAQL6Q\n+PwECiAkQAAhAQLyhcQ9EgogJEBAwpCqWUc6OU6CV/ZTO5MypKqwB9wFQoqOkFwgpOCu9oDz\nyhWSLKT69swQkg+pQqrlI319hOQDIUVHRy5kCqmeX3vSR0guEFJ8rCM5QEiAgEwh1XmPBBcI\nCRCQKaTfm4Rv/gPFKCJZSExdoQxCSoOL2pIyhVT1J5G4PyyLkJIgpLJShVTxoiRraIUlC6la\n/b2rpZ9LlQgph8tN4KWfTYUIKYmLvauln0yFCCkJNoGXRUhp/GVESPYIKRNCKoaQUqGjUggp\nFUIqJUtIHD1HvBBl5AiJN2IURkiAgBQhMVn1h+12ZWQJiV1mHXaulpIjpNOyvs6zCYSQSkkR\nEsfPER+mKCZFSBw/R7wQxRBSJrwQxRBSKufXgZkXYylC4h7phA9TlEJIyZzOR4RkK0dIrENe\nYIHaXpaQ0ENI9ggpIUKyR0gZ0ZE5QsqIkMwRUk5kZCx+SBwycCB6SFzEDGM5wBghZcQCtY4H\nr2nwkJjoHURIGh6+qoSUEJt4VRBSbQhJw+NXNXhI3CMNISQNhFQfOlKQOyTWkYYQkobM90i4\n4/AXzpuMJEKqFJe90vKuI+EBQjJESGmxNGApfkjcVN9BSJaih8T81F2EZImQ8qIjQ8FDYg3/\ngeN/o6P006gDIeXFS2MocEj7qxZCeoCXxlDYkI43ABwsd/EmY4mQ0iIkS1FD+pvb5Ui5g5As\nxQ8J99CRIULKi5AMRQ2J1cYxyMgMIQECwobEp9bgSeCQMBZvOfoIKT0ugi0QUnqEZIGQsmOh\nwAQhZdccP03Ba6qKkLI7bUfkNVVFSOmxwcECIWXH3lUThJQdIUl4+vqFDIk75wkIabkRr2DA\nkJjNnYaOFiMkENJyY07q8UJigXEyMlqGkAABhARI4B4JEJA7JK78YSbvOhJzUfDENKTPt1V3\nMlmtPxcPQUhTcTmsyTCk3Uvz53XhEKzXT8SdpS7DkNZN+77tvvr6aJv1siEIaSL+Gy+6DENq\nm+35623TLhuCkKbhU0nKDEO6eDt8/N7IPZI0QlIW9YxESNNwBldme4/08dV9JXCP9MMOskkI\nSZnl9Pdrb9buZacyBO4gJGULQ3p5+5rwyM91t47Urt6WryNhGjrStTCk/bllUkvTh7j4c+Zv\n5yIkAQ+Ov4Uh7d7/abQ0/KxYU1yEjBZ6ePwJ3CN9vr1It0RI8Ec7pF/b9neAzbQnNXEIPoiE\nsh4ffyIhfbyO2D93/axu/6Rn+CGEhHK0Q9q9/Z6OXj52vzWtZjy9MUMc/5CQUJBuSJ/7yYb1\n9jjS9G82Yoi/P6UjFKR5j7SfZtic1lYfb/uZO0TvTwkJBWmG1Kw+Jjxy8Qf7yGg5ZsHnU1xH\nmvA40Q/2YR7WZZVE/WAf5iEkJWE/RoE52LuqJewH+zAHIU028sXijFQVQppo9MsV+IN9mI5P\nnE/kMSQ+2FceIU0z/gxu+wsi+WBfafxWrkmchuRpiEqxO2QSQsI9ZDSFy3skV0MAI5xCevr2\nEzIkpm9h5pjRswvigCGxEgJjhAQsN+YjpfFCYnEexggJ9/D6TUBIGMYrOA33SBjESzgNIWEI\nJ/XJWEfCLUKSFzIkLENI8gipRnQkjpAqxKeS5BFShZrjp5J4meUQUoX4LeryCKlGdCSOkGpE\nSOIIqU5kJIyQgEdGvuUQEnDf6ItgQqoaq7JPEBKeY4PDM+MXCgipYoT0DCHhOfauPtT79UGE\nhAcI6YHTdkTukfAMIT1ASBiNju76e5NhHQnPENJdk8/WhFQx/hsvdxESxmPv6n1Tz9aEVC8+\nlvQAIWEsQnpo2t0jIdWLkAQRUsXoSE6skJirFUVIciKFxLKHuENGvKjLEVL1eFmHTTtXBwqJ\nrWE6eFWHTL3sJaTa8bIOIiRMw8s6ZPLSQKCQuAZRQUhDCAlT8bIOSB0S87QqCGlI4nskaCGj\nW4QEiEi7jgQtbBNajpCqx447CYRUPUKSQEi141NJIgipdoQkgpBqR0g35iwHEFL1zh2xnNSZ\nt0BNSNU7hsQGhyNCwkyn8xEh/czexEtI6LAJ/IiQsAQhHRESliCkE+6RsAQdHZxmXSYeg4SE\nA0I6aI7/jQ5CwkxktDdzgZqQgAvzNnoQEnCBkLAM++2O5rwQhIQDtq4uQkg4IKRFCAkdPk2x\nDCGhcw6JSfBZCAkdPkyxDCHhgJAWISQc9DuipMkICSeEtAAhoafykBZMWhIS+mruaNH8PyGh\nj5AICSJqzWjhkjQhAR1CAgTM/Gjs6dEmD3E4BHBl0f0hIQEHhAQst2wNjZBw6XiTUN/kHSFB\nTsWbwAkJcioOiXskiLnsqLKSCAlSqg5p0Y0hIaGn8pAWICT01XyPtAghoY+QZiIkXKpzHWnx\n7yEjJEDgV/oREkBIwHISv2WWkFA9QgIEEBIggXsk6KpkEpyQoGV/XFW0LMs6EjSwv2EqQsIA\ntq5ORUi4Vd8e8MU/JSHhVm0hCfychIRbte0BJyToqCskiVMvIWHAaWGlhowICYoWL6wEQkjA\nQt0bBvdIwBJysyqEhIqdN9mxjgTMJvH5idO3MnmIwyEwXtq5O0KCncSrSYQEO4lDkvgg0uk7\nmTzE4RAYKfWOu8ghbV6aZvWhOgSk5N+6KrXwbBjS4Rm/Ht4D1ipDQFRle8AXsQ5p3ax3Pz9f\n62ajMQRE1bV1dRnrkNpmt/9617xoDAFJtX2YYhHrkE5XpI+vTAnJg/PkMBk9Zx3Sv1NIrcYQ\nkCS4ypKfaUirt81H8/775W79eLaBvzoX6Gg805DOfy9N0+40hoAoQhrPch1pu91sVqtuymH9\nsCNC8iJzRoefTer+j50NqJL0jCQhYaxUk3eEhDJyLSeJ/+a+UiGxjhQNIT3+hiYPGfgmhBRL\nsi13aUIqPgSmSRYS90gog5CefD+Th4z6tn06Q2CeRuh3v/kSeB3p823VZbJaf2oNAXFsAh/D\nMKTdS++U86oyBBScLxLI6AHDkNZN+77tvvr6aNm0GgWX26MYhtQ22/PXWz5GEQUhjWL+OxuG\n/h+xISCPkEbhjIQn6GgM23ukj6/uK+6RIrkKiSmHQZbT36+9WbsXPtgXx2VG4SfBVU6vtutI\n624dqV29sY4UVPyQlK5U/exsMB4CcyTYKERIKC9+SFqTkCVCev5DEJJThHT3+5o85OobEFJY\n0TvqhST7YxASpggfktYeXELCWLKfOyiFkFBUng0Ox+s62ZIICePkCWkvRUguhsA0yfauEhLK\nSBaS+KwJIWEUQnqMkDBOuk3g8deRXAyBiS5CCrqcpHhCJSSMFf3TFKrXpoSE6WJuuSMkOBMy\nJN3pEkLCdIR0+91NHuJwCCwRsCNCgj8RQ+IeCQ6Fy4iQ4ErkzQ2sI8GJDPuEVE6mhIQp4oek\ndHtHSJjgeuYr4J0SIaG8y5Aizt1pLYEREiYgpHsICVMMdBSrJEKCB0FD0t+4TkiY5mKmIUZI\nFh+lIiTMFqQjkw/3EhJmCxKSya+bICQsECAjQgJEEBIgwWJfEyFhlkgb7ggJTlnMg0nSz56Q\nMIPFykwshITp4m+5E0dImO4ipDAbHFQvQQkJ08UMSfVpEhJmiLN3VX+76nEck4c4HAJLRJls\nsAuekDCL1Tv9MoSESJxmZHkvR0jIy3CanpCQFyHpD4Ea2G1lIiQscLGHzeGdksV21eNIJg9x\nOASWizAJbrVLnZAwW4SQrBAS5oq5UUgJIWEuQuohJMxFSD2EhNlc3yNZzTKchjN5iMMhsJzj\nkOwWkE4DmjzE4RCQ4HYdybxxQkJChnuDTiOaPMThEMjMfh6EkCDMwxUeIZkNAUH+PuVn94m+\n04AmD3E4BMR4nLxjssFqCIgxf/cfxfgsSUhYKMIGB9aR4F6EkPQREhayX7PxiJCwlLfJBuNd\ndsdBTR7icAiIMd/XNuXZmA1r8hCHQ0BQkXPAHYWyJiSoKHVeKnXHRkhQUO5OiZCMh4AmJyEZ\nTsYTEqT83ZmUXE0qtM+CkCDDy0YhQrIdAsK8hFRoLzohQYTP/Q2EhGB8hsQ6EoK5DKnQOlLB\nhWFCgoxCW3O8PANCgozhw9jyxERIJYaAuMGM7G6Vri8ubRES9JQMyfgmjZAgrNQGh6HNQSYD\nH0Y3eYjDIaCj4Lps0fl3QoIoHyHZb60gJEgquy7bv0EiJJshoMLJBgdCMhsCKgptcLiZ9OYe\nyWoI6Bhcy1E+pAfGJCSrIaCjxEFdIt6b52DyEIdDQIv5ZVbZLQ2nJ2HyEIdDwIr6jb+Pjecm\nD3E4BHQZ7m9wMVNISFBguyx7vceOkAyHgCbbQ7vgboq/52DyEIdDQJH5xVapjbK9Z2DyEIdD\nQNGd23+Nw/t6so6QjIeAosEJaY0D3MNS7PGZmDzE4RDQNLSwQ0jLH+JwCGi6f4CLHuLDS7Gs\nI1kOAV3Ho1t3FsDFnobjUzF5iMMhYEB7XtrFnobjUzF5iMMhYEB9OcnDUuzxmZg8xOEQ0De8\nnCR5rBNS8SGgb+jKS+xoN7kJm/B0TB7icAjoG5oLEDrYjaYFJzwhk4c4HAIG7na0+GgnJC9D\nwIBaSGZbJ8Y/I5OHOBwCJg7HuviNDCG5GQJmFKbW7iwgsY5kPwTMDIa08Jj3M+99ekImD3E4\nBKyIT4LvvxUheRkCVoQnwU/fzMsC0ulpmTzE4RCwMhDSkkNfb0p9EUKCtvOhf30tNufQF85S\nDCFBW3Nh/yd/h/7kw1/8jksIIUHf6Xx0DuD7wsRvde/7yD/vKc/L5CEOh4Cx4VPSpAJuelTZ\nUT6PaUifb6vuRVitP7WGgFdX12S9S7KREdxeIfq4OzowDGn30nsdXlWGgF+LZwlu5ywqDWnd\ntO/b7quvj7ZZawwBxx4sAI1JYSDESkNqm+35623TagwBx26vzCbNFQxN17mYrzswDOniveTx\nL34hpJSaO3t7ntdw/+6owpA4I2HovPI8h/vnsqIbvi/Y3iN9fHVfcY9UrdskRqzOup6uO7Kc\n/n7tvRwvO5Uh4FzzIIp7aVw/wtlF3YHtOtK6W0dqV2+sI9Xq2ZTD982//mgzQ6UheRoCZVyd\nkvb/931j+F9u+hN1zjoiJFhrblxuczgWcnEuOmZ0t7jyCAn2rkO6PS913TSnf3b/4zmjciGx\njlS127PS8BnngdI/whVCQgHLQyr9E1zj0g5F3NwCTUyp9PO/RkgoZ855yc9mhgt+Qrq8/UQN\n5oXkkp+QjIeACwNXeN/HObvbhPxmREgob+zMQ+nn+RAhobym98sYTtf2h3K+i/9S75FMP480\n+jaIkKrUXP2nKyIxDGlDSEjL8tJu2z7+lScCQwBlmN4jbR9/nE9iCKAI28mGTe/T5kpDACUw\nawcIICRAACEBAkqE9HydgJAQDCEBAggJEEBIgABCAgQQEiCA6W9AACEBAggJEEBIgABCAgQ4\nDQkIZsZRLh+Os7EZhVEyDVRsbEZhlEwDFRubURgl00DFxmYURsk0ULGxGYVRMg1UbGxGYZRM\nAxUbm1EYJdNAxcZmFEbJNFCxsRmFUTINVGxsRmGUTAMVG5tRGCXTQMXGZhRGyTQQkBkhAQII\nCRBASIAAQgIEEBIggJAAAYQECCAkQAAhAQIICRBASIAAQgIEEBIggJAAAYQECCgW0rpt2vVO\nf5yNwU+4eTH4WXb/mubfVnmQzqf6Szb7V9VPs92/ZF/aoxyUCum1eyVf1MfZ6v91/ay7n6VV\nLqntRjEoaddqv2Rbm5A+TP5ejgqF9Nm0259t23wqj/M7hP77XvNvtz/z/VMdZb3//utmpTpI\nZ6X+km0tfozft57fY2y3atYWY5UKad18/P7zvXnTHWbTvOqHtDqMoDxQ2+z0B9l71z9VbLT/\n3jvvXUK7pjUYq1hIq2Z/6ar+zvT7UhocesexLAbSPyq+DN57Ns1GeYS9fxbXwWeFQmpM3sV/\ntkbH98/+ne9Vf5C1/hH42nypv2Sr5uNf02pfcr00P29td9ltIXdINkN0Nt3Fqqrfiy716/23\n5l3/JVsd5hqU33maphvH5sqOkIR8tfr3z5tVq31z0V1sq79kzW+tPzvt02uzn9Da/TO5HyMk\nIbvW4MLuZ3/dr3vwvewni43ee3bKqx+HtYIvgzWWbjSTUW60yUJ6tfnb0p6D+tddn1pdDSuP\nY/dm3Q1jMsqNw6zdl8F6gsXr+PXyarR+rvzjNGeao/yNpvrtbZYlTgqF9Na99X0YLJYZvI4f\nFhN2h3Uk5QsVo5BOP4vu2+jhGPuy+Mv5Sb+zwSIkm7+qbmfDbmWxAqP+kq33b6C7tfI85++b\nzm4/2fCuOspJqb12LxYToHv6If2zuRxqrV4x/Zdsd/hZtC9H3sxesZ9yIe263d8GA+mHZHVf\n8fuKvVjsCDB4yXY2P8vHq80xtsfnkQABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQI\nICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQI\nICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIIb02n7///Gz+lX4iOCKk\nkL6a9vefbbsr/URwREgxbZq3n7fmvfTTwAkhBfXabJpV6SeBM0IK6qtpmq/STwJnhBTVulmX\nfgr4Q0hBcUbyhZCCWv3eI72WfhI4I6SY3n8v7N6aTemngRNCCmnXdutIXNy5QUgh/TvubODi\nzgtCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBI\ngABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEPAf\nhKryAhuvC/0AAAAASUVORK5CYII=",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "x<-seq(from=0,to=2*pi, length.out=100);\n",
    "y<-sin(x);\n",
    "\n",
    "\n",
    "M=3; \n",
    "x.expanded<-poly(x, degree = M,raw=F); # Set raw=T to see the raw polynomials \n",
    "\n",
    "mod.poly<-lm(y~x.expanded);\n",
    "\n",
    "plot(y~x,pch=16)\n",
    "points(y=mod.poly$fitted.values,x=x, pch=16,col=\"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "### 6.3.2 Tree-Based Methods\n",
    "\n",
    "The tree-based method (e.g., random forest) is a popular approach for building predictive models. A tree is likely an over-simplification of the true relationship. However, it is very easy to interpret, and has a nice graphical representation: you can easily explain it to a non-statistician, and do not need a computer (or even a calculator) to get an estimate!\n",
    "\n",
    "The general steps for building a regression (or classification) tree is quite simple:\n",
    "\n",
    "- **stratify** (or segment) the predictor space, i.e. the set of possible values for $X_1, X_2, \\ldots, X_p$, into {\\color{red} $J$ distinct and non-overlapping regions}, $R_1, R_2, \\ldots, R_J$.\n",
    "\n",
    "- Use the mean (regression) or mode (classification) of the response values for the training observations in region $R_j$ as the predicted response for that region.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   The main questions in this procedure are: <span style=\"color:red\"> how should we construct the regions</span> $R_1, \\ldots, R_J$, and <span style=\"color:red\"> how many regions should we use</span>? \n",
    "\n",
    "\n",
    "Suppose $J$ is known. How to construct the regions $R_1, \\ldots, R_J$?\n",
    "- Divide the predictor space into high-dimensional <span style=\"color:red\"> rectangles</span> or boxes (in theory any shape can be used, but this is simpler)\n",
    "- Find boxes $R_1, \\ldots, R_J$ that minimize the residual sum of squares\n",
    "$$\n",
    "L(y, \\hat{y})=\\sum_{j=1}^J \\sum_{i \\in R_j} (y_i - {\\hat{y}_{R_j}})^2. \n",
    "$$\n",
    "We can see that the criterion for a (regression) tree still takes the squared error form. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, it is not possible to consider every possible partition of the space into $J$ boxes. We consider instead a *greedy* approach called <span style=\"color:blue\"> recursive binary splitting</span>.  We take a top-down approach that is similar to hierarchical clustering. The best split is determined at each given step, instead of the best global step. In particular, at each step we do the following.                            \n",
    "- For every $j=1,\\ldots, p$, we choose  <span style=\"color:red\">cut point</span> $t_j$ such that splitting the predictor space into regions\n",
    "$$\n",
    " R_1(j, t_j) = \\{ X \\mid X_j < t_j\\}, \\ \\ \\  R_2(j, t) = \\{ X \\mid X_j \\ge t_j\\}.\n",
    "$$\n",
    "Here the $t_j$ is chosen as \n",
    "$$\n",
    "\\underset{t}{\\arg\\min} \\sum_{i: x_i \\in R_1(j,t)} (y_i - \\hat{y}_{R_1})^2 + \\sum_{i: x_i \\in R_2(j,t)} (y_i - \\hat{y}_{R_2})^2\n",
    "$$\n",
    "- We find the value of $j$ with the smallest residual sum of squares \n",
    "$$\n",
    "\\sum_{i: x_i \\in R_1(j,t_j)} (y_i - \\hat{y}_{R_1})^2 + \\sum_{i: x_i \\in R_2(j,t_j)} (y_i - \\hat{y}_{R_2})^2. \n",
    "$$\n",
    "\n",
    "We will repeat the above process to split a subspace (rather than the whole space) in each step. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Extensions of Trees **\n",
    "\n",
    "We have considered the squared error loss. Similarly, trees can extend naturally to binary/survival data choose splits greedily to minimize loss.                          \n",
    "                                                    \n",
    "Trees are often used as <span style=\"color:blue\"> base learners</span>\n",
    "<span style=\"color:red\"> model aggregation/ensembling</span>, e.g. boosting, bagging, stacking. \n",
    "\n",
    "Some `R` functions are better equipped to deal with high dimensional data ($p> n$) than others. Rarely will it hurt your method to prescreen out features that are extremely marginally uncorrelated with outcome.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAMFBMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD////QFLu4AAAACXBIWXMAABJ0AAAS\ndAHeZh94AAAeN0lEQVR4nO3di5aiSgyF4cJ7e8H3f9uRAgHxktZJmSb5v7XOjI2IJmEfBW0n\nnQH8t2T9AAAPCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKg\ngCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoI\nEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCAB\nCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCA\nIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggS\noIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEK\nCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCgjSTO2lBen5aOt1SpuP7nWy0Rf38ba7gp7Z\nVKna1OMHkek9ko9Y3z8+s5gO7m7Bi11rddnvth/dbbkg3T3+Z5Y5Nov+5yNBwufudhx5wfiq\nU6FHUX5Th1Qdz8cqHa4Ljmml9iD+A0Gap/8MUqlHUWpT++X10iY1rwF/hmfU3YdPrsoI0iz1\nL2Z2i7TYjRbsLy/bqk27yuQ2N6veXrm/vF5a7tsbbbrb5xtUu/bi5bhkebpu9Ol9pFQv8vPD\n7S23eb123f5Rd9df77l//NdHMnK6bKO/o1V+Nh09De3S7lctK4wgzdJ1v2sPGJb9gm2bks35\nPki3q95cuWsX7Zrrtt1a7ZFUdzHftqrbjT6/j5RWefn0lttJkIbr+3vuHtPwSHo5Wj/1cB/j\nv/LG9us++3YI0jy1O9JPd8DwM+ynP83SdL7byUer3mWsSsdmhUVzTb/WPi3rc71sXkn9NBfX\nTUTaRDy7j9SsN7llu72bII2uH9/z7SNpNU9G6319cx/jv87XVKbl2RZBmqd2R1rlA4Z9+zwz\nuXKyk49WvQtSSvubS/vmhdMqNftv3V48NJeq8S0fB+nQ3tXolu32boI0un58z7eP5Lp0VZ8n\nS8Z/nbtc1xvrF3gEaZ7GRxyjQ5DL/8P32+XjnXz4axqkzWV/PR7vNtifV57ER76P6S1vgzS6\nfnrPw89dNfIzUqsenRA3QZDm6VmQlv0R0BtBOm+r5hjo9Msgyffx6yDd3XP/c29yjFQ9DpLq\ne8OfIEjz9CRI67TY7U9vB+myu24Ww5HK+TY9kyD97j7uloyD9Pye+59HHpy1O929eUSQ8Inb\nY6TVTbIe7uSjVZ/sdO0+fshrra83yJY3x0jP76P7cXTL9uKhv9mhPUaanN+epuv+8Q3vI227\nOvrTdFU+4rpP1pcRpHlqP5xwc9aufZ/ncD4+PH55ddZu0Z6HG521u55wO++aHXTXnGbbDGft\nnt1H9+PolvvhrN0i7ZpTdenm+vE9n24eyTN3n2zIj6veTMP5bQRpnhapeYIY3hzqFmy6o4/D\nfVqGVe+u+hndKK+2Gm6Qj1du3ke6vY9HzySjW+aVc3zyW0Sr0UFWc/1wz+3jH35+VflNHXXV\nv61liSDN02GRg3TeVe3HFa4L1pd97DB6rTfSr3p/VT6gP7TXrK5rNZ8/SOv2uL85m9Z/suHm\nPh6+JBvdctd81uF6XmHdrTJc399z9/j7n5+q86e/R/fXLFiYf7qBIGHk/UP2X93C+kzAF/iv\nEG8gSJ/yX2FgafDL6z4J0vP7+I/Nzo7/CgMjSN/jv0IUFSAjv0Ib8F8IUos2/AkJBlQnqLkx\nfGq+Y5jxMxJB8me+YyBIBTaGT813DASpwMbwqfmOgSAV2Bg+Nd8xEKQCG8On5jsGglRgY/jU\nfMdAkApsDJ+a7xgIUoGN4VPzHQNBKrAxfGq+YyBIBTaGTzEGAwTJH8ZggCD5wxgMECR/GIMB\nguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxh\nDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB\n8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAG\nAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5\nwxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMB\nguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxh\nDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB\n8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAG\nAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5\nwxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMB\nguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxh\nDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB\n8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAG\nAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5\nwxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMB\nguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxh\nDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB\n8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAG\nAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5\nwxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMB\nguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxh\nDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB\n8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAG\nAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5\nwxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMB\nguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxh\nDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB\n8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAG\nAwTJH8ZggCD5wxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5\nwxgMECR/GIMBguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBI/jAGAwTJH8ZggCD5wxgMECR/GIMB\nguQPYzBAkPxhDAYIkj+MwQBB8ocxGCBIf0CSGjddYf/yVtLWdotUbermUr1OaX3s7iIb3+dk\nAV4hSH/Au0FapJe3Era2yQGpmiRV+WKTpOM0N3cL8BJB+gPeDZKw/utrj2l9ydAurZtI5T9W\neelqutrq4c3xGEH6A74QpP3yemmV+m1Uqb5ubZe2tze4W4CXCNIf0AajOXTZ5Z/3q8tLr017\n3aZKm0lyupdc3X/bVG3z67X2BpetpN1k+6fLRqazGRakqrnZ9EZ3C/ASQfoD8k69yvlonji2\n7dFJDsayubR6EaS88n55vcGwlV5z5fKnvr3Lul9nkxOzSvt1H97zowV4iSD9AU0k9mlZn+tl\n2jc//pzPPzk8P6k6no/T55PUn2xIza123Z9Vt5WUt9JqnozW+0mKzs0TTrvKTxfYNoCjBN4t\nwEsE6Q9oIrHKxyv1cIjfPU0dzk08ngfpkC+dxltJN1tZ3afokq+qW2O3qvLBUA5vvRlez90t\nwEsE6Q9oIzE633zab5f5UhrODEzW74+RJj9Pzlo/fkaqq9EzzXoIS50WkxWnC/AEQfoDphFY\n9pf+O0jnh8dIy3E86ny2Ybzlu7uCiCD9AUMksnVa7PanD4N0fjCGyVm702J5ut/cozsiSL9G\nkP6A9uhmP/7xfOqOkZqlh18HKa//+n2k5oREd6l9H+nUvHy7XuwPru4W4CWC9Ac0Ecjn5867\nZsdtziAc22Ok/ZOzdteTC9Mg5a2k3avd/zScicufbKhXzTHSpjl5V2+GNN8twEsE6Q8YHRhV\np+uH4doTcvks9Prus3bNYc2jII228sx6dBhV9ae462p47ypfNV4AGUH6A/pPNqR1TsBlX18e\n9u3Tyvb+kw3nw+JpkEZbeXpno/MRl4OnRXvOrh4utleNFkBGkPxhDAYIkj+MwQBBmomUbl6T\nvVz1G48HtwjSTBCkv40g+cMYDBCk9yX8H+sBlkCQ3hekzGLonyRIh4KUWQz9kwTpUJAyi6F/\nkiAdClJmMfRPEqRDQcoshv5JgnQoSJnF0D9JkA4FKbOYGfdvl4SL9x8K7r4d+i0z7tA7gpRZ\nzHz7dxzeA3t88TwEqfn9/faXWt7/2ov5dugtQcosZrb9G/2G5eOLg33z22SH7tcyD+/e0Ww7\n9J4gZRYz1/7t0vIamccXB3X+urP294t/3v/y57l26E1Byixmrv0b/Ybl44uD9lsKV/krAT74\n5wjm2qE3BSmzmLn27zh8qdLji8Oao9/Z/+SbmObaoTcFKbOYGffv8TeX3SWlfUIiSIIgZRYz\n4/79LkjH5tuZzgRJEqTMYmbcv98F6folZgTptSBlFjPj/v0uSFW6+ZsgPRGkzGJm3L9fBak/\nTdeetfvg22pn3KF3BCmzmBn371dB6v+xw21+ibd//1s2Z9yhdwQps5gZ9+9XQVrlfyf+zCcb\nJEHKLGbG/XsdpO7vRbp+THWRP2v3/r96OOMOvSNImcXMuH+/CtJwTZ0//f3B3Xz48GYmSJnF\n0D9JkA4FKbMY+icJ0qEgZRZD/yRBOhSkzGLonyRIh4KUWQz9kwTpUJAyi6F/kiAdClJmMfRP\nEqRDQcoshv5JgnQoSJnF0D9JkA4FKbMY+icJ0qEgZRZD/yRBOhSkzGLonyRIh4KUWQz9kwTp\nUJAyi6F/kiAdClJmMfRPEqRDQcoshv5JgnQoSJnF0D9JkA4FKbMY+icJ0qEgZRZD/yRBOhSk\nzGLonyRIh4KUWQz9kwTpUJAyi6F/kiAdClJmMfRPEqRDQcoshv5JgnQoSJnF0D9JkA4FKbMY\n+icJ0qEgZRZD/yRBOhSkzGLonyRIh4KUWQz9kwTpUJAyi6F/kiAdClJmMfRPEqRDQcoshv5J\ngnQoSJnF0D9JkA4FKbMY+icJ0qEgZRZD/yRBOhSkzGLonyRIh4KUWQz9kwTpUJAyi6F/kiAd\nClJmMfRPEqRDQcoshv5JgnQoSJnF0D9JkA4FKbMY+icJ0qEgZRZD/yRBOhSkzGLonyRIh4KU\nWQz9kwTpUJAyi6F/kiAdClJmMfRPEqRDQcoshv5JgnQoSJnF0D9JkA4FKbMY+icJ0qEgZRZD\n/yRBOhSkzGLonyRIh4KUWQz9kwTpUJAyi6F/kiAdClJmMfRPEqRDQcoshv5JgnQoSJnF0D9J\nkA4FKbMY+icJ0qEgZRZD/yRBOhSkzGLonyRIh4KUWQz9kwTpUJAyi6F/kiAdClJmMfRPEqRD\nQcoshv5JgnQoSJnF0D9JkA4FKbMY+icJ0qEgZRZD/yRBOhSkzGLonyRIh4KUWQz9kwTpUJAy\ni6F/kiAdClJmMfRPEqRDQcoshv5JgnQoSJnF0D9JkA4FKbMY+icJ0qEgZRZD/yRBOhSkzGLo\nnyRIh4KUWQz9kwTpUJAyi6F/kiAdClJmMfRPEqRDQcoshv5JgnQoSJnF0D9JkA4FKbMY+icJ\n0qEgZRZD/yRBOhSkzGLonyRIh4KUWQz9kwTpUJAyi6F/kiAdClJmMfRPEqRDQcoshv5JgnQo\nSJnF0D/JDDqUpMc4XWF/v/DXZaar6ZLfbsCn4OX/wgw69G6QFul+4dtBqq4LjgTpPIvdxNgM\nOvRukB6t/2aZ+3S4Xjym1Xu3dWkGu4mxGXToC0HaL29+rKshPLu0Fe4+ghnsJsZm0KE2GLtF\nqnb55/3q8spr0163qdJm+iqufSHW/bdN1SUIm5TaGwxbGZwuG7ltwyrV/eVdmq4f0Qx2E2Mz\n6FDey1c5H80Tx7Y9ZsnBWDaXVi+ClFfeL683GLbSa65c/tTjRccuddkq7dd9bsOawW5ibAYd\naiKxT8v6XC/Tvvnx53z+yeH5SdXxfJw+n6T+ZENqbrVr/ky75vTBaCut5slova9v7+/mCanL\n3m344pnBbmJsBh1qItHu2/Vw4N89TTXnBPYvgnTIl06XMp9sZTVNUfOEtL7Z2iW39Sb4C7wZ\n7CbGZtChNhKjs9Cn/XaZL3UJehGk/tL9VvKWHj0jbYZnrF6dFpolzc4MdhNjM+jQNALL/tJ/\nB+n86BipetST4G8kxa7+N2bQoSES2TotdvvTh0F6tP3JWbvHbxwRJLw0gw61Rzf78Y/nU3eM\n1Cw9/DpIqwcv2rLx+0iT891VPq46BX9bdga7ibEZdKiJQD4/d9nHV+0ZhGN7jLR/ctbudH4c\npNFWnlul4/jHTXMuvH503BTJDHYTYzPo0OjAqDrlN1ez5oRcPje9vvusXXOq+1GQRlt5btGf\n/M43qavhbau4ZrCbGJtBh/pPNqR1TsAlOMvDvn1a2d5/suF8WDwN0mgrwt0Nl+rLIdQi9snv\nWewmxoJ0KEiZxdA/SZAOBSmzGPoncdKhlB69RzS6/suPxxv6J3HSIYJUFv2TBOlQkDKLoX8S\nyw4lnww7WozLolSZBsnwvsvxWRUEBEmbz6ogIEjafFYFAUHS5rMqCAiSNp9VQUCQtPmsCgKC\npG3uVd2cxt9UaTn9BZJDV+Hlumpz/5UXQREkbXOv6pqj5kub2187uf2GzLr7Vfz2uthfZTFC\nkLT5qCp/afMuf33Z+vY3HbuvETx0v1R5eHz7cAiSNhdVtV/avMwxOd38VuPP9UVf/p3hH77Q\nuUOQtLmoqv0GwOuXy4y+0eKU2t/yv6zR/H4k/8TAFUHS5qGq7kubH3xL0zKdht87nlwXGkHS\n5qGq7kubF/lZZ/wtTdv0cyZIjxAkbQ6qun5p87b5RufuG5u6K1ZngvQQQdLmoKr+y8fyNyiN\n/rWPRVUTpMcIkjYHVfVf2lyvm39eqg/LOges/bEiSDcIkrb5VzU5FXfs33Ud//Jie9Yu+jfQ\nDgiStvlX1X9pc/t1zcM3046DtM3PTvvo35zZI0ja5l9V/6XNm+akw2HR/AtRI3yy4RGCpG3+\nVfVf2tx9XXN+Qpp+AW3zzdAp/L9kOCBI2uZf1egfYltfYrSfLOwu1fnT399+bH8WQdLmsyoI\nCJI2n1VBQJC0+awKAoKkzWdVEBAkbT6rgoAgafNZFQQESZvPqiAgSNp8VgUBQdLmsyoICJI2\nn1VBQJC0+awKAoKkzWdVEBAkbT6rgoAgafNZFQQESZvPqiAgSNp8VgUBQdLmsyoICJI2n1VB\nQJC0+awKAoKkzWdVEBAkbT6rgoAgafNZFQQESZvPqiAgSNp8VgUBQdLmsyoICJI2n1VBQJC0\n+awKAoKkzWdVEBAkbT6rgoAgafNZFQQESZvPqiAgSNp8VgUBQdLmsyoICJI2n1VBQJC0+awK\nAoKkzWdVEBAkbT6rgoAgafNZFQQESZvPqiAgSNp8VgUBQdLmsyoICJI2n1VBQJC0+awKAoKk\nzWdVEBAkbT6rgoAgafNZFQQESZvPqiAgSNp8VgUBQdLmsyoICJI2n1VBQJC0+awKAoKkzWdV\nEBAkbT6rgoAgafNZFQQESZvPqiAgSNp8VgUBQdLmsyoICJI2n1VBQJC0+awKAoKkzWdVEBAk\nbT6rgoAgafNZFQQESZvPqiAgSNp8VgUBQdLmsyoICJI2n1VBQJC0+awKAoKkzWdVEBAkbT6r\ngoAgafNZFQQESZvPqiAgSNp8VgUBQdLmsyoICJI2n1VBQJC0+awKAoKkzWdVEBAkbT6rgoAg\nafNZFQQESZvPqiAgSNp8VgUBQdLmsyoICJI2n1VBQJC0+awKAoKkzWdVEBAkbT6rgoAgafNZ\nFQQESZvPqiAgSNp8VgUBQdLmsyoICJI2n1VBQJC0+awKAoKkzWdVEBAkbT6rgoAgafNZFQSM\nHVBAkAAFBAlQQJAABQQJUECQAAXfClK9qVK1qfPFdUrr4/jKy3XLfftwWl96UP9hd32MQ13j\npb1DerAa/PnSPnuqckCq0+Vye3GUpGVesL1cOs4lSMfrY2wf+2KytFdX6X41OPSlfXadNpc/\nN2nd/7Hqr9ulZd08TR2bPXH1fBt/yLHqInNI1bH56XCzdLBql9ysBo++FKRuD2v+qlI9LGgs\n8/51aqK2y89Lf94l+t3j36TmJelPftjD0t5P9+w6Xg0ufSlI3SucVPX321/qQ7Zs9sXddx7P\n/7lkvnvQq9S8WG2fSIelV6drtMarwaUvBWnbvbS7/i95M0rM6Nlqlfbry0H5dx7T5443D/r6\n1/E8DdIyndolt2vDoW+NdtecYqi69Fxe8YzCssj/uz60QcqWX3pQ/+FBkM7TpGzTz5kgBfGt\n0W77M3MXu1U1OlzYplV9PuYXQemy653rzQxe4P0iSPmVHEEK4kuj3TVPQfV6SMjoYns+fDXs\nZfUMzhP/IkiLqiZIYXxptIt8qm6UkHp0tuESsGo73stmsMN1D7F6HqR1PlPXLqkIknffP/19\nu6R3HD0NzWCHuzlrd7qejhs/8NS7Ww3+fPX0d34aat9HOg25aRfsmr3set3f3+G6yGzz087+\neu7kWZAmq8GfLwVpk5oPmm2aPSl/sqFeDcdIecFh0ZxnyCvU7duXf9ujTzY8eirlkw1BfOtF\n1HI4r10NF/NuVrcLVsPFGfyP+xqZxc35+vtjpe7S7Wrw52tHI/njz/3FRft81O5mp/UlRu2T\nUD1c97ddo1KP6noRpNvV4M/fP6wHZoAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEK\nCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAg\nAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKg\ngCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoI\nEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCAB\nCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCA\nIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggS\noIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEKCBKggCABCggSoIAgAQoIEqCAIAEK\nCBKggCABCggSoIAgAQoIEqCAIAEK/gF1IkMA/j+8OgAAAABJRU5ErkJggg==",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "library(tree)\n",
    "\n",
    "mod.adr.tree <- tree(adr~lead_time+ total_of_special_requests+ stays_in_week_nights+ stays_in_weekend_nights, data=hotel)\n",
    "plot(mod.adr.tree)\n",
    "text(mod.adr.tree, all=T, cex=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.4 Model selection\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.1 Overview \n",
    "\n",
    "Model selection has many meanings. \n",
    "\n",
    "- It can mean selecting features and how they are to be included in the model. This is known as feature selection, and sometimes feature generation. \n",
    "- It can mean selecting the tuning parameters in e.g., penalized regression, decision trees, deep learning. This is also known as <span style=\"color:blue\"> tuning parameter selection</span>\n",
    " - It can mean choosing between a range of models, for instance, regression tree (random forest), neural network (deep learning), or logistic regression. \n",
    " \n",
    "Recall that we have identified the criterion for deciding the \"best\" model. It is natural to wonder why not just pick the one model that minimizes that criterion. There are two important questions to address here. \n",
    "\n",
    "1. On what data to evaluate the criterion.\n",
    "2. How to evaluate the criterion in a computational feasible way.\n",
    "\n",
    "\n",
    "The first question is particular important. If the answer is to evaluate the criterion based on same data set, a phenomenon known as **overfitting** is inevitable. The game for model selection is essentially to avoid overfitting. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.2 Resctricting model complexity\n",
    "\n",
    "Roughly speaking, the complexity of a model is tantamount to the number of parameters in the model. A model with more parameters is more flexible for learning the unknown mechanism, but at the same time may be sensitive to random patterns that are unique to this observed data set. A solution is to restrict the model complexity by penalizing the number of parameters. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.4.2.1 Information criteria\n",
    "\n",
    "The criterion in Section 6.2 places no constraint on the number of parameters. A classic strategy for avoiding overfitting is to supplement the loss function with a penalty on the **number of parameters**. Two very famous examples are the Akaike information criterion and the Bayesian information criterion. \n",
    "\n",
    "- Akaike information criterion (AIC): ${\\rm AIC}=2k - \\log (\\hat{L}_k)$ where $k$ is the number of parameters and $\\hat{L}_k$ is the likelihood function with $k$ parameters. \n",
    "\n",
    "- Bayesian information criterion (BIC): ${\\rm BIC}=k\\log(n) - \\log (\\hat{L}_k)$ where $k$ is the number of parameters and $\\hat{L}_k$ is the likelihood function with $k$ parameters. \n",
    "\n",
    "The only difference between AIC and BIC is in the weights of $k$. In AIC, the weight is a constant 2, while in BIC the weight is $\\log(n)$ that increases as the sample size increases. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both criteria capture the trade-off between the model complexity (i.e., $k$) and the goodness-of-fit (i.e., $\\hat{L}_k$). Take AIC for example, AIC increases as the model become more complex (i.e., $k$ increases) and decreases as the model fits the data better (i.e., $\\hat{L}_k$ increases and thus $-\\log(\\hat{L}_k)$ decreases). We will pick the model that yields the minimum information criterion among all competitors. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.4.2.2 Penalization or regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to restricting the number of parameters, we can also restrict the range of the parameters. This can be done by placing penalties on the magnitudes of the parameters. Two popular choices are the $\\ell_1$-norm and $\\ell_2$-norm on the parameters. \n",
    "\n",
    "- $\\ell_1$-norm (lasso): $P(\\beta_1,\\ldots,\\beta_p) = |\\beta_1| + \\ldots + |\\beta_p|.$\n",
    "\n",
    "- $\\ell_2$-norm (ridge, Tikhonov): $P(\\beta_1,\\ldots,\\beta_p) = \\beta_1^2 + \\ldots + \\beta_p^2.$\n",
    "\n",
    "We then choose $\\beta_0,\\beta_1,\\ldots, \\beta_k$, to minimize\n",
    "\t$$\n",
    "\t L\\left( y,  {\\beta_0 + \\beta_1 x_1 + ... +\\beta_k x_k}\\right) + \\lambda P\\left(\\beta_1,\\ldots,\\beta_k\\right),\n",
    "\t$$\n",
    "where $\\lambda$ modulates the degree of penalization. In other words, increasing $\\lambda$ reults in a less complex model.  \n",
    "\n",
    "Remark: For variable selection, the $\\ell_0$-norm $P(\\beta_1,\\ldots,\\beta_p) = 1_{[\\beta_1\\neq 0]} + \\ldots + 1_{[\\beta_p\\neq 0]}$ is the ideal penalty, as it produces sparse estimates and causes no bias. However, it is computationaly intractable to solve the minimization problem when $\\ell_0$-penalty is involved. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Example with lasso and ridge\n",
    "library(glmnet)\n",
    "mod.can.lg.lasso <- glmnet(as.matrix(hotel[,-1]), hotel$is_canceled)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod.can.lg.ridge <- glmnet(as.matrix(hotel[,-1]), hotel$is_canceled,alpha=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Logistics with lasso penalty\n",
    "mod.can.lg.lasso <- glmnet(as.matrix(hotel[,-1]), hotel$is_canceled, family = \"binomial\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.4.2.3 Tree Pruning\n",
    "\n",
    "\n",
    " Complexity of the regression tree is determined by the number of regions $J$\n",
    "\n",
    "- A tree with large $J$ may work well in training data, but would be very bad on test data\n",
    "- A smaller tree with fewer splits might have lower variance and better interpretation at the cost of a little bias\n",
    "- One option is to consider a split only if the drop in residual sum of squares is larger than some (high) threshold\n",
    "- However, this may not be a good strategy since a so-so split at step $j$ may be followed by a great one at step $j+1$\n",
    "- Instead, we <span style=\"color:blue\"> first grow a large tree</span>, e.g. until no region has $>5$ observations, and then <span style=\"color:red\"> prune</span> it to obtain a <span style=\"color:red\"> subtree</span>\n",
    "- And, again, we (basically) use <span style=\"color:red\"> cross validation</span> to select $J$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To prune a tree, we can use cross-validation, however, it will be too computationally expensive to estimate the CV error for every possible subtree. Instead we use a strategy called <span style=\"color:red\"> cost complexity pruning</span> a.k.a. <span style=\"color:red\"> weakest link pruning</span>. \n",
    "\n",
    "Rather than considering every possible subtree, we consider a sequence of trees indexed by a nonnegative tuning parameter $\\alpha$. For each value of $\\alpha$, there exists a subtree $T \\subset T_0$ such that \n",
    "$$\n",
    "\\sum_{k=1}^{|T|}\\sum_{x_i \\in R_k} (y_i - \\hat{y}_{R_k})^2 + \\alpha |T|\n",
    "$$\n",
    " is as small as possible ($|T|$ is the number of leaves of the tree). \n",
    " \n",
    " Here  $\\alpha$ controls the tradeoff between complexity and fit and we can select $\\alpha$ using validation set or CV approach"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAMFBMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD////QFLu4AAAACXBIWXMAABJ0AAAS\ndAHeZh94AAAgAElEQVR4nO3dbVvqOBSF4YSWgkjb//9vhxZU1KOjdifs7PXcH7TONYewTJf0\njZJmAJulRz8BIAKKBBigSIABigQYoEiAAYoEGKBIgAGKBBigSIABigQYoEiAAYoEGKBIgAGK\nBBigSIABigQYoEiAAYoEGKBIgAGKBBigSIABigQYoEiAAYoEGKBIgAGKBBigSIABigQYoEiA\nAYoEGKBIgAGKBBigSIABigQYoEiAAYoEGKBIgAGKBBigSIABigQYoEiAAYoEGKBIgAGKBBig\nSIABigQYoEiAAYoEGKBIgAGKBBigSIABigQYoEiAAYoEGChUpGnIKQ/T7afjyyh3//U3i15M\n+5T254+Lrac67u6fXXeaXxZbDvVD6cP6v0zr8LdHMng2n405LfK4/nR+ebrd+l93v11045rq\n/GGx8VTDdaqWHlyf3WGemw/1Ux+L1L/k//0jGTybz/ZrrYe0X34459vTfU75vPz0/LtFN9Y8\nQ+rfLzae6pz207LNsF++dNPyN/ncfKgf+1iklMa/PtLm5/LPR01v3y7Tc/txSMtmw9NS+d8s\nupHT8md7DXO32Hiq/m2qurUL4/JHsPFQP/a5SH9+pK1P5Z/ybXby8mV4eXr9Wvfz8nf8N4vO\nrKHuFoOkSm9//rooof7fGvnyMtSnfFgWFn98JNPn9eJw27Rb/kid379AvZ+znyz6MqTj+8UQ\nqaalPX9M4jbUD9yKlK/7hv6KNB+Xp5ZfVrk4RXp6O6jzshgg1bL9fdk8260vLc+CRbrsGx7X\noyXeNu0uL0np7vhHnCId+/yS6mUxQKp5zMt22SH103zuBIv0/Lb450cye073jssf62n/shUU\np0jzckTy+G4xQKopd+v3dQunFyzSh8U/PZLVU3pntx7Uml7OLdyeXn77jf9m0Znp7WjDuhgg\nVXebp8ufvnz4dRKnob7zujPkvUgf/kjdvl8P74xvR3p+tujN3S97WWw+1bjr7k+enJc/f82H\n+l4zRbr+kXr92317eof1hMNp2ez7zaIb15NH47Ki3S22nuqUutvSNdRxaUTroX7Ke5GGtFx6\nNbz8bm9Pr/XT5evlDFO/7BjdLTaeanzt0TXU8y49NR/qx7wX6Xb51csUvTy93dt//c2iG/nt\nOeW/RfGXap/Sy4bOdA21bqK1HerH3BfpekHw6yC3Uaa3//qbRT8uz2l3/LjYdqr0VqR5vLSq\nv1793XaoH/NfJEAKRQIMUCTAAEUCDFAkwABFAgxQJMAARQIMUCTAAEUCDFAkwABFAgxQJMAA\nRQIMUCTAAEUCDDyySDFLHDIVoao+WENjlxMyFaGqPlhDY5cTMhWhqj5YQ2OXEzIVoao+WENj\nlxMyFaGqPlhDY5cTMhWhqj5YQ2OXEzIVoao+WENjlxMyFaGqPlhDY5cTMhWhqj5YQ2OXEzIV\noao+WENjlxMyFaGqPlhDY5cTMhWhqj7YF0MAjfnDWm5fnAcMAViiSIABigQYoEiAAYoEGKBI\ngAGKBBigSIABigQYoEiAAYoEGKBIgAGKBBigSIABigQYoEiAgZpFGvcpH+b5uEt5KDQE8BgV\nizTl5Y2Ex8P6fsKuyBDAg1Qs0pAur0NDTvtpntZl+yGAB6lYpLz+w5Sm9VsuMQTwIBWLlNLb\n1/n7m0VQJDTmAa9Iy9eJVySE8oB9pGG6Lf9hiIfecKmcP/5G4QdH7RwQixsS55EcEIsbElc2\nOCAWNySK5IBY3JCqFun50K87SP3wXGqIJonFDanmwYbd3XEqDjbcEYsbUtXD3/npvC6Np8wl\nQnfE4oZU9YTs+XX5zAnZO2JxQ6p+idC/fjAbolFicUPiFckBsbgh1d1HOo3rEvtI74nFDanm\n4e/u7qjdbioyRJvE4oZU9zzSsJ5Hyv2B80j3xOKGxJUNDojFDYkiOSAWN6T6RfrJu2/E1iyx\nuCFRJAfE4oZEkRwQixsSRXJALG5IdYv007sUiK1ZYnFDokgOiMUNiU07B8TihkSRHBCLGxJF\nckAsbkgUyQGxuCFxiZADYnFDokgOiMUNqXqRTv2ybdePBYdojljckGoXqbueREr52yaJrVli\ncUOqXKRj6qalSMe0LzVEg8TihlS5SDlN18N23EXojljckCoXad2so0gfiMUNqXKRdrdXpHPa\nlRqiQWJxQ3rMPtIpp2OpIRokFjek2kft+i9voi/8WZBicUN6yHmk1D+VHKI5YnFD4soGB8Ti\nhkSRHBCLG1LNIk3Dcuf8w+6yh8Sm3T2xuCFVLNKYU5qnzCf2fSIWN6SKRdqnfrp82Y+XTu35\nNIo7YnFDqliklKbbl8tWHp+PdEcsbkhVizQvF9vd/WA+RKPE4oZUddPuPM+H68f2Td/vJImt\nWWJxQ6pYpHPKw3nu86VJp106lRiiUWJxQ6p5+PuU364BOpQZok1icUOqe0L2ab9bWtQfvn+n\nudqaJRY3JK5scEAsbkgUyQGxuCHVPPyd/+cjmLcP0SixuCFVPY+0XNpQdIhGicUNqWqRTjkN\nP6qS2JolFjekulc2TH1K+2/PIG0bolFicUOqfYnQeX2D7PH8/QuT2JolFjek2kW6VGnI/3tb\nBrE1SyxuSPWLdHE+9juK9EYsbkgPKVKxIRolFjckiuSAWNyQuLLBAbG4IVEkB8TihkSRHBCL\nG9KjisTh7zticUOiSA6IxQ2JTTsHxOKGRJEcEIsbEkVyQCxuSFXv/b1Pqbtd+80+0h2xuCFV\nLNLttt/99UEo0huxuCFVLNKwfNzldMzrrSEp0h2xuCFVLNLtbsVj3o0U6R2xuCE94KLVqeso\n0jticUOqWKRdenlb7K6jSPfE4oZUsUjHtL8tjamjSHfE4oZU8/D38NqeE281vycWN6SqJ2TP\n/cvSuKdIb8TihsSVDQ6IxQ2JIjkgFjckiuSAWNyQKJIDYnFDokgOiMUNiSI5IBY3JIrkgFjc\nkCiSA2JxQ6JIDojFDYkiOSAWNySK5IBY3JAokgNicUOiSA6IxQ2JIjkgFjckiuSAWNyQKJID\nYnFDokgOiMUNiSI5IBY3JIrkgFjckCiSA2JxQ6JIDojFDYkiOSAWNySK5IBY3JAokgNicUOi\nSA6IxQ2JIjkgFjckiuSAWNyQKJIDYnFDokgOiMUNiSI5IBY3JIrkgFjckCiSA2JxQ6JIDojF\nDYkiOSAWNySK5IBY3JAokgNicUOiSA6IxQ2JIjkgFjckiuSAWNyQKJIDYnFDokgOiMUNiSI5\nIBY3JD9FSvfKDOGVWNyQ/BSp8hCeiMUNiSI5IBY3JIrkgFjckCiSA2JxQ6JIDojFDYkiOSAW\nNySK5IBY3JAokgNicUOiSA6IxQ2JIjkgFjckiuSAWNyQKJIDYnFDokgOiMUNiSI5IBY3JIrk\ngFjckCiSA2JxQ6JIDojFDYkiOSAWNySK5IBY3JAokgNicUOiSA6IxQ2JIjkgFjckiuSAWNyQ\nKJIDYnFDokgOiMUNiSI5IBY3JIrkgFjckCiSA2JxQ6JIDojFDYkiOSAWNySK5IBY3JAokgNi\ncUOiSA6IxQ2JIjkgFjckiuSAWNyQKJIDYnFDokgOiMUNiSI5IBY3JIrkgFjckCiSA2JxQ6JI\nDojFDalmkcZ9yod5Pu5SHgoN0SaxuCFVLNKU08XxsHxNXZEhGiUWN6SKRRrS5XVoyGk/zdO6\nbD9Eo8TihlSxSHn9hylN67dcYohGicUNqWKRUnr7+vLNeIhGicUN6QGvSMvXiVekO2JxQ3rA\nPtIw3Zbth2iUWNyQOGrngFjckDiP5IBY3JC4ssEBsbghUSQHxOKGVLVIz4d+3UHqh+dSQzRJ\nLG5INQ827NIbDjbcEYsbUtXD3/npvC6Np8zh7zticUOqekL2/Lp85oTsHbG4IVW/ROhfP5gN\n0SixuCHxiuSAWNyQ6u4jncZ1iX2k98TihlTz8Hd3d9RuNxUZok1icUOqex5pWM8j5f7AeaR7\nYnFD4soGB8TihkSRHBCLG1L9In1/4NtkiNaIxQ2JIjkgFjckiuSAWNyQKJIDYnFDqlukt/NI\nxYZokVjckCiSA2JxQ2LTzgGxuCFRJAfE4oZEkRwQixsSRXJALG5IXCLkgFjckCiSA2JxQ6pe\npFO/bNv1Y8EhmiMWN6TaRequJ5FS/rZJYmuWWNyQKhfpmLppKdIx7UsN0SCxuCFVLlJO0/Ww\nHXcRuiMWN6TKRVo36yjSB2JxQ6pcpN3tFemcdqWGaJBY3JAes490yulYaogGicUNqfZRu/7L\nm+ine1uGaI9Y3JAech4p9U8lh2iOWNyQuLLBAbG4IVEkB8TihlSzSNP+sm90uj0Ih7/fiMUN\nqWKRpnz93Mvrg1CkN2JxQ6pYpGE55D0d83rAjiLdEYsbUsUi5es/HPNupEjviMUNqWKRXroz\ndR1FekcsbkgVi7RcHnRb6ijSPbG4IVUs0ttbJ8bUUaQ7YnFDqnn4e3htz+l/rgISW7PE4oZU\n9YTsuX9ZGvcU6Y1Y3JC4ssEBsbghUSQHxOKGVPPwd/6fj2DePkSjxOKGVPU8Uuqn///ftgzR\nKLG4IVUt0imn4UdVEluzxOKGVPfKhqlPaX8qN0SjxOKGVPsSofP6Btnj+fsXJrE1SyxuSPWv\ntTsP+X9vyyC2ZonFDekBF61eunTsdxTpjVjckB5SpGJDNEosbkgUyQGxuCFxZYMDYnFDokgO\niMUNiSI5IBY3pEcVicPfd8TihkSRHBCLGxKbdg6IxQ2JIjkgFjckiuSAWNyQahaJe39/QSxu\nSBWLxL2/vyIWN6SKReLe318RixtSxSJx7++viMUN6QEXrXLv74/E4oZUsUjc+/srYnFDqlgk\n7v39FbG4IdU8/M29v78gFjekqidkuff3v4nFDYkrGxwQixsSRXJALG5IFMkBsbghUSQHxOKG\nRJEcEIsbEkVyQCxuSJuLdOqXU0L9aPR8/jVEeGJxQ9papO56G++UTZsktmaJxQ1pY5GOqZuW\nIr1d/mNCbM0SixvSxiLlNF2vP/3F/Yh/OYQAsbghbSzSullHkTYSixvSxiLtbq9I57Qze0qz\n3JolFjckm32kU17eRm5HbM0SixvS1qN2fbrqrJ7Q5yHiE4sbksl5pNQ/GT2dfw4RnljckLiy\nwQGxuCFRJAfE4oa0tUjTkC9f8zB98T//jdiaJRY3pI1FGvPtLBKXCG0gFjekjUXq0n55LZqG\n1H/1v/+F2JolFjckgysb3i+YEFuzxOKGZHCt3WKiSBuIxQ1pY5GG1D1fvj13abB6Rh+HECAW\nNyST9yNxZcM2YnFD2nwe6Wm5sqEzvdJObs0SixsSJ2QdEIsbEkVyQCxuSBTJAbG4IW0t0mF3\nO9rA4e+/E4sb0sYiHVKiSJuJxQ1p8wlZ4+N1n4cQIBY3JKtLhGyJrVlicUPaWKQ+2b5/4h9D\nCBCLG9Lmt1GslwhZE1uzxOKGtHnTjoMN24nFDYkiOSAWNyROyDogFjckiuSAWNyQrIr0zFvN\n/04sbkhbizSwj7SdWNyQNr9D9sXJ7CnNcmuWWNyQNl8i9DR3aRy7ZHo6SWzNEosbksElQofL\nq9F5+3vNUyp0LN0/sbghGRTptFy4yj7SBmJxQ9p8rd3TPKbd/EyRNhCLG9LGIp2WAq13EuLD\nmP9OLG5Im98hu/y0T7a3tVNbs8TihsSVDQ6IxQ2JIjkgFjekDUVado+4+tuCWNyQKJIDYnFD\nYtPOAbG4IW0sUpE7NsitWWJxQ9p6ZUNnerHqv4YQIBY3pI1FWu6zOtjf/kRszRKLG9LWfaRx\nuWfx7mC8iSe2ZonFDcngYMM45GS8iSe2ZonFDcnmqN2Rw99biMUNyeIVad26ezJ5Ol8MEZxY\n3JBM9pHyMFo9n38MEZ9Y3JAMjtrtOWq3kVjckDafRzLdpPvXEALE4obElQ0OiMUNafPBhlO/\nHLDrbXeSxNYssbghbS1Sd73wO2XTJomtWWJxQ9pYpGPqpqVIR+7ZsIFY3JA23yByut6JixOy\nG4jFDcngvnYUaSuxuCFtPo90fUU6p53ZU5rl1iyxuCHZ7COd8nK3VTtia5ZY3JC2HrXrb3ds\n2Hzr76+HiE8sbkgm55FSb3x9g9iaJRY3JG5+4oBY3JAokgNicUPadF+7Yh9pJLZmicUNiSI5\nIBY3pM1H7fJys4bnbHqFkNqaJRY3pI1FGtJ5/X62/VwXsTVLLG5IBpcIvV8wIbZmicUNafNF\nqy+vSNnm+XweQoBY3JA2b9rl5Y4Np5wOVs/o4xACxOKGZPLGvuXaBqsn9HmI+MTihrT5hOzT\neomQ8a30xdYssbghcWWDA2JxQ6pZpHGf8mVX6rhL+X8OloutWWJxQ6pYpCkvO1PHww/ediG2\nZonFDalikYblpO2Q036ap+H7E7hia5ZY3JAqFimv/zCl9aaS3593EluzxOKGVLFIKb19/Z8r\nIcTWLLG4IT3gFWn5OvGKdEcsbkgP2Ecaptuy/RCNEosbEkftHBCLGxLnkRwQixsSVzY4IBY3\nJIrkgFjckKoW6flwvZ9kP/zPp2WKrVlicUOqebBhd3erFA423EkhPfq3WlfVw9/56fp+2vGU\nOfwdndgcVj0he35d/p+3potNQkhic1j9EqF//WA2BPwQm0NekVCG2BzW3Uc6XT+ymX0kAWJz\nWPPwd3d3SGc3FRkCbojNYd3zSMN6Hin3B84jhSc2h1zZgDLE5pAioQyxOaxfpJ+c8habhJDE\n5pAioQyxOaRIKENsDikSyhCbw7pF+umlwWKTEJLYHFIklCE2h2zaoQyxOaRIKENsDikSyhCb\nQ4qEMsTmkEuEUIbYHFIklCE2h9WLdOqXbbt+LDgEPBCbw9pF6q4nkVL+tklikxCS2BxWLtIx\nddNSpGPalxoCPojNYeUi5TRdD9txF6HoxOawcpHWzTqKpEBsDisXaXd7RTqnXakh4IPYHD5m\nH+mU07HUEPBBbA5rH7Xrb1d/f76JvvAN2EMSm8OHnEdK/VPJIeCB2BxyZQPKEJtDioQyxOaw\nZpH4MGYlYnNYsUhTXg4jHA9fHGywGAJ+iM1hxSINyydQDDntp3ka+DSK6MTmsGKR8voPU1o/\nh4LPR4pObA4rFimlt69cIhSe2Bw+4BVp+TrxihSd2Bw+YB9pmG7L9kPAD7E55KgdyhCbQ84j\noQyxOeTKBpQhNocUCWWIzWHNIk3DcqjusLvsIXH1d3hic1ixSGNO6XbEgYMN8YnNYcUi7VM/\nXb7sx/WwA4e/gxObw6pXNkyvVwhxQjY8sTmsfYlQTnc/mA8BP8TmsOqm3XmeD8uX5RXp250k\nsUkISWwOKxbpnPJwnvt8adJpl04lhoAfYnNY8/D3Kb/dJuhQZgi4ITaHdU/IPu13S4v6w/cf\nRqE2CSGJzSFXNqAMsTmkSChDbA4pEsoQm8NHFYnzSNGJzSFFQhlic8imHcoQm0OKhDLE5pAi\noQyxOaz6xr59St3t0iD2kaITm8PqdxFK/fVBKFJwYnNY9b52x0ubjnm97psiRSc2h9XvtDqP\neTdSpPjE5rD6vb8vL0pdR5HiE5vDikXaXd9lvix1FCk8sTmsWKRj2t+WxtRRpOjE5rDm4e/h\ntT2nRJGiE5vDqidkz/3L0rinSMGJzSFXNqAMsTmkSChDbA4pEsoQm0OKhDLE5pAioQyxOaRI\nKENsDikSyhCbQ4qEMsTmkCKhDLE5pEgoQ2wOKRLKEJtDioQyxOaQIqEMsTmkSChDbA4pEsoQ\nm0OKhDLE5pAioQyxOaRIKENsDikSyhCbQ4qEMsTmkCKhDLE5pEgoQ2wOKRLKEJtDioQyxOaQ\nIqEMsTmkSChDbA4pEsoQm0OKhDLE5pAioQyxOaRIKENsDikSyhCbQ4qEMsTmkCKhDLE5pEgo\nQ2wOKRLKEJtDioQyxOaQIqEMsTmkSChDbA4pEsoQm0OKhDLE5pAioQyxOaRIKENsDv0UKd0r\nMwQqEptDP0WqPAQKE5tDioQyxOaQIqEMsTmkSChDbA4pEsoQm0OKhDLE5pAioQyxOaRIKENs\nDikSyhCbQ4qEMsTmkCKhDLE5pEgoQ2wOKRLKEJtDioQyxOaQIqEMsTmkSChDbA4pEsoQm0OK\nhDLE5pAioQyxOaRIKENsDikSykghfR33D7+hDb9dP0MAv0eRAAMUCTBAkQADFAkwQJEAAxQJ\nMECRAAMUCTBAkQADFAkwQJEAAxQJMECRAAMUCTBAkQADFAkwQJEAAxQJMECRAAM+ijTuUz7M\n83GX8lBoCKAkF0Wa8nIbluNhvRtLV2QIoCgXRRrS5XVoyGk/zdO6bD8EUJSLIuX1H6Y0rd9y\niSGAolwUKaW3r/M3t9rbMARQlIsi5bsiTbwioUEuivSyjzRMt2X7IYCiXBSJo3ZonYsicR4J\nrfNRJFdDAL9HkQADTor0fOjXHaR+eC41BFCQiyJNu7sPbOJgAxrkokhDyk/ndWk8ZQ5/o0Eu\nipTT+XX5zAlZNMhFkd5dFcQlQmiQiyLxioTWuSjSZR/pNK5L7COhTS6KNHd3R+12U5EhgJJ8\nFGl+HtbzSLk/cB4JLXJSJE9DAL9HkQADnor0/YFvkyGAMigSYIAiAQYoEmDAS5HeziMVGwIo\nhyIBBrwUaf3nbNqhVRQJMECRAAMUCTDgqUhOhgB+jyIBBhwV6dQv23b9WHAIoBA/RequJ5FS\n/rZJFAkuuSnSMXXTUqRj2pcaAijGTZFymq6H7biLEBrkpkjrZh1FQqPcFGl3e0U6p12pIYBi\n3BTpto90yulYagigGDdFmvsvb6Kf7m0ZAijFT5HW80ipfyo5BFCIoyJ5GQL4PYoEGHBWpP/d\nB6JIcIkiAQZcFCmlHx+Yo0hwyUWRnjNFQttcFGme+tStF32zaYc2+SjSPD+ltJxBokhok5ci\nzWOX+okioVFuijTPh5RPFAltclSk+bz7/2vpKBJc8lSked5TJLTJV5FcDAH8no8iTUO+fD1c\nNu06rv5Gi1wUacyXTbopf/V+JIshgKJcFGm/HPnep/146dQ+DSWGAIpyUaSUptuXy1ZeyiWG\nAIpyUqR5uR/X3Q/mQwBFuSjSPp2X87HnZXn6fieJIsElF0U6pzyc5z5fmnTapVOJIYCiXBRp\nuQnXq0OZIYCSfBRpnp/2y/VBqT98/2EUFAk+eSmSoyGA36NIgAGKBBjwVyTOI6FBFAkw4K9I\nDx8C+D2KBBigSIABH0Wa9il1t0uD2EdCg1wU6faevv76IBQJ7XFRpGH5uMvpmNfrvikSGuSi\nSLe3Io15N1IkNMlFkV66M3UdRUKTXBRpd32X+bLUUSS0yEWRjml/WxpTR5HQIBdFmofX9pz4\nfCS0yEeR5nP/sjTuKRLa46RInoYALFEkwABFAgxQJMAARQIMUCTAAEUCDFAkwABFAgxQJMAA\nRQIMUCTAAEUCDFAkwABFAgxQJMAARQIMUCTAAEUCDFAkwABFAgxQJMCA0yIBjfnDWm5fHI9j\nhxwqZKhWf38Uqd2hQoZq9fdHkdodKmSoVn9/FKndoUKGavX3R5HaHSpkqFZ/fxSp3aFChmr1\n90eR2h0qZKhWf38Uqd2hQoZq9fdHkdodKmSoVn9/FKndoUKGavX3R5HaHSpkqFZ/fxSp3aFC\nhmr198el2YABigQYoEiAAYoEGKBIgAGKBBigSIABigQYoEiAAYoEGKBIgAGKBBigSIABigQY\noEiAAYoEGHhgkY7lx34/xJ9vkP6LAXcpD1PNEcqHmvYp7c8FB/jHEBWmap6fLR//cUU6l/49\nfRziXH52hnWAXLBJH0eoECqvAxRt0schKqS6tDeHKNI5Fy/ShyHOqS89YNpPy8vgvt4I5UMN\ny2BD0WE+DVE+1UVvugI+qkjH1JUu0schjulQdsDL1KzfCgb7NEL5UDlNc9FM/xiifKp5frJ9\nxXtUkdJQdm7+McQxHcsO+Dpw8V/qfZHqhEq55hAVUo3Gf8kfVaRz+fXt4xB9Ou0vO+plB72Y\nUldvhEqhhvJr9v0QFVJ1aQxRpLnGH+6PRVqVXssvf05P9UaoEuqyFVS6qu+HKJ/qkJ5sV0Ch\nIqXL726eiv9tHXPpHeX7EaqEOva59E7L+yGKp1qPZlCkLUNMaVd0yCkX37D7PELpUPO8L79t\n92mIgql2ywkEirRpiMLDdqVX6X+OUPx3OZU/2vB5iGKp9uu2MUXaNETRYcddNxZ8+C9HqHmg\nsN4QxcZMr+we0uyRfj905am5nq0YS57rOxU/lPFxhPKhXkYo+Er7aYjSqSjSliGG5bjQZQ+2\n3DG1sXiPPo1QPtR62cHUl9xH+jRE+VQLNu1+PcT6fbpe0VXwQO7e/O/cdyNUCnW7EK7on4i7\nIWqlminSH4a4fp+GnHYlDz7ZbzB8N0KlUJfXh+Ij3A1RLVWcIgFxUCTAAEUCDFAkwABFAgxQ\nJMAARQIMUCTAAEUCDFAkwABFAgxQJMAARQIMUCTAAEUCDFAkwABFAgxQJMAARQIMUCTAAEUC\nDFAkwABFAgxQJMAARQIMUCTAAEUCDFAkwABFAgxQJMAARQIMUCTAAEUCDFCkFlX4sEP8DjPS\nIorkDjMCGKBIgAGK1JBTl1J3mq+bdnefb37cpVz6I8DxPYrUjuO1OcePRerX792jn542itSO\nnM7z/JR2dwcb9pf+nFI3zVOXTg99cuooUjvSa1deitQtr0N9mi6LU+of9bwwU6SWDCn15/Oy\ndCvS2qPXbTym8pH47TfkkC91yeNLkbq0X75RJA/47TflNOxe9pHGnIb1v9EgD5iE1lyP2L31\n6LKPxGGGx6NI7dilp7ejdq89uvyXfF6OjXOw4ZEoUjuerrtCz2uR7vaMuvX7su+Eh6FIDVmv\nbLj06EORlisb0p4ePRRFAgxQJMAARQIMUCTAAEUCDFAkwABFAgxQJMAARQIMUCTAAEUCDFAk\nwABFAgxQJMAARQIMUCTAAEUCDFAkwABFAgxQJMAARQIMUCTAAEUCDFAkwABFAgxQJMAARaSh\nsHEAAAAWSURBVAIMUCTAAEUCDFAkwABFAgz8B5mypc25Z6w4AAAAAElFTkSuQmCC",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAABlBMVEUAAAD///+l2Z/dAAAA\nCXBIWXMAABJ0AAASdAHeZh94AAAXTUlEQVR4nO3cgVbbWK+AUfP+L33/Oy3EcQKCoFPZ0t5r\nTQdCSHOkfEMI025vwK9t1XcAOhASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBAS\nJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBAS\nJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBAS\nJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBAS\nJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBAS\nJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBASJBDSLM/3\nvf3oYbB9+k6ubXe3tv+s+71+7cz3jVd9/lB/uu/fPAjWfe62v8bpH6env4O8oENI291VTv84\nPf0d5Of+Pgv6+HX/r+MVbx86fvbtqdXhje3+arvPfNv9vn/f2P5eZdvf0J/P3f96dyu7f929\nc2bnv4f83MdDd3u7exw/+fL0UNfHNe8+d//Gpze4Pbn69uTe7D/3dmN/b+L+a9AtyVN/hySk\nnnZfN568s7/W87qefHS7/8jnN/jJ5319Q+8X7N57/Ip06sfqqe8cL3r6UN+2Z4/7F0P6/AZ/\nEdKT+/H0rp3Rme8br3oM6f1BH4d0/83P9vdp1XZ4/H9+g4er70P6eA17/1vsn7R9+hXp7fGd\nkznzfeNVv3lqd3/h/qM/eWr3cJXHXp/c8ttn3yMdr3Y+Z75vvGpBSE+fkb341O7hc5/lsn36\nzimd+b7xqo9H9P6ft+O3NPcfPX7u16/aPb3B28P9dvXt9u/DvTlc8+khtqfvnNGp7xwv2v8c\n6e3205rHp1fbk8fx936O9HGDu09+dvVt9+/Dz5F29+yLU7zdf+pZnfveUeebj4zo8T3lATbl\nnPyUkH5kyjl5t70LLv12SM9v8Ic3c3lTzskiJ//W5Z8xBn5FSH8YA78ipD+MgV8R0h/GcAob\nBVI3mHljvOq6a7jwVyQh9XPdNQhpwY3xquuuQUgLboxXXXcNQlpwY7zqumsQ0oIb41XXXYOQ\nFtwYr7ruGoS04MZ41UXWcPwDR8/fTP4JzTJC6ucaa9j9QbtbM+93/fEP1J6dkPq5xBq2Wygf\nzdy+DL3/uvvwyQmpnyusYXs7hrT/o+tC4gQusobjV6T9Hd92/1zjQELq5yJr+Cqk+79J5QoH\nElI/F1mDr0j/5sZ41UXW8EVIvkfiBC6yBiH9mxvjVRdZg5D+zY3xqous4eEHsk++R/IDWepc\nZA1P/heh26/+FyHKWUMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9\nWEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81\nFBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMB\nIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBS\nP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVj\nDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQ\nQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE\n1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9\nWEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81\nFBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMB\nIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBS\nP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVj\nDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQ\nQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE\n1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9\nWEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81\nFBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMB\nIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBS\nP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVj\nDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQ\nQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE\n1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9\nWEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81\nFBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMB\nIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBS\nP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVj\nDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQ\nQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE1I81FBBSP9ZQQEj9WEMBIfVjDQWE\n1I81FBBSP9ZQQEj9WEMBIZ3RF4M8fujJVT//7O1/3t/4uGDbdh/d7PA1Qjqjb4b0ydU+/ezt\n/YN3b3zndyUipDNaE9L2/uutISFlEdIZ/XmQvz/NuntCtpvx9v7+9ue6x+dt9zd3fGf77AO8\nQkhntO3+uX8yth2/In1cuO3fu8/t4bZvn7P7qO+QfkNIZ/SRwtM39ld77G27+1rzkMbdF627\np3ePz/X4PiGd0TGbbXsxpMeNbA9veIKXQUhntL2/Fv33+dePviIdXtJ+uOWHN4WUQEhntD08\nzH/6FWl3pe1ZPG9CyiWkM8r7Hun+c+6e8D08f3y4fb5PSGe0vR0r+fpVu7f9L4dX7Q7Xv7u1\n++YeAuT7hHRGf+P5eIVtu11yF9K2PQnp8edIt6u/f/e0/xnV4+/IzwmpH2soIKR+rKGAkC5n\n2+5f4X68wr+8N/whpH6soYCQ+rGGAkLqxxoKCOnnNn6neoErCOnnhhxzGfOLDJnQkGMuY36R\nIRMacsxlzC8yZEJDjrmM+UWGTGjIMZcxv8iQCQ055jLmFxkyoSHHXMb8IkMmNOSYy5hfZMiE\nTnfM/c84t8dLz/Yj0HPdmxc8/iH92xu3Hzn/YuyXn9D3nO2Y+z/Zum0Pl57uD76e6s684PFv\nlnn2Vwf+ZuxXn9A3neyY292v2/HS8/1VDGe6Ly/4mPHjGx/X+OXYLz6h7zrZMfd/VvxNSKtt\nz2a7Ha4hpO842TGfh/Txn0Uhpfvie6SPd4QUO9kxbyu7f1q+bb/f6BJnui8v+Tqk7eHX13+D\n3s52zP/62e7+S/i2q+r9w6dxpvvyEiHlON0xt//+Wq771e2f8G3H74Zrnem+vERIOU55zO3t\n/o/NHRZ5pvt8pvvyEiHlONkxn30l8qrdSl+G9Pw/ZC/+Br2d7Zh3rzF88j3SP79TXzjVnXnF\nd0L61dgvP6HvOd0xH/4Xoe3uUv+LULLPQjq82uN/EQoMOeYy5hcZMqEhx1zG/CJDJjTkmMuY\nX2TIhIYccxnziwyZ0JBjLmN+kSETGnLMZcwvMmRCQ465jPlFhkxoyDGXMb/IkAkNOeYy5hcZ\nMqEhx1zG/CJDJjTkmMuYX2TIhIYccxnziwyZ0JBjLmN+kSETGnLMZcwvMmRCQ465jPlFhkxo\nyDGXMb/IkAkNOeYy5hcZMqEhx1zG/CJDJjTkmMuYX2TIhIYccxnziwyZ0JBjLmN+kSETGnLM\nZcwvMmRCQ465jPlFhkxoyDGXMb/IkAkNOeYy5hcZMqEhx1zG/CJDJjTkmMuYX2TIhIYccxnz\niwyZ0JBjLmN+kSETGnLMZcwvMmRCQ465jPlFhkxoyDGXMb/IkAkNOeYy5hcZMqEhx1zG/CJD\nJjTkmMuYX2TIhIYccxnziwyZ0JBjLmN+kSETGnLMZcwvMmRCQ465jPlFhkxoyDGXMb/IkAkN\nOeYy5hcZMqEhx1zG/CJDJjTkmMuYX2TIhIYccxnziwyZ0JBjLmN+kSETGnLMZcwvMmRCQ465\njPlFhkxoyDGXMb/IkAkNOeYy5hcZMqEhx1zG/CJDJjTkmMuYX2TIhIYccxnziwyZ0JBjLmN+\nkSETGnLMZcwvMmRCQ465jPlFhkxoyDGXMb/IkAkNOeYy5hcZMqEhx1zG/CJDJjTkmMuYX2TI\nhIYccxnziwyZ0JBjLmN+kSETGnLMZcwvMmRCQ465jPlFhkxoyDGXMb/IkAkNOeYy5hcZMqEh\nx1zG/CJDJjTkmMuYX2TIhIYccxnziwyZ0JBjLmN+kSETGnLMZcwvMmRCQ465jPlFhkxoyDGX\nMb/IkAkNOeYy5hcZMqEhx1zG/CJDJjTkmMuYX2TIhIYccxnziwyZ0JBjLmN+kSETGnLMZcwv\nMmRCQ465jPlFhkxoyDGXMb/IkAkNOeYy5hcZMqEhx1zG/CJDJjTkmMuYX2TIhIYccxnziwyZ\n0JBjLmN+kSETGnLMZcwvMmRCQ465jPlFhkxoyDGXMb/IkAkNOeYy5hcZMqEhx1zG/CJDJjTk\nmMuYX2TIhIYccxnzi1xvQl/c4+OHtk8/8vipf9y998Kd68osIteb0DdD2j79yDc+/3pTWcxA\nItebkJAKGEjkehP67x5/PPH6eAr2///eHWZ7f3/785H91R5v7vDm9YaymolErjehbffP7Zf/\n/jl+Rfq4cNuOn/j3Sg+f8n6x75DumEbkehPa3u/00zf2V9v1tj272iGWw0sT15vMQoYRud6E\njj3898XjlZA+f43v+fuTmUXkehPa9q9Vv1f03ZAOL3Ifb/jL9yczi8j1JrTt7vRrT+12V9q3\nJKTPmUXkehPK+x7p/p2HpK43mnXMInK9CT28AHd7SS4M6fiq3fF2j5/MX4YRud6E/pbx96vG\nx4+PDj9Herv9HOltF9Ljz5Hubze+2kymERkyoSHHXMb8IkMmNOSYy5hfpNmEtu3+Fe6Py0vu\nTR/mFxkyoSHHXMb8IkMmNOSYy5hfZMiEhhxzGfOLVE5o66lwosu0PFSq0pAKf+91ep6KgJCy\n9TwVASFl63kqAkLK1vNUBISUreepCAgpW89TERBStp6nIiCkbD1PRaBrSH9/MHr4GenHmwt/\nbtohpN3YjpP6+HNg//5enVrTkO7+iOv2cOnKPwHb5hH2dFLb8vldVM+Q7v/She146dK/k6HN\nA+xWy7a77NmljAhpe7hUSN/w8aXnfoJCem5SSJ7a/YSQfqR/SPfP8L3Y8E3PRri9CekzPUP6\ns+6HffuK9H27L+fb7m0hfaJpSLu/nev+mZ3vkb7p46vQYZBCeq5rSLfbF9JLtoe3b39wUUiP\neoa02/T25FIhhe5eqDlGJaRHPUP6ZN2+R/q24+sL9x9YOb+LahrS7vW5/W/mVbvvur3ifXjN\nZru7lHddQ6rT81QEhJSt56kICClbz1MREFK2nqciIKRsPU9FQEjZep6KgJCy9TwVASFl63kq\nAkLK1vNUBISUreepCAgpW89TERBStp6nIiCkbD1PRUBI2XqeioCQsvU8FQEhZet5KgJCytbz\nVASElK3nqQgIKVvPUxEQUraepyIgpGw9T0VASNl6noqAkLL1PBUBIWXreSoCQsrW81QEhJSt\n56kICClbz1MREFK2nqciIKRsPU9FQEjZep6KgJCy9TwVASFl63kqAkLK1vNUBISUreepCAgp\nW89TERBStp6nIiCkbD1PRUBI2XqeioCQsvU8FQEhZet5KgJCytbzVASElK3nqQgIKVvPUxEQ\nUraepyIgpGw9T0VASNl6noqAkLL1PBUBIWXreSoCQsrW81QEhJSt56kICClbz1MREFK2nqci\nIKRsPU9FQEjZep6KgJCy9TwVASFl63kqAkLK1vNUBISUreepCAgpW89TERBStp6nIiCkbD1P\nRUBI2XqeioCQsvU8FQEhZet5KgJCytbzVASElK3nqQgIKVvPUxEQUraepyIgpGw9T0VASNl6\nnoqAkLL1PBUBIWXreSoCQsrW81QEhJSt56kICClbz1MREFK2nqciIKRsPU9FQEjZep6KgJCy\n9TwVASFl63kqAkLK1vNUBISUreepCAgpW89TERBStp6nIiCkbD1PRUBI2XqeioCQsvU8FQEh\nZet5KgJCytbzVASElK3nqQgIKVvPUxEQUraepyIgpGw9T0VASNl6noqAkLL1PBUBa4cEQoIE\nQoIEQoIEQoIEQoIEQoIE/zSkbdsOb+zf2f7zL+/Prz0eZ3vy8audihf8yxVv77/fxxsPl17L\n+38B3nZJPX787rQ09Q83/PGou8vm4bF4Hdv+JNvukruP7z5MXxUhPfn3dsGH2v1/Era34ym2\nNyHNUfLU7m33HOgW0uW+Q3oI6U1IY5W/2PD8Cd9FfBXS9iakQaq+Ih1ebPjkVa+z+yKkw+sM\n1zoXP1b2PdLujW17cuklfB7S8TWUa52LHztBSJ9fenpfhLS9/1hMSCOUvfx9/8i76APuyxcb\n3nxFGqTie6Tt4ZulwzcUl/H4Q7DHkK54Ln6s9lW77eml1/HFi5D79y53Ln7MiiGBkCCBkCCB\nkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCB\nkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCB\nkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCB\nkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCB\nkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCB\nkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCB\nkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCCBkCDB/wE2oLW/agtg\nJAAAAABJRU5ErkJggg==",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Example of tree pruning:\n",
    "mod.adr.tree.cv <- cv.tree(mod.adr.tree );\n",
    "plot(mod.adr.tree.cv)\n",
    "\n",
    "\n",
    "best_size = mod.adr.tree.cv$size[which.min(mod.adr.tree.cv$dev)];\n",
    "\n",
    "mod.adr.tree.pruned=prune.tree(mod.adr.tree, best = best_size)\n",
    "\n",
    "plot(mod.adr.tree.pruned)\n",
    "text(mod.adr.tree.pruned, all=TRUE, cex=.8)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.3 Split-sample validation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than constraining the complexity, we can directly deal with overfitting. The issue of overfitting is when the model over fits the random patterns in the data set, where these patterns will not be seen in the new observations. A straight-forward way to address this issue is to employ **independent Validation**. That is, the chosen criterion will be evaluated on an independent testing set, where the model is trained on a training set. \n",
    "\n",
    "In practice, we may not be able to collect a new set of data from the same population using the same sampling scheme. However, we can create the training-test sets from one data set. We can *randomly* split the data set into two sets, where $q\\%$ of the data belong to the training set and $(1-q)\\%$ the test set. When the split is random, it is reasonable to believe that the two sets share the same data generating mechanism. The split needs some careful thoughts when there are structures in the data (e.g., time series data). \n",
    "\n",
    "After splitting the data set into training set and test set, we can explore any models in the training set, and to evaluate their performance in the test data. \n",
    "\n",
    "\n",
    "Split sample validation protects against over-optimism due to fishing through hypotheses on the training data. However, split Sample Validation does not necessarily give a great estimate of performance \"in the wild\". For instance, \n",
    "- If all samples were run in one lab, your data cannot say how well things generalize to other labs\n",
    "- Effects found in a subpopulation defined by one criteria may not reproduce if the subpopulation is defined by a different (but nominally similar) criteria."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example** Split Sample Validation - Lasso\n",
    "\n",
    "For choosing $\\lambda$ in the lasso...\n",
    "- Split your data into a <span style=\"color:blue\"> training</span> set and a <span style=\"color:orange\"> validation</span> set\n",
    "- Choose candidate $\\lambda$-values ($\\lambda_1,\\ldots,\\lambda_m$)\n",
    "- For each candidate $\\lambda$-value, fit your Lasso model on the <span style=\"color:blue\"> training</span> data to get a set of coefficients\n",
    "- Evaluate each of the $m$ sets of coefficients on your <span style=\"color:orange\"> validation</span> data.\n",
    "\n",
    "<span style=\"color:red\"> Final Step):</span> Refit the Lasso model to **all the data**, using the best $\\lambda$-value. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "More Generally\n",
    "\n",
    "Suppose we are deciding between a <span style=\"color:blue\"> neural network </span>, a <span style=\"color:red\"> regression tree with depth $3$</span>, and a <span style=\"color:orange\"> regression tree with depth $5$</span>.\\\\\n",
    "- Split your data into a training set and a validation set\n",
    "- For each of our $3$ procedures, build a predictive model on the training data\n",
    "- Evaluate each of the $3$ predictive models on the validation data.\n",
    "\n",
    "<span style=\"color:red\"> Final Step:</span> Refit a predictive model to the full data, using the **best** procedure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.3 Cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The split-sample validation does not seem an efficient use of the data set. Only part of the data is used in training the model, and the other part produces estimates of the criterion with low precision due to limited sample size. \n",
    "\n",
    "How about \"rotating\" the roles of training and validation? For instance, we can split the data equally into two sets. We train the model on Set 1 and evaluate the fitted model on Set 2. Then, we train the model on Set 2 and evaluate the fitted model on Set 1. This way, all data are used at least once in fitting a model, and once in evaluating the model fitted from other data. In both cases, overfitting is curbed since training and validation happen on independent data sets. This procedure is known as **cross-validation**.\n",
    "\n",
    "In general, the $K$-fold cross-validation works as follows. We randomly partition our data into $K$ non-overlapping sets, a.k.a. folds. Suppose that there are $K$ folds. Each time, we then use $1$ fold as the test set, and all $K-1$ folds as the training set. After iterating through all $K$ folds, we use the average criterion (e.g., loss function) from the $K$ iterations as the final performance measure of the procedure. We will pick the procedure with the best performance and refit the procedure on the full data set. \n",
    "\n",
    "A special case of cross-validation is the leave-one-out validation, where 1 observation is left out as the test set each time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "6"
      ],
      "text/latex": [
       "6"
      ],
      "text/markdown": [
       "6"
      ],
      "text/plain": [
       "[1] 6"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAANlBMVEUAAABNTU1oaGh8fHyM\njIyampqnp6epqamysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD/AAD///+Vwh5YAAAACXBIWXMA\nABJ0AAASdAHeZh94AAAgAElEQVR4nO2dbWOquhJGg6i31iqH//9nr/jSYrdSjJPwTLLWhx57\ngJmEPauOETH0APA2YekBAJQAIgEYgEgABiASgAGIBGAAIgEYgEgABiASgAGIBGAAIgEYgEgA\nBiASgAGIBGAAIgEYgEgABiASgAGIBGAAIgEYgEgABiASgAGIBGAAIgEYgEgABiASgAGIBGAA\nIgEYgEgABiASgAGIBGAAIgEYgEgABiASgAGIBGAAIgEYgEgABiASgAGIBGAAIgEYgEgABiAS\ngAGIBGAAIgEYgEgABiASgAGIBGAAIgEYgEgABiASgAGIBGAAIgEYgEgABiASgAGIBGAAIgEY\nsJRI4cKr28w3FZxs4CvTpm4Twubw+JCpbbtVaLbdk1y758OY2BR72JssJNJhogImtplvKjjZ\nQNdk2tSch/HYlolt2/Om5rFJh+fzmtgUe9i7LCbSOmab+aaCkw2sn1eO6aZt2Aw/Hg5mYtsh\nbLrhaWLz6LhD83QYE5tiD3ubhUTahY+YbeabCk524vP5k5XtpiYMTyovb1tf/ufDbbvQPhvG\nxKbYw95nMZF2MdvMNxWcrO+PzyvHfNNAaCK3PYoZtk/EnNwUe9j7LCTSOuw3p5eZr24z31Rw\nsr5vw/FZ5Zhv6of27bnVU9u60D74v4dnz3CTm2IPe5/FRDrz6AxObTPfVHCy/iN8Pqsc803n\nru+Z0JPbhqfV/eMNE2U/+cQYd9ibLCRSOP2b9N3jv1MT28w3FZzsvBDxbPXQeNOJ3bp5+oJt\nalt/bJ4tmCDSXLqwitlmvqnIZKthVflx5ZhvurCZ6N+ebuuax8+nPSK9kt38fGQ9+crJNueG\n6eFB5puudBMrCk+3tc//4iDS/OxLl1u5ycI36TfNGOKTbcdVe4wJh0hXLm8uHB++TTexzXxT\nucmyinQbx6Nnl6lt+yfrJLeMMZvqEmk7LOOcXiU/Wq+Z2Ga+qeBkZzI9056vXujWD18HTWw7\nTnqESDPoLhdgPVwUndhmvqngZGdytazNxDL8822b6WYRkWbQbZuweraQ83yb+aaCkw1ke+03\nNY6n2/541YVIALWBSAAGIBKAAYgEYAAiARiASAAGIBKAAYgEYAAiARiASAAGIBKAAYgEYAAi\nARiASAAGIBKAAYgEYAAiARiASAAGIBKAAYgEYAAiARiASAAGIBKAAYgEYAAiARiASAAGIBKA\nAYgEYEAGkQKAMyKq3F6cBVIAWIJIy1HHLGVIe7oRaTnqmKUMiAQgDyIBGIBIy1HHLGWgtSuV\nOmYpAyIByINIAAYg0nLUMUsZaO1KpY5ZyoBIAPLkFOm4Cc1H3+9WodkmSgGwDBlF6prhItnd\nx/la2TZJCl/UMUsZimnttuH0PLRtwqbru/Nj+xS+qGOWMhQjUnM+MITu/J8mRQqAhcgoUgg/\nP/vpD0IhEjhjgWek4WfHM1Its5ShmNbu9hpp210f26fwRR2zlKEYkVi1g3LhfSQAA7iyYTnq\nmKUMxbR2WikEqGOWMrx3uv93h0n06AF1m9Nro/01CMvf4I8HBl3JvtgQ1pcgiAT+kBBpG3Yn\nm3bNecEOkWqZpQwWp1tCpMsbsv2xWR0RaaCOWcpQjEg3d7q2RSRwiYRIq8vlqsOjFpHAIxIi\n7cLm+ugYWkSqZZYyFNPa9dtve/YPvgbjze/IcEgds5ShHJH6w/r26LjhGQn8ISKSUgqA10Ek\nReqYpQwFtXZKKQSoY5YylCkSq3bgEEQCMEBPpMVTCFDHLGUos7VbPIUAdcxSBkQCEAGRAAxA\nJEXqmKUMtHalUscsZShGpBBmX5dKiYEkEiLtEAmcIyFSf2im769qkMIVdcxShldP96MbcGmI\n1B+m7/htkcITdcxShsjT/b+nv7wb/Y1//104pE4BYIqkSEIpAGaBSPLUMUsZSmrthFIIUMcs\nZUAkgOVAJAADEEmeOmYpA61dqdQxSxkQCWA5EAnAAESSp45ZykBrVyp1zFIGRAJYDkQCMACR\n5KljljLQ2pVKHbOUAZEAlgORAAxAJHnqmKUMtHalUscsZUAkgOVAJAADEEmeOmYpA61dqdQx\nSxkQCWA5EAngRf68SzEiKVLHLGWYfbr/9/w3RFKkjlnKgEgAOUEkAAMQyRl1zFIGWrtSqWOW\nMiASQE4QCcAARHJGHbOUgdauVOqYpQyIBJATRAIwAJGcUccsZaC1K5U6ZikDIgHkBJEADEAk\nZ9QxSxlo7UqljlnKgEgAOUEkAAMQyRl1zFIGWrtSqWOWMiASQE4QCcAARHJGHbOU4eHp/vOW\nkIikTx2zlGHqdE/YgkgAc0EkAAMQyTN1zFIGWrtSqWOWMiASQGIQCcAARPJMHbOUgdauVOqY\npQyIBJAYRAIwAJE8U8csZaC1K5U6ZikDIgEkBpEADEAkz9QxSxlo7UqljlnKgEgAiUEkAAMQ\nyTN1zFIGWrtSqWOWMiASgCEP7huESABxzLYFkeSpY5Yy3J9uRCqHOmYpAyIBmINIAAYgUjnU\nMUsZaO1KpY5ZyoBIAOYgEoABiFQOdcxSBlq7UqljljIgEoA5iARgACKVQx2zlIHWrlTqmKUM\niARgDiIBvMj/HnyWD5HKoY5ZykBrVyp1zFIGRAIwYMIPRAKYCyKVSh2zlIHWrlTqmKUMiARg\nAK0dgAGIVCp1zFIGWrtSqWOWMiASgAHltHbdtjn9/FiF0H4mSgHwjGJEOjYh9N3px0CbJIUv\n6pilDMW0dpuw7k4/NseTU5uwTZHCF3XMUoZiRAqhu/44dXmhSZEC4MyLH5zwJtLpRxNGv5in\nABgz1w9fIm3Coe8/hh/DM9Lki6Q6RKpjlktyV/fFtHaH0GwP/bo5mbRfhX2KFL6oY5ZLUqZI\n/f66YjfwkSYFwIgyW7sTn5vVYNH645gsBcA3xYoklEKAOma5JIW2dlIpBKhjlktSqEjdJoT2\nusjA8jekp8zW7np10PoSBJEgOWWKtA27k0275vwOEiLVMstc/HkxQzGt3fWihmOzOiLSQB2z\nzMuEBMWIdHOna1tEgjTE+eFLpNXletXhUYtIkIQaRNqFzfXRMbSIVMss81JDa9dvv+3Zh39F\nCmNiU7iijlnmpQqR+sP69ui44RkJElBDa6eVAooEkXKnEKCOWealjtZOKYUAdcwyL9WJxKod\nvIvdN1oiEkCcBKWItHgKAeqYZXLmSlBma7d4CgHqmGVyMor0339PR4FI4Jx8rd1//z03CZHA\nOdlE+u+/CZMQaTnqmGVysrV2iCRKHbNMTr7XSCKtXQizr0ulxGA2GZe/NRYbdogECajvfaRD\nM/2tSAYpXFHHLJNT4ftIh+lvRbJI4Yk6ZpmcCkU6dXeH1CmgNupr7aRSgEceXKaKSMumEKCO\nWSYgToKSWjuhFALUMcsEIJJQCvCLhQSIBNWDSEIpBKhjlgmgtRNKIUAds0wAIgmlAL/Q2gml\nAL8gklAKAeqYZQJo7YRSCFDHLBOASEIpwC+0dkIpwC+IJJRCgDpmmQBaO6EUAtQxywQgklAK\n8EvO1u7+Ng2IBE6xu1N+lEi/bhyESIrUMUsTLBYKYlq737eyQyRF6pilCYgUByUGd9gvXdPa\nQYUsJhKLDQ6oY5YmLNXaTW96Hn0WiGREHbM0AZHioMTgjuVaO0SCgkCkOOoQqY5ZmkBrF0cd\nJVbHLE1ApDgoMbiD1i4ORII7ECmOOkSqY5Ym0NrFUUeJ1TFLExApDkoM7qC1iwOR4A5EiqMO\nkeqY5cu8+FE+WrtlUwhQxyxjMX82QSSokcRtGSJBHSCSAXWIVMcsY6G1M6COEqtjlrEgkgGU\nGNDaGYBIgEgG1CFSHbOMZbHW7v52J4ikTx2zjGUpkX7dgAuRwDcLtXa/bwmZTaT9Opz+x/r4\nepzZKaBGKhOpDWEQKTSmJtUhUh2zjKWu1m4X2m4QaRc2rweal6Jc6phlLHUtNjSh6weRLj/M\noMSgruXvc1uHSPAeDz44UZlIq+sz0iGsXg80L0W51DHL+SSWQPoSoetrpH0Tdq8HmpeiXOqY\n5XxqFqlfhwvt63HmpoBKyNqWqYl0fh8prD9fDzM/BdRB3SIloQ6R6pjlfKpu7dJQR4nVMcv5\nVC1St21OP5tt93qcuSmgEmpu7Y7N9V0kLhGCd6lZpDZshueibhvWrweal6Jc6pjlfGpu7b4v\naODKhtepY5bzqVmk4Vq7gQ6R4F1qbu22of06/eerDdvXA81LASXy4n2Jixfp8nkkrmyIoo5Z\nTmFR22W0dn3/OVzZ0JpeaVdLidUxyykQKTGUWB1kFUm7tUsDItUBIiWmDpHqmOUUKq3dxF0a\ncon0sbquNrD8/TJ1zHIKEZGm7huUSaSPEBAJYtFo7X7fgWuZN2SN1+v+TQHlgkjfj9OUfB0i\n1THLKWjtbqyD7ecnHqQolzpmeSPVdyyXsdhwbM6XCFlTV4lVhbkfZSx/BxYb4CUQ6eEhiPQG\ndczyF8uJxCVCpVLHLH+BSO8cIpgCloHWbvqQLz5qDnNApMeHbHmNFE0ds/wFrd3DQ3482r8e\naF6Kcqljlr9ApIeHNOGzb8Px2AbTt5OqLLHy+PM9WFq778dhuHB13x9sP2uOSAWRuLbLEWk/\nXLjKa6TXqWOWMiJJt3brU2t3DKv+C5Fep45ZItKcQ/aDQOc7CfFlzPAYFZGkW7vTC6TTj02w\nva0dIpWEpEjzv7o8k0hpqEOkOmYpI9L4dP/6BBIiuabAWS5wA9UYkX5/JnZZkYaXR1z9DQ/I\n+bIlJhkigQvURaK1K4pyZykp0t3pVltsSHLHhpJLbEy5s9QXSW35O7SmF6s+SgH+kBRJ+n2k\n4T6rW/vbnyCSHx6s0yHSLO4OOQ73LF59GLd4dYhU0iyXq+0yWruB47YJxi1eSSX2nJJmiUhP\nt7xyyI7l78pxIJJ2a9ffurvP1wPNTwFK2F2+gEjfnC1qtsfXw8xOUSy+Z6lS22W0dsOq3YZV\nuzh8zxKR7nj7fSTTlu5RClieVDfAp7W7Ebfs/efKBCJJIlnbZYjU9/v1oMX6pRdJGiL9z5wX\nB2A4S/sBP9pRsrbn7vifdGt3+Zj56f81f5sU7pmd4sLbVf6kiiz/9V5M/eq5f2cuLw94dox3\ndswo0n//hbgvQcoj0i603eDEbsY9G76ad0S64K4XeLjjbOEWqO0FO6WUO05/Aml5kZrQXRq1\nOW/IduvQnp+44ls7d6VSaF0KxShDpMuH+/q597X7DOFzzs51iBSeb9IcsGSMmTuKt3ar6zPS\nIaxmHXtsw7pDpAuIlHPA2osN19dI+2a42+osPkKzRyTdGAx4EZH69XXpYP6tvw+rv69wRaTF\nYjDgZUQ6v48U1i9d37BBpDO0dlkHLN3aJQKRopK5OzuIFEW3+fkAYNXvIynHYMBLiLTfDC94\n2nl3begub8hevmwWkURjMOD8Ih3b78sU2hnX2m2Hlb1u15zXJRCJ1i7zgGVbu9MzzGo/XP59\n/FyF5u/jmsuBx2Z1RKQBREo84Pu7QMqKtB2tebfh4+/jrgd2bYtIsjEKGvD8i4KWFWkVfvq5\n44w3koarIK6PWkRSjVHOgH9fXicr0p0LM661+7lC/KQdItHaZRZJtrV7VaRTL3jbaf/g4oZZ\nn7FwVyqItNyAf7V25YjUH9a3R8cNz0iaMUoa8MRXt/gW6eUU/+CuVEqqS80YCw54zFsizf7E\na2yKf3B35qd2pLXLOmDh1g6R3toRkRDpbVi1E43BgBHJekd3yRhw5DdaOhYpMoW7UpnakdbO\nesCT37FcZmsXmcJdqSBSxgFP3yoIkUa4KxXPdekjxmyR7Ac8BpHSxvBclz5ijH+bbO0QaYS7\nUpnakdbOfMBTiw20diPclQoiyQy4GJHeu4n+BXelUm5dqsRYcMBjMl7ZsEMkBzEYsLxI/aGZ\nexvJOkSitcs64GJau+EO4ds3U7grFURKHWP2ZyUKEunU3R3eS+GuVNzVpbcBm3xbi5BIX+tH\n/zcWRFoshrMBv3BjBm2RtnyMInpHWrusIkm3dj8e7V8PNC/FHe5KBZESx5jf2kmL1ITPvg3H\nYxtm3bU4JsUd7krFW136G/DsxQb7AY95U6Sho/s4PRsdXviCpBdT3OGuVNzVJQNeTKT9cE9v\nXiO9viOtXdYBS7d261Nrdwyr/guRXt8RkRDpxvnrYM9fSrF5un8EdYgkGYMBLyLS6QVSf/4u\ny5mXLMSkGOPuzFOXBQ94DB+jSBtjakdau6wDlm7tEoFIUcncnZ24GJG3CpIVaXh5xA0ilZLV\nMeDYz5PbD3gMIqWNoV+X3gYcfYcTWZESUodItHZZRXqvtfvfHf0/IFLaGIhkHiO2tbN4jfQc\nREobQ78u/Q04crHB5Ow85e2PUTS8RpJJxoDdirRlsSF+R1q7rAOWbu3CcMGqPYgUlczd2UGk\n78dpXjDVIZJkDAa8iEjb0L0e4LUUd/g789RlSQN+zruLDW17fD3CaynG+DvzEzvS2mUdsHRr\nN3yOgsUGRPIwYGmRPli1E0rGgJMne8rbNz9h1U4nGQN2KxKrdm/sSGuXdcDirR2rdoiUPMbU\nRUFliNR/tKY3tHuUYoy7UpGsS5lkM2NMXqaa9+w85e3WjsUGnWSFDnj6gxOINDPFHe5KZWpH\nWrusImm3dmlApKhk7s7OzBg2rR0ijXBXKop1qZNsbgyTxQaTs/OUt0X6HG4Puf58Pcz8FCPc\nlYpkXcokczfg57x9rd31FZLpPfQrEYnWLuuApVu7XWiGL0baG1/hgEhRydydHUS6sbp+Kewh\nrF4PNC/FHe5Kpdy6VImRN9lTrC4RYvlbIBkDdivSzzNS83qgeSnu8HfmJ3aktcs6YOnWjtdI\nb+yISIj0Dat2QskYsGWMv+6tes/77yOteR9JJRkDThBjJlzZkDbG1I60dlkHHNnazQSR0sZA\nJJkBI9IId6VSbl0mjjH7MtXUA57J2x/sW/ExCplkBQ14/gcnyhCJuwi9sSOt3dNNvz+CZJFM\nurXjLkJv7IhIiPT9mLsICSUraMC1tXZr7iIklKykAVe22HBsuItQ9I60dlkHLN7asdiASIvG\nQKSZKe5wVyrl1qVKjNTJZsIbsmljUJfeBzwTREobY2pHWrusA5Zt7YZujtbujR0RCZHO/0Uk\nsWQMOEGMmdDapY1BXXof8EwQKW2MqR1p7e5+mbqdqkUy2dbuxlcbmq3tBQ6IFJXM3dkZ/zJ5\ng++yRTqcDNr1h/MrpMbUpDpEkoyx1ICnv3JiwQHP5A2Rvs4Gbdvm0Hdt2MZkf3lU7koFkRDp\nr0PO8mxDGO7H1XFfu9d3pLUb/1Jva3dZ8b6ue7P8/fqOiHT3S7WLDYiklowBJ4gxE0RKG4O6\n9DTg1+4JeQcipY0xtSOtXdYBz27tonhLpDveHsqcUbkrFUSSGTAijXBXKuXWpUoM+2RRcIlQ\n2hjUpb8BR4FIaWNM7Vh9a2fyZeWzd5Rt7RKCSFHJnJ2dybdgEcmCOkSSjJEv2fRFQbR2FiDS\nYjEQKQ5EShtjakdaO1q7xCBSVDJvZ4fFhtTUIZJkDAYcByKljUFd+htwFIiUNsbUjtW3dnkH\nTGs3wl2pIJLMgBFphLtSKbcu42LYf1uL/YCjQKS0MRDp7pcE3x+GSDEpNM985I71tXYpvtFy\n9o60diPkS+WVHREJkRJTh0iSMWjt4kCktDEQ6f4XFhveOsQsheaZj9yxvtYuRYzZO9LajXBX\nKogkM+CHIr1x36B7ECltjHLrUiWGSbL3QaS0MeqsS3cDfh9EShtjasc6WrvUdyKevePUa6T3\nQaS0MaoXKfm98REpJoViqegkExxw+m9robWLSSFz5iWTCQ4YkUwPMUshc+Zp7Wjt7kCktDGq\nF4nFBstDzFJIlopMMgZMazczhcyZl0ymMuCsd9lCpJgUMmee1m6qmct530dau5gUIqWCSBOb\nXvjQESIlpg6RJGOUK1I5rV23CaHdX4NMRkGkxWKU29oVI1LXnL/Zb30Jgkiltnaqiw3FtHbb\nsDvZtGvacxBEKkmkyDeLECmG5nLgsVkdEUk2RlSy2MsXZM7w+2QU6eZO17aIJBsjJln0BXWL\nDNjqM7H3ZBRpFbrboxaR+oJaOx8ipV1tzijSLmyuj46hRSTnIt2J46K1K0akfvttzz4gkmiM\nmTv+UsfDYoNdG/eIrG/IHta3R8cNImnGmLej0fe/IlJi6hDJcWvnUaRyWjuDFIi04IDH5th8\nkTIiRcElQg5iPN/x3h2TL1KmtYuBS4Q8xPjlzvix/TdJIFIMXCLkobV72r+5F6mY1o5LhLKK\ndN97TSxWP31H6Jc6Cb6SBZGiUnGJUNoYE++RThjx/Gnn93OQ/Vey0NrFwCVCFjGeP5tMNGIT\nfvz129sDFj3D1pRzidDc5uWFLuf9GJGt3UxbJhqxSJFeWJpzJ1Ixrd0flwiFMU8izGzup5qX\nl7qcd2NMGxee7xdhyx9GRA1Y5WkYke559xKhuOY+7o9zgj/wc2v7jVH9hE/xFOpVJLNvE5vA\n05UNJrWdVSTzZH88T/ZjHEiQ9RkpLaWIpNnaTdsSItqyyWcTydURGZEKau3eTTFVYJqLDZN+\nhGebYm1BpPpEilu1M2nus/4zx1nroS5VYlTe2vE+kmiMcgecFletncNSmdpR8lq7ckUqs7WL\nTOGuVBBJZsCINMJdqZRblyoxaO1iUrgrlXLrUiUGIsWkcFcqtHYyA6a1G+GuVBBJZsDFiBTC\nnOtSp1O4K5Vy61IlxsMdc1xdd0/Wj1Egkn6MogackZyt3eFyu4Z3UrgrFVo7GZGKae1OJoXt\nmynclQoiIZLhITd24fBeCnelUlRdSsaosLUzSOGuVIqqS8kYiBSTwl2p0NrJiFRSa/d2Cnel\ngkiIZHiIWQp3pVJUXUrGoLWLSeGuVIqqS8kYiBSTwl2p0NrlGvCfFzPQ2o1wVyqIlHfAEyDS\nCHel4rwuHcRYrpu7A5HSxnBelw5iIFJMCnelQmsnIxKt3Qh3pYJIiGR4iFkKd6XivC4dxKC1\ni0nhrlSc16WDGIgUk8JdqdDayYhEazfCXakgEiIZHmKWwl2pOK9L6Rj5b8wwASKljeGoLhdI\nZjJgDRApbQxaOxmRaO1GuCsVREIkw0PMUrgrFX916W7AGiBS2hj+6tLdgDVApLQxaO1kRKK1\nG+GuVBAJkQwPMUvhrlT81aW7AWuASGlj+KtLdwPWAJHSxqC1kxGJ1m6Eu1JBJEQyPMQshbtS\n8VeX0gNWurruHkRKG0O7LpdOFhlDEURKG4PWTkYkWrsR7koFkRDJ8BCzFO5KxUNduhuwIoiU\nNoaHunQ3YEUQKW0MWjsZkWjtRrgrFURCJMNDzFK4KxUPdeluwIogUtoYHurS3YAVQaS0MWjt\nZESitRvhrlQQCZEMDzFL4a5UPNSluwErgkhpY3ioS3cDVgSR0sagtZMRidZuhLtSQSREMjzE\nLIW7UvFQl9IDlrrB9wSIlDaGWl1qJZsdQx9EShuD1k5GJFq7Ee5KBZEQyfAQsxTuSkWzLlWS\n0dolBpEWi6E5YH0QKW0MWjsZkWjtRrgrFURCJMNDzFK4KxXNulRJRmuXGERaLIbmgPVBpLQx\naO1kRKK1G+GuVBAJkQwPMUvhrlQ061IlGa1dYhBpsRiaA9YHkdLGoLWTEYnWboS7UkEkRDI8\nxCyFu1LRrEuVZA83efkE0j2IlDbG8nWpnCzxc1BOECltDFo7GZFo7Ua4KxVEQiTDQ8xSuCsV\nmbqUTEZrlxhEWiyGzICdgUhpY9DayYhEazfCXakgEiIZHmKWwl2pyNSlZDJau8Qg0mIxZAbs\nDERKG4PWTkYkWrsR7koFkRDJ8BCzFO5KRaYuJZPR2iUGkRaLITNgZyBS2hi0dnM2Zbnem9Zu\nhLtSQaT5mxKDSCPclYpMpySZzHUzdw8ipY2BSIhkeIhZCnelQmsnIxKt3Qh3pYJIiGR4iFkK\nd6VCaycjUloQKW0MREIkw0PMUrgrFVo7GZFo7Ua4KxVEQiTDQ8xSuCsVWjsZkdKCSGljINLj\nTT7vAjkBIqWNQWs3uWNOaO1G+CsVREIku0PMUvgrFVo7FZHSgkhpYyASIhkeYpbCX6nQ2qmI\nRGs3wl+pIBIi2R1ilsJfqdDaqYiUFkRKGwOREMnwELMU/kqF1k5FJFq7Ef5KBZGuvyx+LQMi\njZAulcWTeRhwqSBS2hiIhEiGh5il8FAqtHaaItHajfBQKoiESKkOMUvhoVRo7TRFSktWkb4+\n1mFgvf2KTOGhVBAJkVIdcqZbhR/auBQeSoXWTlOkYlq7bWg+D+dHx30TtlEpPJQKIiFSqkPO\nNOHw/fgQmqgUHkqF1k5TpLRkFCmEZ7/MT+GhVBDp9tviFzNkhGektDFo7WQoprU7vUbaH8+P\neI10BpGyUoxIfTtatVt1USk0S0UlmeaA6yDv+0jb8/tIzfqD95FUYyBSHFzZkDYGrZ0M5bR2\nBik0SwWRECmnSN0mhHZ/DcLyt2aMckVKS85LhJrLhXaXIIikGeO9ZDW9c3RP1uXv3cmmXXO+\nzA6RSm7tJCmmtWsuBx6b1RGRBhApK8WIdHOna1tEko1RrkhpySjSKtzehF21iKQaA5HiyCjS\nLmyuj46hRSRau8wU09r122979uFfkcKYZyFkSgWRbr+4WacrR6T+sL49Om54RtKMEZmseriy\nIW0MRKoEREobo/zWzg0FtXbvp0AkRIqlTJFYtRONMWPHeq8DmgCR0sYoUSR4AK1d2hjltnbu\nKLO1i0yBSIgUCyKNKEokyRgPd+RV0Z8gUtoYZYgEf4JIaWMU1dr5htZuBCLlGnB53VwxIoUw\n67rUyRRFiSQWozx1cpL1YxSIJBYDdczI2dodmulvRZqRwl1tL9/a/W+CviaKae2GW+dP3vF7\nRgpE4okmloJEOnV3h793mkpRlEi8vikIVu3SxkiWDHW0QKS0Md5r7Xh9Y0hJrd3bKUoUCVny\ngEhnpvFzdqEAAAWZSURBVP44+yb96YT0uBEJQBlEWo46ZikDrV2p1DFLGRAJQB5EAjAAkZaj\njlnKQGtXKnXMUgZEApAHkQAMQKTlqGOWMtDalUods5QBkQDkQSQAAxBpOeqYpQy0dqVSxyxl\nQCQAeRAJwABEWo46ZikDrV2p1DFLGRAJQB5EAjAAkZajjlnKUGVrB+CMiCq3F0ck2Tz0hsSI\nZqA3JERSgxHNQG9IiKQGI5qB3pAQSQ1GNAO9ISGSGoxoBnpDQiQ1GNEM9IaESGowohnoDQmR\n1GBEM9AbEiKpwYhmoDckRFKDEc1Ab0iIpAYjmoHekBBJDUY0A70h6Y0IwCGIBGAAIgEYgEgA\nBiASgAGIBGAAIgEYgEgABiASgAGIBGAAIgEYgEgABiASgAGIBGAAIgEYgEgABuQU6Y1blKfj\nsAlhc1x6FCMEz9GJL60BdcO/2mHpUYxZQqQmY86/2F9G1C09jm8OkiJ1jdaAmvM5UjIp//nZ\nh6/sOZ/TNIe+W4ft0uP45hDWSw/hAWsts7dhM/xQOlPZz0/XKE3/86xQJ/QkuQsfSw/hXz7F\nniKbMLQQUmPKPpZ10Gmj+n4j1R4M7MJu6SH8wzG0UkV7RejPX36RDkJd1IlV6D+asBFyex32\nm9BInaS+DUdBkbZSf3Jynx+tJ6RTd7AWW/5YX9Ya2qXHMeIjfGq1UQOnblPqr03m83MYXiUK\ncXLoMKyl6rwuCaeq7Tulv7bn5Q85kXbrRuhfLbtI27DPm/APLkuox7BaeiC/6IRGtBreHJAT\nqR9e4Or8tcki0uiNEZW3I25Dug5LoEx+vXukM6LN+W+fwHj63ydJabE1s0gyb5LchrRGpKdc\nR/R9PcryIxI8Sd/kHYrc2u7H+c/tUei1/eUtkqPKX5xeSqQbt5Ok0/9mFmmt9rbN6d+iGxYb\nPpceyDfbYTGqU3stKfXH/3plQ7dW+rOc9/ystBa/++EpSWyxubtcRia1tNuriXS91k7ony2z\nSGL/HAP7Vuztz27bhJXQn9oLav9ycidJ7PwA+ASRAAxAJAADEAnAAEQCMACRAAxAJAADEAnA\nAEQCMACRAAxAJAADEAnAAEQCMACRAAxAJAADEAnAAEQCMACRAAxAJAADEAnAAEQCMACRAAxA\nJAADEAnAAEQCMACRAAxAJP90v7657kPtBus1gEjuOf7zDZDr4xLjqBtE0ubvm9cfH3wnwwqT\ncoNI2vwtUvt1/k+3an6+Ummv9IUndYBI2vwp0uf1W+s2n/3q57VRI/Y9ZeWDSNr8KdLq+uVO\np/12P987uBX6Usg6QCRtxiLtVt9frrVtwva87Wv0HZmHn++d/QxfuUYIZxBJm5FI7c/XPZ4f\nboZtH6Nv5d033w8P4Z+lPEgKImnzI9JnaA79oRm+N3p/fRiGr7f+2Xf187jT+Vb0SkAkbX5E\nWp+buP3wlHR7GO6esU6/Hx4cB1ngfGvzI8T10cieXyKtwubzwXGQBc63NvNF2of15/bBcZAF\nzrc280Vqw2G0bIdImeF8a/Pva6T13Wukdbi+C3sYNvz8a7LYkBlE0uavVbvv5e/18KDtrv+g\nLH/nBpG0CVf6f99HCpc3ZC8LDOcnpH73+XW5pGHPG7KZQSRtRiL1u2Z8ZUP7df6/10uE1pdn\npra5CMQlQrlBJL+cn51GlzOMWHHRamYQySFh6Oe6dTg/GbUPnPniYxS5QSSHfFzavctz0fFB\nF9fywb7cIJJHdm0It89P9Md/Vro/8Cg7iFQA3PxkeRAJwABEAjAAkQAMQCQAAxAJwABEAjAA\nkQAMQCQAAxAJwABEAjAAkQAMQCQAAxAJwABEAjAAkQAMQCQAAxAJwABEAjAAkQAMQCQAAxAJ\nwABEAjAAkQAMQCQAAxAJwABEAjAAkQAMQCQAAxAJwABEAjAAkQAM+D/cD+qlKdniUwAAAABJ\nRU5ErkJggg==",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Logistics with lasso penalty\n",
    "mod.can.lg.lasso.cv <- cv.glmnet(as.matrix(hotel[,-1]), hotel$is_canceled, family = \"binomial\")\n",
    "plot(mod.can.lg.lasso.cv)\n",
    "\n",
    "## Report number of features included in the optimal model\n",
    "sum(coef(mod.can.lg.lasso.cv, s = \"lambda.min\") != 0) \n",
    "\n",
    "## Find names of the selected features\n",
    "coefs <- coef(mod.can.lg.lasso.cv, s = \"lambda.min\") \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAMFBMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD////QFLu4AAAACXBIWXMAABJ0AAAS\ndAHeZh94AAAeJElEQVR4nO3diWKiSgIFUFxiEuPy/3/bioqARo1cUexzZl7aBaqKgiu7Fmug\ns+LZDYB3IEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQ\nIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQ\nIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQ\nIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQ\nIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQ\nIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQ0EOQChiYO5byfHCeUAUkCRIECBIECBIECBIECBIE\nCBIECBIECBIE9Bqkn89peRJ4Ovt5VBXwFD0GaTWuXVAxeUgV8CQ9BmlWjL4X5aPlfFTMHlEF\nPEmPQRoVi+rxohg9ogp4kh6D1LhA9vLVsoLEwFgjQUC/+0jzZfnIPhLvps/D35PaUbvx6iFV\nwHP0ex5pVp5HGk0/nUfivbiy4Q633Fh8HGb/qDHS9snu/uTdy4fnh5f6ta7+nHvafnJ3LY3J\nPLx07JDD092jw9NqxObwnWZb55LOFd7LKC9Yxf32c/bGYarloDZStaAcF6/68vWEr6xopfcB\nYa7lo/nK+tA/xwGqfql6rxoiMNtumX93FN/LKC9Yxf2K2t/rwxw+V+sjVenZv74brlxUmuuq\nw1IXXa7PFXY9SGfHuljedrKKw4fGcQHeTeTu9WOHFIe/h9X38eX1+lhclyBVRd0w/+4t/tGj\nnClksOeRita/l4eprYOqN6qPxvVxeSqqZP2+3PawqrpQ/29rqfX5rcB17f/Vv/vJ3a92qv45\nBqaoElQc+7Hqmy5JOm5o3zD/OpT/2FHOFHJSSmuuvSxBao8iSPcV959v2glSexRBuq+4/zxI\n9pFar9lHurO8/z5IxwTcMEy1nqmNVFQfytXCVV8aH7/iOXE9SJ1rqMpdN19ZH/rnOMDxU+YY\nvv2fwGy7Zf7dUXwvo7xgFV3cMkOPwxSHD9mi+W5RNLP0uIX4muOCfO5p+8ndtTQm8/DSsUMa\nqaq6qxqxOXyn2da5pHOF9zLKC1YBSYIEAT0G6XRNH68CnqTHIH0JEm+rz027xejyV54EqoDn\n6HUfaXH5dr5EFfAU/R5s+Krdbf6gKuAZHLWDAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGC\nAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGC\nAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGC\nAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGC\nAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGCAEGC\nAEGCAEGCAEGCAEGCAEGCAEGCAEGCgD6DtPwoRp/r9de4GM0eVAU8R49BWo2Kja/P7d9i8pAq\n4El6DNKs2KyHZqPiY7VelY/zVcCT9BikUTliUazKf0aPqAKepMcgFcXx7+GfcBXwJE9YI23/\nrqyReCtP2EearfaP81XAkzhqBwHOI0GAKxsgQJAgQJAg4FlBch6Jt/I6QSrqElVAf2zaQYAg\nQYAgQUCvQfr5nJZ7QNPZz6OqgKfo8xKhce1ogkuEeCu9XrQ6+l6Uj5bzkYtWeSu93kaxqB4v\n3EbBW+n9xr5zT2JVwJNYI0FAv/tI82X5yD4S76bPw9+T2lG78eohVcBz9HseaVaeRxpNP51H\n4r24sgECBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkC\nBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkC\nBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkC\nBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkC\nBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCBAkCOgSpaHpyq+CZBAkCum7aTUfzzd+f0Ueo\nPWeqgNfXMUizYlH+uyhmmfacVgED0DFIRdF+ECFIDEzHII2qNdIo057TKmAAOm/ajX42/8xH\nxWeqRe0qYAC6HmyY7I/ZTVMNOq0CXl/nE7Lf022M5reMuJptt/8+x0Ux+Y63Cp6pxysblqOi\nWK9GuzXY5CFVwJP0GKSPYrra/PlYbjL1cflwuSAxMJ2DNJ9uj3xPlzeMV6z2fzZbeZeP8gkS\nAxM52LB5bXQ9SeWpplFRe5JsFTxTxyB9FZPVNhNfxfVrhD6255w+dyeeVpd3kgSJgel8Qna1\nW7nccGXDohjNFuvpaJOk+bi4eJxPkBiYwCVCtwZpe9q2cvn8rSAxMB2DNN6vkRbF+JZRvz/G\n5dnbzyt7VILEwGT2kTbrmq9Yk9aCxOB0vh+puOUEa6cq4PVFziMV0yuX/HSrAl7es778xHkk\n3srrBOlhXwABj+fLTyBAkCDAF0RCQK9B+vncHS2fzn4eVQU8RY+3UazGtQ1BN/bxVnq8jWJW\njL533zm0nI/c2Mdb6fE2isNXd21d+fouQWJgeryNojGIE7K8lR5vo7BG4n31eBvFZh9pvtuT\nso/Eu+nzNopJ7ajdeBVuFTxTr7dR/MzKwUfTT+eReC9uo4AAlwhBgCBBQNcg7b4YfzS7eOyg\nWxXw+joGqfxi/O1ZpBsuEbqzChiAjkGaFB/bddFqlv2BJEFiYAJXNjQfRAgSAxO41m5rJUj8\n1zoGaVZMtudWfyaXL/npUgUMQOR+JF8Qyf+u83mk8jdkJ9EvLBYkBscJWQgQJAgQJAjoGqTP\n6quBUi06qQJeX8cgffqmVVgHTsiGj9edVgEDkLpEKEuQGJiOQZoW2fsnzlQBA9D5NorJla9f\nuIsgMTCdN+0cbABBgggnZCFAkCAgFaQft5rzP+sapJl9JAjcIXswjzVpLUgMTudLhL7Xk2K5\nnBTR00mCxMAELhH63KyNFtl7zQWJgQkEab69cNU+Ev+1ztfafa+XxXj9I0j81zoGab4NUPlN\nQtd/jPnOKmAAOt8hu332UWS/1k6QGBpXNkCAIEGA30eCAL+PBAF+HwkC/D4SBPh9JAjw+0gQ\n4PeRIMDvI0GAE7IQkAxS7oCDIDEwggQBggQBggQBggQBggQBggQBggQBggQBggQBggQBrrWD\ngK5B+hqv18txMc7+JLMgMTCJb1odbW9I8msU/M86f/nJ93pRjNfffo2C/1rgy08W29vMfWcD\n/7VAkKbbX+sTJP5rnTftFvNitLZpx3+u+8GGovjcrpD8hiz/s86Hv0flF3GNv0PtOVMFvD4n\nZCFAkCCgQ5C2B+qKoye3Cp5JkCDAph0ECBIEdApS+bOX669xMcr+qLkgMTQdgrTa/ezltNxD\nGkV/RFaQGJgOQZoVk016forxar3y+0j83zoEafdrfR/lxUGrYhRslCAxNPcHqTjx1FbBM3Vd\nI81323TWSPzfOgTpY5Oh1bhYbB6upvaR+K91CNKy3J77KF8qRstgowSJoelyHmkxOZxAGn1E\nj34LEkPjygYIECQI6HT194MOfgsSgyNIEGDTDgIECQI6B2k+Lb8kMnoaSZAYmq5Bmux2j5yQ\n5f/WMUhfxWS1DdLX7gqHFEFiYDoGaXvhannAzlE7/muBL9H/e5CuDixIDEzHII33a6TtbyT9\noQRB4s1k9pHmo+Lr+ni3n8AVJAam61G76T4WN/yqy89IkHhXkfNIxfSmH6NYTYtJeZTcph3v\npt8rG76LYhs5QeLd9HyJ0HJSTFeCxNsJHP4ujW798pPPYjQXJN5NKEjL288jLcbX77kQJAam\nQ5DmjYNwfziP9CFIvJsua6RxPUc/T24VPFNqHylLkBiYZ93Y54Qsb6VzkL4nN5+QbRRyUvHD\nvgACHi9yY99tlwjdWwW8vs4XrY62v+py00Wrd1YBA9D5NopF+e/fbqP4UxUwAKmjdjft1vx8\n7i4Wn86uHCwXJAYmtka6fonQqn7e6fI+lSAxMD3uI82K0fcudsvN8Bd/T0mQGJgej9qN9muv\nrStrMEFiYLqfR7r5xr7i7M5VqlXwTD1e2WCNxPvqMUibfaT57vtY7SPxbvq81m5SO2o3vvhb\nmYLEwHQN0ld1TPuGMX9m5Xmk0fTTeSTeS8cgffqhMVgHvvs7eo3duSpgANzYBwEdgzQrLh40\nuJcgMTCdv7J4Ev2yhnNVwOvrECS/ag4HggQBz/ryk6dXAUmCBAGCBAGCBAGCBAGCBAGCBAGC\nBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGC\nBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGC\nBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGC\nBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGCBAGC\nBAGCBAF9Bmn1URST+b6Qi6UIEgPTY5BWo2JruitEkHgnPQZpVnxt0vQ1mpSFCBLvpMcgjXYj\nLkfjpSDxZnoM0iE7q8lEkHgzPQZpXKwOjyaCxHvpMUhfxcf+0bKYCBJvpc/D37MqPfNCkHgr\nvZ6QXUwPj5YfgsQ7cWUDBAgSBAgSBDwrSA428FZeJ0hFXaIK6I9NOwgQJAgQJAjoNUg/n9Pd\nLUmzn0dVAU/R541949rRhMlDqoAn6fXGvtH3ony0nI+K2SOqgCfp9ca+RfV4UYweUQU8yRNu\n7Dt9EqsCnsQaCQL63UeaL8tH9pF4N30e/p7UjtqNV5eGFCQGpt/zSLPyPNJo+uk8Eu/FlQ0Q\nIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQ\nIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQ\nIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQ\nIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQ\nIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQIEgQ0GuQfj6n\nxdZ09vOoKuApegzSalwcTR5SBTxJj0GaFaPvRfloOR8Vs0dUAU/SY5BGxaJ6vChGj6gCnqTH\nIBXFb09iVcCTWCNBQL/7SPNl+cg+Eu+mz8Pfk9pRu/HqIVXAc/R7HmlWnkcaTT+dR+K9DOvK\nht3a7PKo2/fPDXV8af/u8e9xRVmrpV7E/nHRGnD3Qv3l1gsnb/3y2mu73OJqplwd/jhvitp8\n2j8/0+vr5sBnZubJ8Kcz+4+K1iGx4sK7zbfuqOvvo2Sq2M+iS0f8jgk5mSPtdxuPa3P6XLDW\n69OFY31oDDer9dm63fGNwQ4zrT7w6VyuvX1uKbhtWft9xHYxF4sdVJBq/780YlH72yyxaL9b\nvrA+Rms7Z46PawPVY7PLWzliY/RDEIvGC/eE7bHx/L30xrLdmIRfxmm9Ww3V7K51o2vXtb9F\n1X+N/1Uz7ThwfaZXn29FNdCZxefuRbs4++xKsc8K0h3nkQ4JupCkQ0ZOknTsm+Pm27qafcd1\n0TFTtdXWuhqj+V+Vo9NV2bo4Lkm1N4a0/mp8CDT/KVpTsq4+NxoT2shi7eNmfXx3fZwBp0kq\nWvOzNTOL6nOuvd13aSm65HyCfvv3/Lh3VNfJufVxc9acq1eQ+iNIvxT3WkG6owpB6pUg/VLc\n4IO0T9CFHB2TVP1tlli03z1E4ZCaeoKKoqj3aHPhWheHOVhfRI6D1BfDv8fnsYH7vfR1/WFj\nEn4Zp/VuNVSzu9aNrl3X/hZV/53mqOrgopmj4ww59P/5xefuRft8nq4UO6gg1T7bfhux6vmi\nFaSi/W7j8XE5WDcXp3qhrWWotqxwm1qfrdsd3xjsMNPqA5/O5drb55aC25a130dsF3Ox2F6D\n1PnGvmY//zbI+pfePXZQ82PxNDitGVp9KLbn9bml4NcNoAuvvbbLLa5mytXha59b7c+ws72+\nbg58ZmaeDH86s/+oOWK7mAvF9hgkN/bxvnoMkhv7eF89BsltFLyvHoPU2vp8RBXwJNZIENDv\nPpIb+3hTfR7+dmMfb6vf80hu7ONNDenKBnhZggQBggQBggQBggQBggQBggQBggQBggQBggQB\nLxokGJg7lvJ8cF7GIKZNI2Oe2sphdNF9BjFtGhkjSA8yiGnTyBhBepBBTJtGxgjSgwxi2jQy\nRpAeZBDTppExgvQgg5g2jYwRpAcZxLRpZIwgPcggpk0jYwTpQQYxbRoZI0gPMohp08gYQXqQ\nQUybRsYIEgydIEGAIEGAIEGAIEGAIEGAIEGAIEGAIEGAIEGAIEGAIEGAIEGAIEGAIEGAIEHA\nWwfpa1yMZqtnt+Kqr9eeCbPRIHrxyd342vOwm1n5wwKjV18GFvf8+EF/JmUvjp/djKue3I0v\nPQ+7WRQfq+3n1MezG3LZYvTSQfopRottG3+e3ZArnt2NrzwPO5rupu2lF9Nt0Ccv3cJZMd/8\n/S4+n92Qy57eja88DzNeejHdNG/22i2cFsv1du0+fXZDLnt6N77yPIxYFZNnN+GixYtHvRjE\nev353fji/dPdV7lp8tJeeiEdSJDWz27i6/dPN8vRi2+TrJ+9BFwhSDfW/szKH281eu0Nu9JL\nL6SCdGPtz6z8Meq/Sz151fMf9Ua+9EI6EqTban9m5Y9xXEaX48ny2a35xWCCtDtqt3z1o3br\nZ3fjK8/DruYvfsDu4KWD9FkerJkXs2c35CpBepDlQHL02kEaypUNz+7GV56HHX0URX0D6nW9\ndgvHZR8O4DNJkB6kEKSEVXn197NbcQNBgsETJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQ\nJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQ\nJAgQJAgQJAgQJAgQJAh4tSBtf3at9dNr88O/N/2w9h9/t+384KuPopj9WtIr/cLevPa43a5t\nh51v67z1fNfr54b9+tO0HktoltV4dtN8LOdAvYALjWu8+evMKX94cLV5MJ3Xxto/mbZ75M9e\naJkonQZpvH+2LFa3FvDH+k5NN538OYQgjetNabWr7LCzbR23X/w1SIu/TestQbptPpZzoF7A\nhcbdFKRJ+Suo482jVbE8jrV/sv+ng9dZJnZO++HwyuS23zGNBKm42K8vFKRLTSk77OwAJy+e\n2Q4oLUb3Bun312+bj605cKVx14NU/3H22aQ26P7JrOuvTb/OMrHza5C+b1shpYIUrOKRLjRl\n12GdgvRVTM4O+ufm1F6/cT62SrrSuOtBmhXzsvLtWm5VfB8H3T85vHa3PpeJTcNn+x/ILorV\nuNhuK3+Ni9HX7u3NRuysNks3TyfL/W+Tb56OJ/siPovR57Zj9pvQmwLG7QIO9kVsN8uLquLl\ntCzg2KaTMfe/hb5rSjV4vYz2dB3L3E/PT/FRjlPOvvXHZp5vPwr3r553LH893ywik+NW+8lE\nV4PuO2ffm5uHtXqqDpvV2lZ2VfVT71Xn/xKkVnfWOmynPZU/x9oa87DeivH+o7+aN/U5eGjQ\n4dfoq4640rjrQZqWa7hFucytJ+PaoPsn+3/u1m+QPrf9MykfTsuFYlocXtltxE6rWVo+Ha0O\nc/2n+KoVMS/fnR0Gaxewdyhi/bmbMbtFb1TUN7+PdX2cD9Jh8EYZ7emqyqymZ1QO87Fb8IvR\nshht/h2NVo0q6hv3tfK/dg+/Wv1WTfRx0CpIZW9uHh7rqTpseiir6qpDvcfO/yVIi19WXpXW\nVNZqa8zD2uv7Zh3fqs/BaaOFRa0jrjSuHaTT7q12p9ZlB/+cPNn/c7d+g7TbTv3ePpxsO3G+\n/Wc12X6ife/f3M/S7+07H7uFZTvurFjsiti8/LX/OzqO9d0oYKdeRPl+cSxgfGzTthWtMQ9z\noz54o4z2dB0GOk7PZ7mpUJTL9XaD4mvz3+dx8+HcnK7KH22n9bvexuZEnzRl35vbJ1U9zQ4b\nN7pqN1at83/bR7oWpPZU1rurOQMOr++bdXyr1qxag9atjrjSuD8GaVHMTp4sitv2wX/Tb5C2\nXTQvyqOyZf6n5QbzavvKtHxlfpil5dPVbrHZDjnZbVrvxtvtie4G2xU5aRSwcyziUH2tgPqL\nh0LOBul08DNB+qlKOkzPctukn80H8WLb9uX2z1dx/bjvvvx568XmRJ80pah9wh7qaXVYo6t2\nY9U6/8YgnSyf7ak81taeh4fX9806vtWYg1WD2h3xxyD9Oujun1UxOXmy/+duPe8jHf6pHlYz\npvVm+7Ok0RHHv+eKbFVXWs4/J40lrz7QyZj1IB3faZXRnq7d4NX0bJeYzcfv5mO6XNo2fy8f\nCqyVv9kVmi4W58r/pSn1CTjUc9JhJ13V6vx7gtSeymZt5+bhb3Op1XvrVkdkg3RumfptxFv9\nH0GaNGfQXUFql9GertaiMN8sXKPxejw+bNHN2icYm4tkrfz153afa7Q8Lf+XpjQmYFbfbskG\n6fSF1lSGg3TsiD8G6bR7R+8epN/ejAbpoxh/zZfdgnRSRnu62gtjMf7ZLNKz7QG17RZLc410\nOqdr5W/MZ+PmftzFptQn4C9rpMYU3BWk1lR2DdJpNfuO6Byk3VG75X7revBB2u3FfFStnh73\nBnYPfw4dOvltH6n5d3rc7aoVsNMq4vcgXdxHWtcfXQtSbXo2C9fH5tlmanfHiKebfZdLW+G1\n8uulnpvo3yan/OdQz0mHTet7qK3Ovz9IzalstrE9D3evlc2anOwjTZu997eUXw/B576acm09\n/H2k3QGaeTXl5SGbdbl3PG8etfvaHsGZ7Y74bD9LZvUjlse/jWM+rWNv9SJ+1ovf95EuHrVb\nV49aZaybQx+bs5uebaR3H9L7U4Gzzcz8Wv+qVv54d1Tu1zVSoymttU9Vz0mHNY7aLZudf3+Q\nmlPZDFJ9Bhxenx0ONh/eah933fdeqyM6B6l+ZcP6p771u3/yM6ijdrtTPevjlO+29svdgfIk\nwkc1S6sTDePy6OrP7tRPe5k6OQvxUVvUqyJm+/X8T2PU459Je9v8TJDaZZysO2ol7XZvdhtn\nk/Kw4WpUnt+5cLihVv539WjdblO7KbvOOQbpWM9phx27ajdWrbG1ILWWyauLaH0q22vN2nmk\nw+v7Zp0/j1TrvXKMY0dcadzZFXm7nVU1m0+a2ofM/snnoM4jbRb2cXUqvfS1mcCP3eL12bqy\nYRO57Rs/43ImjWtr4/rfr1F1Xrwq4FD8oYjNXkUx+alt1bSCVJ5m/7kcpHYZ54PUmJ7yM273\n92N/xcGFzYdj+bsT+sd5fTLRx0F3nXNczGr1nHZY1VX7Lj02tkOQ6lPZDlI1A2qtqK5sOMyb\n+hw89t5ujKojugdpVV79vXs8/CsbOow873x97jV/3Ejus+fu8dcOO8ydB0/XffPxSuP+tmQt\ni8ONFMcny6LjjRSDCdKtV3/fobxSYDX920by94XL5l7DHztsP3cePl13zccrjfvbkjX4q7+7\nBenG+5HusL92bXR9yJqb7jJ8qj922H4n8eHTddd8vNi4+mHuGwz/fqRuQVrPH/ZZ+bXZFB8/\nbIX3NH/rsD8ujve7Zz5ebNwfW/5Rv0N2/+Tj7e6QhUESJAgQJAgQJAgQJAgQJAgQJAgQJAgQ\nJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQJAgQ\nJAj4B8mKcjYsgyc/AAAAAElFTkSuQmCC",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mod.can.lg.final<-glmnet(as.matrix(hotel[,1+which(coefs[,1]!=0) ]), hotel$is_canceled, family = \"binomial\",lambda=0)\n",
    "plot(x=predict(mod.can.lg.final,newx=as.matrix(hotel[,1+which(coefs[,1]!=0) ])),y=hotel$is_canceled)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.1.0"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}