Rossman.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rossman Competition\n",
    "This exercise aims to develop a neural network to predic sales of Rossman stores for 6 weeks. I will use data provided by Kaggle's Rossman challenge (https://www.kaggle.com/c/rossmann-store-sales). Important things learnt from this exercise include application of neural networks to structure data, and utilizing embedding matrices for categorical variables. I will follow codes given in fast.ai, and fast.ai study group material provided at https://github.com/dlnerds/dlnerds "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install fastai\n",
    "from fastai.structured import *\n",
    "from fastai.column_data import *\n",
    "np.set_printoptions(threshold=50, edgeitems=20)\n",
    "\n",
    "PATH='data/rossmann/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning and Feature Engineering\n",
    "Fast.ai's Jeremy goes over lot of data cleaning and feature engineering steps in the fast.ai notebook. Even though data preprocessing is crucial in any machine learning model, since my main focus is on deep learning, I will just use his codes and quickly jump to the deep learning part."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature Space:\n",
    "\n",
    "* train: Contains store information on a daily basis, tracks things like sales, customers, whether that day was a holdiay, etc.\n",
    "* store: general info about the store including competition, etc.\n",
    "* store_states: maps store to state it is in\n",
    "* state_names: Maps state abbreviations to names\n",
    "* googletrend: trend data for particular week/state\n",
    "* weather: weather conditions for each state\n",
    "* test: Same as training table, w/o sales and customers\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "table_names = ['train', 'store', 'store_states', 'state_names', \n",
    "               'googletrend', 'weather', 'test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use pandas to load all the given tables in to a list of dataframes. Then take a look at those by displaying the headers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "tables = [pd.read_csv(f'{PATH}{fname}.csv', low_memory=False) for fname in table_names]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sales</th>\n",
       "      <th>Customers</th>\n",
       "      <th>Open</th>\n",
       "      <th>Promo</th>\n",
       "      <th>StateHoliday</th>\n",
       "      <th>SchoolHoliday</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>5263</td>\n",
       "      <td>555</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>6064</td>\n",
       "      <td>625</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>8314</td>\n",
       "      <td>821</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>13995</td>\n",
       "      <td>1498</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>4822</td>\n",
       "      <td>559</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Store  DayOfWeek        Date  Sales  Customers  Open  Promo StateHoliday  \\\n",
       "0      1          5  2015-07-31   5263        555     1      1            0   \n",
       "1      2          5  2015-07-31   6064        625     1      1            0   \n",
       "2      3          5  2015-07-31   8314        821     1      1            0   \n",
       "3      4          5  2015-07-31  13995       1498     1      1            0   \n",
       "4      5          5  2015-07-31   4822        559     1      1            0   \n",
       "\n",
       "   SchoolHoliday  \n",
       "0              1  \n",
       "1              1  \n",
       "2              1  \n",
       "3              1  \n",
       "4              1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>StoreType</th>\n",
       "      <th>Assortment</th>\n",
       "      <th>CompetitionDistance</th>\n",
       "      <th>CompetitionOpenSinceMonth</th>\n",
       "      <th>CompetitionOpenSinceYear</th>\n",
       "      <th>Promo2</th>\n",
       "      <th>Promo2SinceWeek</th>\n",
       "      <th>Promo2SinceYear</th>\n",
       "      <th>PromoInterval</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>a</td>\n",
       "      <td>1270.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>570.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>1</td>\n",
       "      <td>13.0</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>Jan,Apr,Jul,Oct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>14130.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2006.0</td>\n",
       "      <td>1</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2011.0</td>\n",
       "      <td>Jan,Apr,Jul,Oct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>c</td>\n",
       "      <td>c</td>\n",
       "      <td>620.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2009.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>29910.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Store StoreType Assortment  CompetitionDistance  CompetitionOpenSinceMonth  \\\n",
       "0      1         c          a               1270.0                        9.0   \n",
       "1      2         a          a                570.0                       11.0   \n",
       "2      3         a          a              14130.0                       12.0   \n",
       "3      4         c          c                620.0                        9.0   \n",
       "4      5         a          a              29910.0                        4.0   \n",
       "\n",
       "   CompetitionOpenSinceYear  Promo2  Promo2SinceWeek  Promo2SinceYear  \\\n",
       "0                    2008.0       0              NaN              NaN   \n",
       "1                    2007.0       1             13.0           2010.0   \n",
       "2                    2006.0       1             14.0           2011.0   \n",
       "3                    2009.0       0              NaN              NaN   \n",
       "4                    2015.0       0              NaN              NaN   \n",
       "\n",
       "     PromoInterval  \n",
       "0              NaN  \n",
       "1  Jan,Apr,Jul,Oct  \n",
       "2  Jan,Apr,Jul,Oct  \n",
       "3              NaN  \n",
       "4              NaN  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>State</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>HE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>TH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>NW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>BE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>SN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Store State\n",
       "0      1    HE\n",
       "1      2    TH\n",
       "2      3    NW\n",
       "3      4    BE\n",
       "4      5    SN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StateName</th>\n",
       "      <th>State</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BadenWuerttemberg</td>\n",
       "      <td>BW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Bayern</td>\n",
       "      <td>BY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Berlin</td>\n",
       "      <td>BE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Brandenburg</td>\n",
       "      <td>BB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bremen</td>\n",
       "      <td>HB</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           StateName State\n",
       "0  BadenWuerttemberg    BW\n",
       "1             Bayern    BY\n",
       "2             Berlin    BE\n",
       "3        Brandenburg    BB\n",
       "4             Bremen    HB"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file</th>\n",
       "      <th>week</th>\n",
       "      <th>trend</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Rossmann_DE_SN</td>\n",
       "      <td>2012-12-02 - 2012-12-08</td>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Rossmann_DE_SN</td>\n",
       "      <td>2012-12-09 - 2012-12-15</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Rossmann_DE_SN</td>\n",
       "      <td>2012-12-16 - 2012-12-22</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Rossmann_DE_SN</td>\n",
       "      <td>2012-12-23 - 2012-12-29</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Rossmann_DE_SN</td>\n",
       "      <td>2012-12-30 - 2013-01-05</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             file                     week  trend\n",
       "0  Rossmann_DE_SN  2012-12-02 - 2012-12-08     96\n",
       "1  Rossmann_DE_SN  2012-12-09 - 2012-12-15     95\n",
       "2  Rossmann_DE_SN  2012-12-16 - 2012-12-22     91\n",
       "3  Rossmann_DE_SN  2012-12-23 - 2012-12-29     48\n",
       "4  Rossmann_DE_SN  2012-12-30 - 2013-01-05     67"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file</th>\n",
       "      <th>Date</th>\n",
       "      <th>Max_TemperatureC</th>\n",
       "      <th>Mean_TemperatureC</th>\n",
       "      <th>Min_TemperatureC</th>\n",
       "      <th>Dew_PointC</th>\n",
       "      <th>MeanDew_PointC</th>\n",
       "      <th>Min_DewpointC</th>\n",
       "      <th>Max_Humidity</th>\n",
       "      <th>Mean_Humidity</th>\n",
       "      <th>...</th>\n",
       "      <th>Max_VisibilityKm</th>\n",
       "      <th>Mean_VisibilityKm</th>\n",
       "      <th>Min_VisibilitykM</th>\n",
       "      <th>Max_Wind_SpeedKm_h</th>\n",
       "      <th>Mean_Wind_SpeedKm_h</th>\n",
       "      <th>Max_Gust_SpeedKm_h</th>\n",
       "      <th>Precipitationmm</th>\n",
       "      <th>CloudCover</th>\n",
       "      <th>Events</th>\n",
       "      <th>WindDirDegrees</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NordrheinWestfalen</td>\n",
       "      <td>2013-01-01</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>94</td>\n",
       "      <td>87</td>\n",
       "      <td>...</td>\n",
       "      <td>31.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>39</td>\n",
       "      <td>26</td>\n",
       "      <td>58.0</td>\n",
       "      <td>5.08</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Rain</td>\n",
       "      <td>215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NordrheinWestfalen</td>\n",
       "      <td>2013-01-02</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>93</td>\n",
       "      <td>85</td>\n",
       "      <td>...</td>\n",
       "      <td>31.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>24</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Rain</td>\n",
       "      <td>225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NordrheinWestfalen</td>\n",
       "      <td>2013-01-03</td>\n",
       "      <td>11</td>\n",
       "      <td>8</td>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>100</td>\n",
       "      <td>93</td>\n",
       "      <td>...</td>\n",
       "      <td>31.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>26</td>\n",
       "      <td>21</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.02</td>\n",
       "      <td>7.0</td>\n",
       "      <td>Rain</td>\n",
       "      <td>240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NordrheinWestfalen</td>\n",
       "      <td>2013-01-04</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>100</td>\n",
       "      <td>94</td>\n",
       "      <td>...</td>\n",
       "      <td>11.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23</td>\n",
       "      <td>14</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.25</td>\n",
       "      <td>7.0</td>\n",
       "      <td>Rain</td>\n",
       "      <td>263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NordrheinWestfalen</td>\n",
       "      <td>2013-01-05</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "      <td>6</td>\n",
       "      <td>100</td>\n",
       "      <td>94</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>16</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.0</td>\n",
       "      <td>Rain</td>\n",
       "      <td>268</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 file        Date  Max_TemperatureC  Mean_TemperatureC  \\\n",
       "0  NordrheinWestfalen  2013-01-01                 8                  4   \n",
       "1  NordrheinWestfalen  2013-01-02                 7                  4   \n",
       "2  NordrheinWestfalen  2013-01-03                11                  8   \n",
       "3  NordrheinWestfalen  2013-01-04                 9                  9   \n",
       "4  NordrheinWestfalen  2013-01-05                 8                  8   \n",
       "\n",
       "   Min_TemperatureC  Dew_PointC  MeanDew_PointC  Min_DewpointC  Max_Humidity  \\\n",
       "0                 2           7               5              1            94   \n",
       "1                 1           5               3              2            93   \n",
       "2                 6          10               8              4           100   \n",
       "3                 8           9               9              8           100   \n",
       "4                 7           8               7              6           100   \n",
       "\n",
       "   Mean_Humidity       ...        Max_VisibilityKm  Mean_VisibilityKm  \\\n",
       "0             87       ...                    31.0               12.0   \n",
       "1             85       ...                    31.0               14.0   \n",
       "2             93       ...                    31.0                8.0   \n",
       "3             94       ...                    11.0                5.0   \n",
       "4             94       ...                    10.0                6.0   \n",
       "\n",
       "   Min_VisibilitykM  Max_Wind_SpeedKm_h  Mean_Wind_SpeedKm_h  \\\n",
       "0               4.0                  39                   26   \n",
       "1              10.0                  24                   16   \n",
       "2               2.0                  26                   21   \n",
       "3               2.0                  23                   14   \n",
       "4               3.0                  16                   10   \n",
       "\n",
       "   Max_Gust_SpeedKm_h  Precipitationmm  CloudCover  Events  WindDirDegrees  \n",
       "0                58.0             5.08         6.0    Rain             215  \n",
       "1                 NaN             0.00         6.0    Rain             225  \n",
       "2                 NaN             1.02         7.0    Rain             240  \n",
       "3                 NaN             0.25         7.0    Rain             263  \n",
       "4                 NaN             0.00         7.0    Rain             268  \n",
       "\n",
       "[5 rows x 24 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Store</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>Promo</th>\n",
       "      <th>StateHoliday</th>\n",
       "      <th>SchoolHoliday</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Store  DayOfWeek        Date  Open  Promo StateHoliday  SchoolHoliday\n",
       "0   1      1          4  2015-09-17   1.0      1            0              0\n",
       "1   2      3          4  2015-09-17   1.0      1            0              0\n",
       "2   3      7          4  2015-09-17   1.0      1            0              0\n",
       "3   4      8          4  2015-09-17   1.0      1            0              0\n",
       "4   5      9          4  2015-09-17   1.0      1            0              0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for t in tables: display(t.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, store, store_states, state_names, googletrend, weather, test = tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1017209, 41088)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train),len(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jeremy's Data cleaning and feature engineering steps are performed below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert state Holidays to booleans\n",
    "train.StateHoliday = train.StateHoliday!='0'\n",
    "test.StateHoliday = test.StateHoliday!='0'\n",
    "\n",
    "# A function to perform pandas join\n",
    "def join_df(left, right, left_on, right_on=None, suffix='_y'):\n",
    "    if right_on is None: right_on = left_on\n",
    "    return left.merge(right, how='left', left_on=left_on, right_on=right_on, \n",
    "                      suffixes=(\"\", suffix))\n",
    "\n",
    "# Join weather/state names\n",
    "weather = join_df(weather, state_names, \"file\", \"StateName\")\n",
    "\n",
    "# Add googletrend by date and state\n",
    "googletrend['Date'] = googletrend.week.str.split(' - ', expand=True)[0]\n",
    "googletrend['State'] = googletrend.file.str.split('_', expand=True)[2]\n",
    "googletrend.loc[googletrend.State=='NI', \"State\"] = 'HB,NI'\n",
    "\n",
    "# Extracting date/time fields for constructing categoricals\n",
    "add_datepart(weather, \"Date\", drop=False)\n",
    "add_datepart(googletrend, \"Date\", drop=False)\n",
    "add_datepart(train, \"Date\", drop=False)\n",
    "add_datepart(test, \"Date\", drop=False)\n",
    "\n",
    "# Taking googletrend data - germany as a whole\n",
    "trend_de = googletrend[googletrend.file == 'Rossmann_DE']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can outer join all of our data into a single dataframe. Recall that in outer joins everytime a value in the joining field on the left table does not have a corresponding value on the right table, the corresponding row in the new table has Null values for all right table fields. One way to check that all records are consistent and complete is to check for Null values post-join, as we do here.\n",
    "\n",
    "Aside: Why note just do an inner join? If you are assuming that all records are complete and match on the field you desire, an inner join will do the same thing as an outer join. However, in the event you are wrong or a mistake is made, an outer join followed by a null-check will catch it. (Comparing before/after # of rows for inner join is equivalent, but requires keeping track of before/after row #'s. Outer join is easier.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "0 0\n",
      "0 0\n",
      "0 0\n",
      "0 0\n"
     ]
    }
   ],
   "source": [
    "store = join_df(store, store_states, \"Store\")\n",
    "print( len(store[store.State.isnull()]) )\n",
    "\n",
    "joined = join_df(train, store, \"Store\")\n",
    "joined_test = join_df(test, store, \"Store\")\n",
    "print( len(joined[joined.StoreType.isnull()]),len(joined_test[joined_test.StoreType.isnull()]) )\n",
    "\n",
    "joined = join_df(joined, googletrend, [\"State\",\"Year\", \"Week\"])\n",
    "joined_test = join_df(joined_test, googletrend, [\"State\",\"Year\", \"Week\"])\n",
    "print( len(joined[joined.trend.isnull()]),len(joined_test[joined_test.trend.isnull()]) )\n",
    "\n",
    "joined = joined.merge(trend_de, 'left', [\"Year\", \"Week\"], suffixes=('', '_DE'))\n",
    "joined_test = joined_test.merge(trend_de, 'left', [\"Year\", \"Week\"], suffixes=('', '_DE'))\n",
    "print( len(joined[joined.trend_DE.isnull()]),len(joined_test[joined_test.trend_DE.isnull()]) )\n",
    "\n",
    "joined = join_df(joined, weather, [\"State\",\"Date\"])\n",
    "joined_test = join_df(joined_test, weather, [\"State\",\"Date\"])\n",
    "print( len(joined[joined.Mean_TemperatureC.isnull()]),len(joined_test[joined_test.Mean_TemperatureC.isnull()]) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "for df in (joined, joined_test):\n",
    "    for c in df.columns:\n",
    "        if c.endswith('_y'):\n",
    "            if c in df.columns: df.drop(c, inplace=True, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[24  3 19  9  0 16 17  7 15 22 11 13  2 23 12  4 10  1 14 20  8 18  6 21  5]\n"
     ]
    }
   ],
   "source": [
    "# Pick arbitrary signal values for missing values in these cases\n",
    "for df in (joined,joined_test):\n",
    "    df['CompetitionOpenSinceYear'] = df.CompetitionOpenSinceYear.fillna(1900).astype(np.int32)\n",
    "    df['CompetitionOpenSinceMonth'] = df.CompetitionOpenSinceMonth.fillna(1).astype(np.int32)\n",
    "    df['Promo2SinceYear'] = df.Promo2SinceYear.fillna(1900).astype(np.int32)\n",
    "    df['Promo2SinceWeek'] = df.Promo2SinceWeek.fillna(1).astype(np.int32)\n",
    "    \n",
    "# Extract features \"CompetitionOpenSince\" and \"CompetitionDaysOpen\"\n",
    "for df in (joined,joined_test):\n",
    "    df[\"CompetitionOpenSince\"] = pd.to_datetime(dict(year=df.CompetitionOpenSinceYear, \n",
    "                                                     month=df.CompetitionOpenSinceMonth, day=15))\n",
    "    df[\"CompetitionDaysOpen\"] = df.Date.subtract(df.CompetitionOpenSince).dt.days\n",
    "\n",
    "# Replace some erroneous / outlying data\n",
    "for df in (joined,joined_test):\n",
    "    df.loc[df.CompetitionDaysOpen<0, \"CompetitionDaysOpen\"] = 0\n",
    "    df.loc[df.CompetitionOpenSinceYear<1990, \"CompetitionDaysOpen\"] = 0\n",
    "\n",
    "# Add \"CompetitionMonthsOpen\" field, limiting the maximum to 2 years to limit number of unique categories\n",
    "for df in (joined,joined_test):\n",
    "    df[\"CompetitionMonthsOpen\"] = df[\"CompetitionDaysOpen\"]//30\n",
    "    df.loc[df.CompetitionMonthsOpen>24, \"CompetitionMonthsOpen\"] = 24\n",
    "print( joined.CompetitionMonthsOpen.unique() )\n",
    "\n",
    "# Same process for Promo dates\n",
    "for df in (joined,joined_test):\n",
    "    df[\"Promo2Since\"] = pd.to_datetime(df.apply(lambda x: Week(\n",
    "        x.Promo2SinceYear, x.Promo2SinceWeek).monday(), axis=1).astype(pd.datetime))\n",
    "    df[\"Promo2Days\"] = df.Date.subtract(df[\"Promo2Since\"]).dt.days\n",
    "\n",
    "for df in (joined,joined_test):\n",
    "    df.loc[df.Promo2Days<0, \"Promo2Days\"] = 0\n",
    "    df.loc[df.Promo2SinceYear<1990, \"Promo2Days\"] = 0\n",
    "    df[\"Promo2Weeks\"] = df[\"Promo2Days\"]//7\n",
    "    df.loc[df.Promo2Weeks<0, \"Promo2Weeks\"] = 0\n",
    "    df.loc[df.Promo2Weeks>25, \"Promo2Weeks\"] = 25\n",
    "    df.Promo2Weeks.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "joined.to_feather(f'{PATH}joined')\n",
    "joined_test.to_feather(f'{PATH}joined_test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# define a function get_elapsed for cumulative counting across a sorted dataframe. \n",
    "# Given a particular field fld to monitor, this function will start tracking time since \n",
    "#the last occurrence of that field. When the field is seen again, the counter is set to zero\n",
    "\n",
    "def get_elapsed(df,fld, pre):\n",
    "    day1 = np.timedelta64(1, 'D')\n",
    "    last_date = np.datetime64()\n",
    "    last_store = 0\n",
    "    res = []\n",
    "\n",
    "    for s,v,d in zip(df.Store.values,df[fld].values, df.Date.values):\n",
    "        if s != last_store:\n",
    "            last_date = np.datetime64()\n",
    "            last_store = s\n",
    "        if v: last_date = d\n",
    "        res.append(((d-last_date).astype('timedelta64[D]') / day1))\n",
    "    df[pre+fld] = res\n",
    "    return df\n",
    "\n",
    "# define a function to easily apply the same set of operations for both test and train    \n",
    "def add_durations( df, joinedDF ):\n",
    "    fld = 'SchoolHoliday'\n",
    "    df = df.sort_values(['Store', 'Date'])\n",
    "    df = get_elapsed(df, fld, 'After')\n",
    "    df = df.sort_values(['Store', 'Date'], ascending=[True, False])\n",
    "    df = get_elapsed(df, fld, 'Before')\n",
    "    \n",
    "    fld = 'StateHoliday'\n",
    "    df = df.sort_values(['Store', 'Date'])\n",
    "    df = get_elapsed(df, fld, 'After')\n",
    "    df = df.sort_values(['Store', 'Date'], ascending=[True, False])\n",
    "    df = get_elapsed(df, fld, 'Before')\n",
    "    \n",
    "    fld = 'Promo'\n",
    "    df = df.sort_values(['Store', 'Date'])\n",
    "    df = get_elapsed(df, fld, 'After')\n",
    "    df = df.sort_values(['Store', 'Date'], ascending=[True, False])\n",
    "    df = get_elapsed(df, fld, 'Before')\n",
    "    \n",
    "    # set the active index to Date\n",
    "    df = df.set_index(\"Date\")\n",
    "    \n",
    "    # set null values from elapsed field calculations to 0\n",
    "    columns = ['SchoolHoliday', 'StateHoliday', 'Promo']\n",
    "\n",
    "    for o in ['Before', 'After']:\n",
    "        for p in columns:\n",
    "            a = o+p\n",
    "            df[a] = df[a].fillna(0).astype(int)\n",
    "    \n",
    "    # window functions in pandas to calculate rolling quantities\n",
    "    bwd = df[['Store']+columns].sort_index().groupby(\"Store\").rolling(7, min_periods=1).sum()\n",
    "    fwd = df[['Store']+columns].sort_index(ascending=False).groupby(\"Store\").rolling(7, min_periods=1).sum()\n",
    "    \n",
    "    # drop the Store indices grouped together in the window function\n",
    "    bwd.drop('Store',1,inplace=True)\n",
    "    bwd.reset_index(inplace=True)\n",
    "\n",
    "    fwd.drop('Store',1,inplace=True)\n",
    "    fwd.reset_index(inplace=True)\n",
    "    \n",
    "    df.reset_index(inplace=True)\n",
    "    \n",
    "    df = df.merge(bwd, 'left', ['Date', 'Store'], suffixes=['', '_bw'])\n",
    "    df = df.merge(fwd, 'left', ['Date', 'Store'], suffixes=['', '_fw'])\n",
    "    \n",
    "    df.drop(columns,1,inplace=True)\n",
    "    \n",
    "    df[\"Date\"] = pd.to_datetime(df.Date)\n",
    "    \n",
    "    finalDF = join_df(joinedDF, df, ['Store', 'Date'])\n",
    "    \n",
    "    return finalDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# applying the above function to a subset of columns\n",
    "columns_new = [\"Date\", \"Store\", \"Promo\", \"StateHoliday\", \"SchoolHoliday\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "joined = add_durations( train[columns_new], joined )\n",
    "joined_test = add_durations( test[columns_new], joined_test )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "joined = joined[joined.Sales!=0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Backup\n",
    "joined.reset_index(inplace=True)\n",
    "joined_test.reset_index(inplace=True)\n",
    "joined.to_feather(f'{PATH}joined')\n",
    "joined_test.to_feather(f'{PATH}joined_test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "joined = pd.read_feather(f'{PATH}joined')\n",
    "joined_test = pd.read_feather(f'{PATH}joined_test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Store</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DayOfWeek</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <td>2015-07-31 00:00:00</td>\n",
       "      <td>2015-07-31 00:00:00</td>\n",
       "      <td>2015-07-31 00:00:00</td>\n",
       "      <td>2015-07-31 00:00:00</td>\n",
       "      <td>2015-07-31 00:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sales</th>\n",
       "      <td>5263</td>\n",
       "      <td>6064</td>\n",
       "      <td>8314</td>\n",
       "      <td>13995</td>\n",
       "      <td>4822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Customers</th>\n",
       "      <td>555</td>\n",
       "      <td>625</td>\n",
       "      <td>821</td>\n",
       "      <td>1498</td>\n",
       "      <td>559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Open</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Promo</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>StateHoliday</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SchoolHoliday</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <td>2015</td>\n",
       "      <td>2015</td>\n",
       "      <td>2015</td>\n",
       "      <td>2015</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Month</th>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Week</th>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Day</th>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dayofweek</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dayofyear</th>\n",
       "      <td>212</td>\n",
       "      <td>212</td>\n",
       "      <td>212</td>\n",
       "      <td>212</td>\n",
       "      <td>212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Is_month_end</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Is_month_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Is_quarter_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Is_quarter_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Is_year_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Is_year_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Elapsed</th>\n",
       "      <td>1438300800</td>\n",
       "      <td>1438300800</td>\n",
       "      <td>1438300800</td>\n",
       "      <td>1438300800</td>\n",
       "      <td>1438300800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>StoreType</th>\n",
       "      <td>c</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>c</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Assortment</th>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>c</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CompetitionDistance</th>\n",
       "      <td>1270</td>\n",
       "      <td>570</td>\n",
       "      <td>14130</td>\n",
       "      <td>620</td>\n",
       "      <td>29910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CompetitionOpenSinceMonth</th>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CompetitionOpenSinceYear</th>\n",
       "      <td>2008</td>\n",
       "      <td>2007</td>\n",
       "      <td>2006</td>\n",
       "      <td>2009</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Promo2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Promo2SinceWeek</th>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Promo2SinceYear</th>\n",
       "      <td>1900</td>\n",
       "      <td>2010</td>\n",
       "      <td>2011</td>\n",
       "      <td>1900</td>\n",
       "      <td>1900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PromoInterval</th>\n",
       "      <td>None</td>\n",
       "      <td>Jan,Apr,Jul,Oct</td>\n",
       "      <td>Jan,Apr,Jul,Oct</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>State</th>\n",
       "      <td>HE</td>\n",
       "      <td>TH</td>\n",
       "      <td>NW</td>\n",
       "      <td>BE</td>\n",
       "      <td>SN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>file</th>\n",
       "      <td>Rossmann_DE_HE</td>\n",
       "      <td>Rossmann_DE_TH</td>\n",
       "      <td>Rossmann_DE_NW</td>\n",
       "      <td>Rossmann_DE_BE</td>\n",
       "      <td>Rossmann_DE_SN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>week</th>\n",
       "      <td>2015-08-02 - 2015-08-08</td>\n",
       "      <td>2015-08-02 - 2015-08-08</td>\n",
       "      <td>2015-08-02 - 2015-08-08</td>\n",
       "      <td>2015-08-02 - 2015-08-08</td>\n",
       "      <td>2015-08-02 - 2015-08-08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trend</th>\n",
       "      <td>85</td>\n",
       "      <td>80</td>\n",
       "      <td>86</td>\n",
       "      <td>74</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>file_DE</th>\n",
       "      <td>Rossmann_DE</td>\n",
       "      <td>Rossmann_DE</td>\n",
       "      <td>Rossmann_DE</td>\n",
       "      <td>Rossmann_DE</td>\n",
       "      <td>Rossmann_DE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>week_DE</th>\n",
       "      <td>2015-08-02 - 2015-08-08</td>\n",
       "      <td>2015-08-02 - 2015-08-08</td>\n",
       "      <td>2015-08-02 - 2015-08-08</td>\n",
       "      <td>2015-08-02 - 2015-08-08</td>\n",
       "      <td>2015-08-02 - 2015-08-08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trend_DE</th>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date_DE</th>\n",
       "      <td>2015-08-02 00:00:00</td>\n",
       "      <td>2015-08-02 00:00:00</td>\n",
       "      <td>2015-08-02 00:00:00</td>\n",
       "      <td>2015-08-02 00:00:00</td>\n",
       "      <td>2015-08-02 00:00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 0                        1  \\\n",
       "index                                            0                        1   \n",
       "Store                                            1                        2   \n",
       "DayOfWeek                                        5                        5   \n",
       "Date                           2015-07-31 00:00:00      2015-07-31 00:00:00   \n",
       "Sales                                         5263                     6064   \n",
       "Customers                                      555                      625   \n",
       "Open                                             1                        1   \n",
       "Promo                                            1                        1   \n",
       "StateHoliday                                 False                    False   \n",
       "SchoolHoliday                                    1                        1   \n",
       "Year                                          2015                     2015   \n",
       "Month                                            7                        7   \n",
       "Week                                            31                       31   \n",
       "Day                                             31                       31   \n",
       "Dayofweek                                        4                        4   \n",
       "Dayofyear                                      212                      212   \n",
       "Is_month_end                                  True                     True   \n",
       "Is_month_start                               False                    False   \n",
       "Is_quarter_end                               False                    False   \n",
       "Is_quarter_start                             False                    False   \n",
       "Is_year_end                                  False                    False   \n",
       "Is_year_start                                False                    False   \n",
       "Elapsed                                 1438300800               1438300800   \n",
       "StoreType                                        c                        a   \n",
       "Assortment                                       a                        a   \n",
       "CompetitionDistance                           1270                      570   \n",
       "CompetitionOpenSinceMonth                        9                       11   \n",
       "CompetitionOpenSinceYear                      2008                     2007   \n",
       "Promo2                                           0                        1   \n",
       "Promo2SinceWeek                                  1                       13   \n",
       "Promo2SinceYear                               1900                     2010   \n",
       "PromoInterval                                 None          Jan,Apr,Jul,Oct   \n",
       "State                                           HE                       TH   \n",
       "file                                Rossmann_DE_HE           Rossmann_DE_TH   \n",
       "week                       2015-08-02 - 2015-08-08  2015-08-02 - 2015-08-08   \n",
       "trend                                           85                       80   \n",
       "file_DE                                Rossmann_DE              Rossmann_DE   \n",
       "week_DE                    2015-08-02 - 2015-08-08  2015-08-02 - 2015-08-08   \n",
       "trend_DE                                        83                       83   \n",
       "Date_DE                        2015-08-02 00:00:00      2015-08-02 00:00:00   \n",
       "\n",
       "                                                 2                        3  \\\n",
       "index                                            2                        3   \n",
       "Store                                            3                        4   \n",
       "DayOfWeek                                        5                        5   \n",
       "Date                           2015-07-31 00:00:00      2015-07-31 00:00:00   \n",
       "Sales                                         8314                    13995   \n",
       "Customers                                      821                     1498   \n",
       "Open                                             1                        1   \n",
       "Promo                                            1                        1   \n",
       "StateHoliday                                 False                    False   \n",
       "SchoolHoliday                                    1                        1   \n",
       "Year                                          2015                     2015   \n",
       "Month                                            7                        7   \n",
       "Week                                            31                       31   \n",
       "Day                                             31                       31   \n",
       "Dayofweek                                        4                        4   \n",
       "Dayofyear                                      212                      212   \n",
       "Is_month_end                                  True                     True   \n",
       "Is_month_start                               False                    False   \n",
       "Is_quarter_end                               False                    False   \n",
       "Is_quarter_start                             False                    False   \n",
       "Is_year_end                                  False                    False   \n",
       "Is_year_start                                False                    False   \n",
       "Elapsed                                 1438300800               1438300800   \n",
       "StoreType                                        a                        c   \n",
       "Assortment                                       a                        c   \n",
       "CompetitionDistance                          14130                      620   \n",
       "CompetitionOpenSinceMonth                       12                        9   \n",
       "CompetitionOpenSinceYear                      2006                     2009   \n",
       "Promo2                                           1                        0   \n",
       "Promo2SinceWeek                                 14                        1   \n",
       "Promo2SinceYear                               2011                     1900   \n",
       "PromoInterval                      Jan,Apr,Jul,Oct                     None   \n",
       "State                                           NW                       BE   \n",
       "file                                Rossmann_DE_NW           Rossmann_DE_BE   \n",
       "week                       2015-08-02 - 2015-08-08  2015-08-02 - 2015-08-08   \n",
       "trend                                           86                       74   \n",
       "file_DE                                Rossmann_DE              Rossmann_DE   \n",
       "week_DE                    2015-08-02 - 2015-08-08  2015-08-02 - 2015-08-08   \n",
       "trend_DE                                        83                       83   \n",
       "Date_DE                        2015-08-02 00:00:00      2015-08-02 00:00:00   \n",
       "\n",
       "                                                 4  \n",
       "index                                            4  \n",
       "Store                                            5  \n",
       "DayOfWeek                                        5  \n",
       "Date                           2015-07-31 00:00:00  \n",
       "Sales                                         4822  \n",
       "Customers                                      559  \n",
       "Open                                             1  \n",
       "Promo                                            1  \n",
       "StateHoliday                                 False  \n",
       "SchoolHoliday                                    1  \n",
       "Year                                          2015  \n",
       "Month                                            7  \n",
       "Week                                            31  \n",
       "Day                                             31  \n",
       "Dayofweek                                        4  \n",
       "Dayofyear                                      212  \n",
       "Is_month_end                                  True  \n",
       "Is_month_start                               False  \n",
       "Is_quarter_end                               False  \n",
       "Is_quarter_start                             False  \n",
       "Is_year_end                                  False  \n",
       "Is_year_start                                False  \n",
       "Elapsed                                 1438300800  \n",
       "StoreType                                        a  \n",
       "Assortment                                       a  \n",
       "CompetitionDistance                          29910  \n",
       "CompetitionOpenSinceMonth                        4  \n",
       "CompetitionOpenSinceYear                      2015  \n",
       "Promo2                                           0  \n",
       "Promo2SinceWeek                                  1  \n",
       "Promo2SinceYear                               1900  \n",
       "PromoInterval                                 None  \n",
       "State                                           SN  \n",
       "file                                Rossmann_DE_SN  \n",
       "week                       2015-08-02 - 2015-08-08  \n",
       "trend                                           82  \n",
       "file_DE                                Rossmann_DE  \n",
       "week_DE                    2015-08-02 - 2015-08-08  \n",
       "trend_DE                                        83  \n",
       "Date_DE                        2015-08-02 00:00:00  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined.head().T.head(40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert input into a compatible form with neural network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "844338"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_vars = ['Store', 'DayOfWeek', 'Year', 'Month', 'Day', 'StateHoliday', 'CompetitionMonthsOpen',\n",
    "    'Promo2Weeks', 'StoreType', 'Assortment', 'PromoInterval', 'CompetitionOpenSinceYear', 'Promo2SinceYear',\n",
    "    'State', 'Week', 'Events', 'Promo_fw', 'Promo_bw', 'StateHoliday_fw', 'StateHoliday_bw',\n",
    "    'SchoolHoliday_fw', 'SchoolHoliday_bw']\n",
    "\n",
    "contin_vars = ['CompetitionDistance', 'Max_TemperatureC', 'Mean_TemperatureC', 'Min_TemperatureC',\n",
    "   'Max_Humidity', 'Mean_Humidity', 'Min_Humidity', 'Max_Wind_SpeedKm_h', \n",
    "   'Mean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE',\n",
    "   'AfterStateHoliday', 'BeforeStateHoliday', 'Promo', 'SchoolHoliday']\n",
    "\n",
    "n = len(joined); n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "dep = 'Sales' # dependent variable\n",
    "joined = joined[cat_vars+contin_vars+[dep, 'Date']].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a column for dependent variable in test\n",
    "joined_test[dep] = 0\n",
    "joined_test = joined_test[cat_vars+contin_vars+[dep, 'Date', 'Id']].copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change categorical data columns into categories identified by the neural network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "??apply_cats # is a function to convert strings of categorical variables to codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_cats(df, trn):\n",
    "    \"\"\"Changes any columns of strings in df into categorical variables using trn as\n",
    "    a template for the category codes.\n",
    "    Parameters:\n",
    "    -----------\n",
    "    df: A pandas dataframe. Any columns of strings will be changed to\n",
    "        categorical values. The category codes are determined by trn.\n",
    "    trn: A pandas dataframe. When creating a category for df, it looks up the\n",
    "        what the category's code were in trn and makes those the category codes\n",
    "        for df.\n",
    "    Examples:\n",
    "    ---------\n",
    "    >>> df = pd.DataFrame({'col1' : [1, 2, 3], 'col2' : ['a', 'b', 'a']})\n",
    "    >>> df\n",
    "       col1 col2\n",
    "    0     1    a\n",
    "    1     2    b\n",
    "    2     3    a\n",
    "    note the type of col2 is string\n",
    "    >>> train_cats(df)\n",
    "    >>> df\n",
    "       col1 col2\n",
    "    0     1    a\n",
    "    1     2    b\n",
    "    2     3    a\n",
    "    now the type of col2 is category {a : 1, b : 2}\n",
    "    >>> df2 = pd.DataFrame({'col1' : [1, 2, 3], 'col2' : ['b', 'a', 'a']})\n",
    "    >>> apply_cats(df2, df)\n",
    "           col1 col2\n",
    "        0     1    b\n",
    "        1     2    a\n",
    "        2     3    a\n",
    "    now the type of col is category {a : 1, b : 2}\n",
    "    \"\"\"\n",
    "    for n,c in df.items():\n",
    "        if (n in trn.columns) and (trn[n].dtype.name=='category'):\n",
    "            df[n] = pd.Categorical(c, categories=trn[n].cat.categories, ordered=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "for v in cat_vars: joined[v] = joined[v].astype('category').cat.as_ordered()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "apply_cats(joined_test, joined)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pytorch wants the continous values to be in Float32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "for v in contin_vars:\n",
    "    joined[v] = joined[v].astype('float32')\n",
    "    joined_test[v] = joined_test[v].astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do the final preprocessing steps: Replace NAs by median, and add a new column to indicate NA rows, and numericalize the whole data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn_pandas import DataFrameMapper\n",
    "from pandas.api.types import is_string_dtype, is_numeric_dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use just a subset as a sample. Later we will use the full set of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150000"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idxs = get_cv_idxs(n, val_pct=150000/n)\n",
    "joined_samp = joined.iloc[idxs].set_index(\"Date\")\n",
    "samp_size = len(joined_samp); samp_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "??proc_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "process data frame function:\n",
    "1. Pulls out the dependent variable, puts it into a separate variable, and deletes it from the original data frame. In other words, df do not have `Sales` column, and `y` only contains `Sales` column.\n",
    "2. `do_scale` : Neural nets really like to have the input data to all be somewhere around zero with a standard deviation of somewhere around 1. So we take our data, subtract the mean, and divide by the standard deviation to make that happen. It returns a special object which keeps track of what mean and standard deviation it used for that normalization so you can do the same to the test set later (`mapper`).\n",
    "3. It also handles missing values — for categorical variable, it becomes ID: 0 and other categories become 1, 2, 3, and so on. For continuous variable, it replaces the missing value with the median and create a new boolean column that says whether it was missing or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "df, y, nas, mapper = proc_df(joined_samp, 'Sales', do_scale=True)\n",
    "yl = np.log(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "joined_test = joined_test.set_index(\"Date\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test, _, nas, mapper = proc_df(joined_test, 'Sales', do_scale=True, skip_flds=['Id'],\n",
    "                                  mapper=mapper, na_dict=nas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking a random set of validation data is not ideal for time series data. I will take the final 25% of data sorted by date as the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "112500\n"
     ]
    }
   ],
   "source": [
    "train_ratio = 0.75\n",
    "train_size = int(samp_size * train_ratio); print(train_size)\n",
    "val_idx = list(range(train_size, len(df)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a model data object below, which has a validation set, training set, and optional test set built into it. Unlike image recognition data, where these data sets can be read-in by directory structure, here we use `ColumnarDataset` class and get data from our data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "((val_df, trn_df), (val_y, trn_y)) = split_by_idx(val_idx, df, yl.astype(np.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ColumnarDataset(Dataset):\n",
    "    def __init__(self, cats, conts, y, is_reg=True, is_multi=False):\n",
    "        n = len(cats[0]) if cats else len(conts[0])\n",
    "        self.cats = (np.stack(cats, 1).astype(np.int64) if cats else np.zeros((n,1)))\n",
    "        self.conts = (np.stack(conts, 1).astype(np.float32) if conts else np.zeros((n,1)))\n",
    "        self.y = (np.zeros((n,1)) if y is None else y)\n",
    "        if is_reg:\n",
    "            self.y =  self.y[:,None]\n",
    "        self.is_reg = is_reg\n",
    "        self.is_multi = is_multi\n",
    "\n",
    "    def __len__(self): return len(self.y)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return [self.cats[idx], self.conts[idx], self.y[idx]]\n",
    "\n",
    "    @classmethod\n",
    "    def from_data_frames(cls, df_cat, df_cont, y=None, is_reg=True, is_multi=False):\n",
    "        cat_cols = [c.values for n,c in df_cat.items()]\n",
    "        cont_cols = [c.values for n,c in df_cont.items()]\n",
    "        return cls(cat_cols, cont_cols, y, is_reg, is_multi)\n",
    "\n",
    "    @classmethod\n",
    "    def from_data_frame(cls, df, cat_flds, y=None, is_reg=True, is_multi=False):\n",
    "        return cls.from_data_frames(df[cat_flds], df.drop(cat_flds, axis=1), y, is_reg, is_multi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_datasets = {'train': ColumnarDataset.from_data_frame(trn_df, cat_vars, trn_y),\n",
    "                   'val': ColumnarDataset.from_data_frame(val_df, cat_vars, val_y)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "112500"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(column_datasets['train'].y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_dataloaders = {x: torch.utils.data.DataLoader(column_datasets[x], batch_size=128,\n",
    "                                             shuffle=True, num_workers=0)\n",
    "              for x in ['train', 'val']}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "879"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(column_dataloaders['train'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using embedding matrices for categorical variables enables the neural network to identify patterns between classes within a given categorical variable. Moreover, it will be computationally faster and less memory consuming compared to one-hot-encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Store', 1116),\n",
       " ('DayOfWeek', 8),\n",
       " ('Year', 4),\n",
       " ('Month', 13),\n",
       " ('Day', 32),\n",
       " ('StateHoliday', 3),\n",
       " ('CompetitionMonthsOpen', 26),\n",
       " ('Promo2Weeks', 27),\n",
       " ('StoreType', 5),\n",
       " ('Assortment', 4),\n",
       " ('PromoInterval', 4),\n",
       " ('CompetitionOpenSinceYear', 24),\n",
       " ('Promo2SinceYear', 9),\n",
       " ('State', 13),\n",
       " ('Week', 53),\n",
       " ('Events', 22),\n",
       " ('Promo_fw', 7),\n",
       " ('Promo_bw', 7),\n",
       " ('StateHoliday_fw', 4),\n",
       " ('StateHoliday_bw', 4),\n",
       " ('SchoolHoliday_fw', 9),\n",
       " ('SchoolHoliday_bw', 9)]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_sz = [(c, len(joined_samp[c].cat.categories)+1) for c in cat_vars]\n",
    "cat_sz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jeremy's rule of thumb for the size of the embediing matrix is to use the minimum of the cardinality+1 and 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1116, 50),\n",
       " (8, 4),\n",
       " (4, 2),\n",
       " (13, 7),\n",
       " (32, 16),\n",
       " (3, 2),\n",
       " (26, 13),\n",
       " (27, 14),\n",
       " (5, 3),\n",
       " (4, 2),\n",
       " (4, 2),\n",
       " (24, 12),\n",
       " (9, 5),\n",
       " (13, 7),\n",
       " (53, 27),\n",
       " (22, 11),\n",
       " (7, 4),\n",
       " (7, 4),\n",
       " (4, 2),\n",
       " (4, 2),\n",
       " (9, 5),\n",
       " (9, 5)]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb_szs = [(c, min(50, (c+1)//2)) for _,c in cat_sz]\n",
    "emb_szs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def emb_init(x):\n",
    "    x = x.weight.data\n",
    "    sc = 2/(x.size(1)+1)\n",
    "    x.uniform_(-sc,sc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will pass these embedding sizes to the learner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Learning Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model will contain an embedding layer with dropout, a fully connected linear layer followed by a ReLu, another linear layer, then a softmax."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Rossman challenge, the metric used to judge the performance is Root Mean Square Percentage Error (RMSPE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def inv_y(a): return np.exp(a)\n",
    "def exp_rmspe(y_pred, targ):\n",
    "    targ = inv_y(targ)\n",
    "    pct_var = (targ - inv_y(y_pred))/targ\n",
    "    return math.sqrt((pct_var**2).mean())\n",
    "max_log_y = np.max(yl)\n",
    "y_range = (0, max_log_y*1.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 12.63454087942548)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_range"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Plan:\n",
    "* We started by creating model data object which has a validation set, training set, and optional test set built into it. From that, we will get a `learner`, we will then optionally call `lr_find`, then train the model (call `learn.fit` in fast.ai) and so forth.\n",
    "* In image recognition, we can use path and directory structure to divide the data in to training, and validation sets. However, in structure data we use `columanarDataSet` class and load data from data frame.\n",
    "* We defined the embedding matrices for our categorical variables.\n",
    "* Feed the aforementioned standard model data object into the `learner`.\n",
    "\n",
    "The Model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MixedInputModel(nn.Module):\n",
    "    def __init__(self, emb_szs, n_cont, emb_drop, out_sz, szs, drops,\n",
    "                 y_range=None, use_bn=False, is_reg=True, is_multi=False):\n",
    "        super().__init__()\n",
    "        self.embs = nn.ModuleList([nn.Embedding(c, s) for c,s in emb_szs])\n",
    "        for emb in self.embs: emb_init(emb)\n",
    "        n_emb = sum(e.embedding_dim for e in self.embs)\n",
    "        self.n_emb, self.n_cont=n_emb, n_cont\n",
    "        \n",
    "        szs = [n_emb+n_cont] + szs\n",
    "        self.lins = nn.ModuleList([\n",
    "            nn.Linear(szs[i], szs[i+1]) for i in range(len(szs)-1)])\n",
    "        self.bns = nn.ModuleList([\n",
    "            nn.BatchNorm1d(sz) for sz in szs[1:]])\n",
    "        for o in self.lins: kaiming_normal(o.weight.data)\n",
    "        self.outp = nn.Linear(szs[-1], out_sz)\n",
    "        kaiming_normal(self.outp.weight.data)\n",
    "\n",
    "        self.emb_drop = nn.Dropout(emb_drop)\n",
    "        self.drops = nn.ModuleList([nn.Dropout(drop) for drop in drops])\n",
    "        self.bn = nn.BatchNorm1d(n_cont)\n",
    "        self.use_bn,self.y_range = use_bn,y_range\n",
    "        self.is_reg = is_reg\n",
    "        self.is_multi = is_multi\n",
    "\n",
    "    def forward(self, x_cat, x_cont):\n",
    "        if self.n_emb != 0:\n",
    "            x = [e(x_cat[:,i]) for i,e in enumerate(self.embs)]\n",
    "            x = torch.cat(x, 1)\n",
    "            x = self.emb_drop(x)\n",
    "        if self.n_cont != 0:\n",
    "            x2 = self.bn(x_cont)\n",
    "            x = torch.cat([x, x2], 1) if self.n_emb != 0 else x2\n",
    "        for l,d,b in zip(self.lins, self.drops, self.bns):\n",
    "            x = F.relu(l(x))\n",
    "            if self.use_bn: x = b(x)\n",
    "            x = d(x)\n",
    "        x = self.outp(x)\n",
    "        if not self.is_reg:\n",
    "            if self.is_multi:\n",
    "                x = F.sigmoid(x)\n",
    "            else:\n",
    "                x = F.log_softmax(x)\n",
    "        elif self.y_range:\n",
    "            x = F.sigmoid(x)\n",
    "            x = x*(self.y_range[1] - self.y_range[0])\n",
    "            x = x+self.y_range[0]\n",
    "        return x\n",
    "    \n",
    "    def inv_y(self,a): return np.exp(a)\n",
    "\n",
    "    def exp_rmspe(self,y_pred, targ):\n",
    "        targ = self.inv_y(targ.cpu().data)\n",
    "        pct_var = (targ - self.inv_y(y_pred.cpu().data))/targ\n",
    "        return math.sqrt((pct_var**2).mean())\n",
    "\n",
    "    def calculate_loss(self, outputs, targets, **kwargs):\n",
    "        return F.mse_loss(input=outputs, target=targets)\n",
    "        \n",
    "    def calculate_metrics(self, outputs, targets, **kwargs):\n",
    "        classification_accuracy = 1-self.exp_rmspe(outputs,targets)\n",
    "        return classification_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MixedInputModel(emb_szs, len(df.columns)-len(cat_vars),\n",
    "                   0.04, 1, [1000,500], [0.001,0.01], y_range=y_range)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Description:\n",
    "* `emb_szs` : sizes of embedding matrices\n",
    "* `len(df.columns)-len(cat_vars)` : number of continuous variables\n",
    "* `0.04` : how much dropout to use in embedding layer\n",
    "* `1` : output size\n",
    "* `[1000,500]` : how many activations to have in each layer\n",
    "* `[0.001,0.01]` : how many dropout to use at later layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use Adam as the optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer_col= optim.Adam(model.parameters(),lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train_col = train.Trainer(m~odel,column_dataloaders['train'],column_dataloaders['val'],optimizer_col)\n",
    "# the above command returned an error about Trainer because I didn't want to import the custom library dllib\n",
    "# So decided to use the fast.ai columnar data loader instead\n",
    "# Use fast.ai columnar data loader\n",
    "md = ColumnarModelData.from_data_frame(PATH, val_idx, df, yl.astype(np.float32), cat_flds=cat_vars, bs=128, \n",
    "                                  test_df=df_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fast.ai `get_learner` builds a MixedInputModel for us"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars),\n",
    "                   0.04, 1, [1000,500], [0.001,0.01], y_range=y_range)\n",
    "lr = 1e-3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learning rate survey:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fce94f0c925643b4a5d3eaf0fbde2f47",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 66%|██████▌   | 581/879 [00:30<00:15, 19.36it/s, loss=0.683]"
     ]
    }
   ],
   "source": [
    "m.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEOCAYAAAB4nTvgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VNX5+PHPk4QkQELYQoAkEMAAhh3CakVc2FwAxQVcikt/iJWvVmtbrbW1WqvW2mrdKW5VEREqRUVwRRTZEvYtEMKShC0QlrAkZHl+f8wFh5hkJpDJJJPn/XrNK3PPPefOk0vIk3PPveeIqmKMMcZUJMjfARhjjKn5LFkYY4zxyJKFMcYYjyxZGGOM8ciShTHGGI8sWRhjjPHIkoUxxhiPfJosRGSEiKSJSLqIPFjG/ltFJEdEVjmvX7jtmyAiW5zXBF/GaYwxpmLiq4fyRCQY2AwMBbKA5cB4Vd3gVudWIFlVJ5dq2xRIAZIBBVKBPqp60CfBGmOMqZAvexb9gHRVzVDVk8B0YLSXbYcDX6hqrpMgvgBG+ChOY4wxHoT48NixQKbbdhbQv4x6Y0VkMK5eyH2qmllO29iKPqx58+aakJBwTgEbY0xdk5qaul9Voz3V82Wy8MbHwPuqWiAidwJvA5d421hEJgITAdq0aUNKSopvojTGmAAlIju8qefLy1DZQLzbdpxTdpqqHlDVAmdzKtDH27ZO+ymqmqyqydHRHhOjMcaYs+TLZLEcSBSRdiISCowD5rhXEJFWbpujgI3O+/nAMBFpIiJNgGFOmTHGGD/w2WUoVS0Skcm4fskHA2+o6noReQxIUdU5wD0iMgooAnKBW522uSLyOK6EA/CYqub6KlZjjDEV89mts9UtOTlZbczCGGMqR0RSVTXZUz17gtsYY4xHliyMMcZ4VOeTRWFxCV9v2svuwyf8HYoxxtRYdT5Z7Msr4Pa3UpiZkuXvUIwxpsaq88kitnF9+rVrykersgmUwX5jjKlqdT5ZAFzbJ46MnGMsSMvxdyjGGFMjWbIAru4VS1yT+ry2cKu/QzHGmBrJkgVQLziIsb3jWLotl31H8v0djjHG1DiWLBxXdG+FKny5cZ+/QzHGmBrHkoUjsUUEMY3CWLR1v79DMcaYGseShUNEuKBDc5ZsPUBJid0VZYwx7ixZuBl0XnMOHDtJ2t48f4dijDE1iiULN4M6NANgUbpdijLGGHeWLNy0blyfxBYRfLZuj79DMcaYGsWSRSnXJ8eTuuMgW+xSlDHGnGbJopTRvVojgvUujDHGjSWLUlpEhtMzvjFfbNjr71CMMabG8GmyEJERIpImIuki8mAF9caKiIpIsrOdICInRGSV83rVl3GWNjQphrXZh23acmOMcfgsWYhIMPASMBJIAsaLSFIZ9SKBe4GlpXZtVdWezmuSr+Isy7CkGAC+tN6FMcYAvu1Z9APSVTVDVU8C04HRZdR7HHgaqDGTMnWIjqBd84Z8bsnCGGMA3yaLWCDTbTvLKTtNRHoD8ar6aRnt24nIShH5VkQu9GGcPyEiDE2KYUnGAY7kF1bnRxtjTI3ktwFuEQkC/gH8uozdu4E2qtoLuB+YJiKNyjjGRBFJEZGUnJyqXYviqu6tKSxWPliW6bmyMcYEOF8mi2wg3m07zik7JRLoCiwQke3AAGCOiCSraoGqHgBQ1VRgK9Cx9Aeo6hRVTVbV5Ojo6CoNvltcFBec14yXF6Sz/2hBlR7bGGNqG18mi+VAooi0E5FQYBww59ROVT2sqs1VNUFVE4AlwChVTRGRaGeAHBFpDyQCGT6MtUyPXtWFwycKeXfJjur+aGOMqVF8lixUtQiYDMwHNgIzVHW9iDwmIqM8NB8MrBGRVcBMYJKq5voq1vIkxkTSNTbK5ooyxtR5Ib48uKrOBeaWKvtjOXWHuL2fBczyZWzeGtShOVO/yyAvv5DI8Hr+DscYY/zCnuD2YGhSC4pKlM/X2220xpi6y5KFB73bNCG+aX1mr8r2XNkYYwKUJQsPRITRPWJZlL6ffXk15rlBY4ypVpYsvDCmV2tKFD5ZvdvfoRhjjF9YsvDCeS0i6RrbyC5FGWPqLEsWXhrTM5Y1WYfJyDnq71CMMabaWbLw0lU9XIsizV61y9+hGGNMtbNk4aWYRuEM6tCM2SuzUVV/h2OMMdXKkkUljOkZy87c46zMPOTvUIwxplpZsqiEEV1bEhYSxP9W2kC3MaZusWRRCZHh9bjs/Bg+WbObwuISf4djjDHVxpJFJY3pFcuBYyf5fotNLmiMqTssWVTSRR2jadygnj1zYYypUyxZVFJoSBCXd2vF5+v3cqygyN/hGGNMtbBkcRau7hXLicJiPlu3x9+hGGNMtbBkcRaS2zahXfOGzEix9bmNMXWDT5OFiIwQkTQRSReRByuoN1ZEVESS3coectqlichwX8ZZWSLC1b1iWb49lz2HbSZaY0zg81mycNbQfgkYCSQB40UkqYx6kcC9wFK3siRca3Z3AUYAL59ak7umuLxbK1Ths3U2E60xJvD5smfRD0hX1QxVPQlMB0aXUe9x4GnA/U/00cB0VS1Q1W1AunO8GuO8FhF0bhnJp2ssWRhjAp8vk0Us4H5RP8spO01EegPxqvppZdvWBFd2b0XKjoNk5h73dyjGGONTfhvgFpEg4B/Ar8/hGBNFJEVEUnJycqouOC9d3TuOIIH3lu6s9s82xpjq5MtkkQ3Eu23HOWWnRAJdgQUish0YAMxxBrk9tQVAVaeoarKqJkdHR1dx+J7FNq7PiK4teW/JDg4dP1ntn2+MMdXFl8liOZAoIu1EJBTXgPWcUztV9bCqNlfVBFVNAJYAo1Q1xak3TkTCRKQdkAgs82GsZ+2eSxPJKyjig+V2G60xJnD5LFmoahEwGZgPbARmqOp6EXlMREZ5aLsemAFsAOYBd6tqsa9iPRedWzaiW2wUc+0BPWNMAAvx5cFVdS4wt1TZH8upO6TU9hPAEz4LrgqN6NqSZ+ansevQCVo3ru/vcIwxpsrZE9xVYGTXlgDMs96FMSZAWbKoAu2jXc9c2Ey0xphAZcmiiozv14Y1WYdZbUuuGmMCkCWLKnJN71gahgbzn8U7/B2KMcZUOUsWVSQyvB5X947l4zW7OHjMnrkwxgQWSxZV6JYBCZwsKuHDVHvmwhgTWCxZVKFOLSPpEd+YT9faXVHGmMBiyaKKXda5BaszD7Evz9a5MMYEDksWVWyE88zFRyvsNlpjTOCwZFHFEmMi6d+uKe8t3Ymq+jscY4ypEpYsfGBs7zh25h5n/a4j/g7FGGOqhCULHxiaFENwkDB3ra2iZ4wJDJYsfKBJw1AGtm/G3LW77VKUMSYgWLLwkcu7tWL7geOk7Djo71CMMeacWbLwkTG9WtM8IoxnP0+z3oUxptazZOEjDUJDuPviDizJyGVR+gF/h2OMMefEkoUP3di/Da2jwnnGehfGmFrOp8lCREaISJqIpIvIg2XsnyQia0VklYh8LyJJTnmCiJxwyleJyKu+jNNXwkKCuefSRFZnHuLLjfv8HY4xxpw1nyULEQkGXgJGAknA+FPJwM00Ve2mqj2BvwH/cNu3VVV7Oq9JvorT18b2iSOhWQNeXpDu71CMMeas+bJn0Q9IV9UMVT0JTAdGu1dQVfen1hoCAXetpl5wEDf1b8vKnYdI35fn73CMMeas+DJZxALuc3VnOWVnEJG7RWQrrp7FPW672onIShH5VkQu9GGcPnd171hCQ4KY+t02f4dijDFnxe8D3Kr6kqp2AH4H/MEp3g20UdVewP3ANBFpVLqtiEwUkRQRScnJyam+oCupeUQYNyTHM2tFli2MZIyplXyZLLKBeLftOKesPNOBMQCqWqCqB5z3qcBWoGPpBqo6RVWTVTU5Ojq6ygL3hfH92lBYrHxiU4AYY2ohXyaL5UCiiLQTkVBgHDDHvYKIJLptXgFsccqjnQFyRKQ9kAhk+DBWnzu/VSSdYiL5aEWWv0MxxphK81myUNUiYDIwH9gIzFDV9SLymIiMcqpNFpH1IrIK1+WmCU75YGCNUz4TmKSqub6KtTqICFf3jmXFzkNs33/M3+EYY0ylSKA8LJacnKwpKSn+DqNCuw+fYNBTX3PPJYncN/QnV9WMMabaiUiqqiZ7quf3Ae66pFVUfQa2b8bM1CwKior9HY4xxnjNkkU1m3RRB7IPneA/P+zwdyjGGOM1SxbVbHDHaAa2b8ZbP2ynuCQwLgEaYwKfJQs/mDAogexDJ5iRkum5sjHG1ACWLPxgeJcY+iU05el5m8i1h/SMMbWAJQs/EBEeH9OVvPwipiys1Y+PGGPqCEsWftKpZSSXdG7BzNRMjhYU+TscY4ypkCULP5o4uD2Hjhfymw9X+zsUY4ypkCULP+qb0JT/N7g989fvYd+RfH+HY4wx5bJk4WfXJ8ejwMsLtvo7FGOMKZclCz9r17whPx/QlrcXb2drzlF/h2OMMWWyZFEDTL4kkXrBQUz9zu6MMsbUTJYsaoDoyDCu7RPHrNRs9trYhTGmBrJkUUNMvLA9CDzw4WoCZSZgY0zgsGRRQyQ0b8hvh3fiuy37WbHzkL/DMcaYM1iyqEHG92tDo/AQ/vnFZutdGGNqFEsWNUjDsBB+M7wT36fvZ966Pf4OxxhjTvNpshCRESKSJiLpIvJgGfsnichaEVklIt+LSJLbvoecdmkiMtyXcdYkN/ZvS2KLCJ75PI2i4hJ/h2OMMYCXyUJE7hWRRuLyuoisEJFhHtoEAy8BI4EkYLx7MnBMU9VuqtoT+BvwD6dtEjAO6AKMAF52jhfwgoOEB4Z3IiPnGLNWZPk7HGOMAbzvWdyuqkeAYUAT4BbgKQ9t+gHpqpqhqieB6cBo9wrOMU9pCJy6UD8amK6qBaq6DUh3jlcnDEuKoWd8Y/75xRZOnLTlV40x/udtshDn6+XAO6q63q2sPLGA++o+WU7ZmQcWuVtEtuLqWdxTmbaBSkR4aGRn9hzJ54Wvt/g7HGOM8TpZpIrI57iSxXwRiQSq5IK6qr6kqh2A3wF/qExbEZkoIikikpKTk1MV4dQY/ds345pesUz9fht7DtuDesYY//I2WdwBPAj0VdXjQD3gNg9tsoF4t+04p6w804ExlWmrqlNUNVlVk6Ojoz2EU/vcN7Sj6+sHq2y9bmOMX3mbLAYCaap6SERuxtUDOOyhzXIgUUTaiUgorgHrOe4VRCTRbfMK4NQ1lznAOBEJE5F2QCKwzMtYA0Z80wb8ZXRXFmccYPrynf4OxxhTh3mbLF4BjotID+DXwFbgPxU1UNUiYDIwH9gIzFDV9SLymIiMcqpNFpH1IrIKuB+Y4LRdD8wANgDzgLtVtU6O9F6XHMfA9s144tONpO/L83c4xpg6Srx5UlhEVqhqbxH5I5Ctqq+fKvN9iN5JTk7WlJQUf4fhE3uP5DPy+e+IbVyf//5yEPWC7VlKY0zVEJFUVU32VM/b3zp5IvIQrltmPxWRIFzjFqYaxDQK54kxXVmbfZgnPt1oU4EYY6qdt8niBqAA1/MWe3ANOD/js6jMT4zs1opbByXw1g/beXj2OksYxphq5VWycBLEe0CUiFwJ5KtqhWMWpur96aok7ryoPdOW7uTDVHu62xhTfbyd7uN6XHcjXQdcDywVkWt9GZj5KRHht8M7M6B9Ux6ds54MW4bVGFNNvL0M9TCuZywmqOrPcU298YjvwjLlCQ4S/nlDT0JDgrh3+ipOFtlkg8YY3/M2WQSp6j637QOVaGuqWKuo+jw9tjtrsw/z7Odp/g7HGFMHePsLf56IzBeRW0XkVuBTYK7vwjKeDO/Skpv6t+G1hRl8tyWwpjoxxtQ83g5w/waYAnR3XlNU9Xe+DMx49ocrkjivRQS/nbmGYwVF/g7HGBPAvL6UpKqzVPV+5/WRL4My3qkfGszTY7ux+3A+by7a5u9wjDEBrMJkISJ5InKkjFeeiBypqK2pHn3aNuWijtG8sWg72YdO+DscY0yAqjBZqGqkqjYq4xWpqo2qK0hTsd9ffj6FxSXc8/5KSmx2WmOMD9gdTQGgU8tI/nhlEqk7DvL24u3+DscYE4AsWQSIsb3juLRzCx77ZANTv8vwdzjGmABjySJABAUJL97Ym+FJLfnLpxv5MCXTcyNjjPGSJYsAUj80mJdu6k3/dk15+KN1zF27298hGWMChCWLABMcJEy5JZlucVFMnraC1ZmH/B2SMSYAWLIIQFEN6vHmbX1p2jCM382yB/aMMefOp8lCREaISJqIpIvIg2Xsv19ENojIGhH5SkTauu0rFpFVzmtO6bamYo3C6/H367qzeW8e932wyta/MMacE58lCxEJBl4CRgJJwHgRSSpVbSWQrKrdgZnA39z2nVDVns5rFKbShnRqwe8vP5/PN+zlvaU7/R2OMaYW82XPoh+QrqoZqnoSmA6Mdq+gqt+o6nFncwmuFfhMFbr9gnZcmNicP81Zz/z1e/wdjjGmlvJlsogF3O/fzHLKynMH8JnbdriIpIjIEhEZ44sA64KgIOGVm/vQLTaKe95fybb9x/wdkjGmFqoRA9wicjOQzJnrerdV1WTgRuA5EelQRruJTkJJycmxabrLExEWwmu39CE0JIhbXl/KzgPHPTcyxhg3vkwW2UC823acU3YGEbkM10p8o1S14FS5qmY7XzOABUCv0m1VdYqqJqtqcnR0dNVGH2BiGoXz7h39ycsv4oYpi1m2LdffIRljahFfJovlQKKItBORUGAccMZdTSLSC3gNV6LY51beRETCnPfNgQuADT6MtU7oEd+Yd+7oR3i9YG5+fSkzU7MoKrZlWY0xnvksWahqETAZmA9sBGao6noReUxETt3d9AwQAXxY6hbZ84EUEVkNfAM8paqWLKpA97jGzLprEE0bhPLAh6t58rNN/g7JGFMLSKDcf5+cnKwpKSn+DqPW2L7/GPd+sIq1WYd45MokJgxMIChI/B2WMaaaiUiqMz5coRoxwG2qX0Lzhkz7RX+GdGrBnz/ewIvfpPs7JGNMDWbJog5rGBbC6xOSGdWjNc9/tcUGvY0x5bJkUceJCH+5uiuxjetz09Ql/HuhrYVhjPkpSxaGRuH1mH33BVzSuQVPzN3IlIVb/R2SMaaGsWRhAGjaMJSXb+rDFd1a8de5m5j0Tir5hcX+DssYU0NYsjCnBQcJ/7yhJ/dd1pF56/cw6d1UCoosYRhjLFmYUkJDgrj3skSevKYbC9JyuPu9FZwssgf3jKnrLFmYMo3v14bHx3Tly437eHqePbhnTF0X4u8ATM11y4C2bNmbx+vfbyOhWQNuGZjg75CMMX5iPQtToT9ckcSlnVvwyP/Wc/2rNgGhMXWVJQtTodCQIP5+XQ+u6xNH1sHj/PyNpWzac8TfYRljqpklC+NRk4ahPHNdD/43+Wc0CA3hV9NXkZlra2IYU5dYsjBei44M45839CT74AlGPv8dj3+ygR0HbOU9Y+oCSxamUi7qGM2n91zIkE7RvP3Ddob+YyEL0vZ5bmiMqdUsWZhKa9OsAS/e2JvvfncxiTER3PlOKvPW7fZ3WMYYH7JkYc5aq6j6vHNHfzq3asRd763g2c/T+GTNLlt9z5gAZMnCnJOmDUP5YOIA+iY05YWv05k8bSW3v51CTl6B58bGmFrDp8lCREaISJqIpIvIg2Xsv19ENojIGhH5SkTauu2bICJbnNcEX8Zpzk14vWCmTkjmoZGduffSRBal72fEcwvZsMtusTUmUPhsWVURCQY2A0OBLGA5MN59LW0RuRhYqqrHReQuYIiq3iAiTYEUIBlQIBXoo6oHy/s8W1a15ti8N48JbyyjsLiEuy8+j/bREfSIi6JReD1butWYGsbbZVV9Od1HPyBdVTOcgKYDo4HTyUJVv3GrvwS42Xk/HPhCVXOdtl8AI4D3fRivqSIdYyKZOiGZB2et5c8fn/7nJjQ4iBv6xvPwFecTXi/YjxEaYyrLl8kiFsh0284C+ldQ/w7gswraxlZpdManurSOYs7kC1iTdZjMg8d5Zn4araPq8+7SHSzblkuvNo1p26whCc0aMKJrS0Ssx2FMTVYjJhIUkZtxXXK6qJLtJgITAdq0aeODyMy5EBF6xDemR3xjruzeGoDP1u7mb/PTmL78x78FmjYMpVtsFJkHj9OyUTj3De1I34Sm/grbGFMGXyaLbCDebTvOKTuDiFwGPAxcpKoFbm2HlGq7oHRbVZ0CTAHXmEVVBG18a2S3Vozs1oqMnKMcyS9i/vo9vLJgKynbczl2spiMnGP8sHUxf7jifO74WTvrcRhTQ/hygDsE1wD3pbh++S8HblTV9W51egEzgRGqusWtvCmuQe3eTtEKXAPc5U55agPctZOqkrrjIN3iosjLL6JhaAj3Tl/J5xv2cknnFgzpFE3/ds1oH92QesF2p7cxVc3vA9yqWiQik4H5QDDwhqquF5HHgBRVnQM8A0QAHzp/Qe5U1VGqmisij+NKMACPVZQoTO0lIiQ7l5zCIlyD3q/e3IfXv9/Gv77awtebXFOJJLaI4O3b+9G6cX2/xWpMXeaznkV1s55F4CksLmHT7jw+W7ebdxbvoGFYCO/c0Y/EmEh/h2ZMwPC2Z2H9elNj1QsOoltcFL8d0ZkZkwZSrMqtby5n/1F7OtyY6mbJwtQK57dqxBsT+nLgWAGD//YNM5Znem5kjKkylixMrdEtLoo3bu1Lh+gIfjtrDf9ZvJ1AuYxqTE1nycLUKoM6NOfDSQO5uFM0f/zfeh7671qKSyxhGONrlixMrRNeL5jXJ/Rl8sXnMX15Jhc89TWPfbyBQpsa3RifsWRhaqWgIOGB4Z145Mok9hzJ541F27j1zWUcOn7S36EZE5AsWZha7Y6ftWPlI0P5y5iuLNuWy53vpHKsoMjfYRlTri827OXOd1JI25Pn71AqxZKFqfWaNAzl5gFt+ft1PVi+PZerX17EB8t3sibrkL9DM+YnnvxsI/PX72X4cwu5/a3lnDhZXGH9nQeO8/uP1nLAz7eMW7IwAWN0z1jevr0fucdO8rtZaxn14iLeXLStwjZ7j+Rz5Qvf8fHqXXZnlfG5z9fvISPnGGN7x/HAsI4sSNvHxHdSyC8sP2G8t2wH05bu5JJnv2VR+v5qjPZM9gS3CThH8gvZeeA4z325hS837uXCxOYkt21Km2b12X04n9sGtSM0JIjjJ4v4ZM1uHvrv2tNtnx7bjRv62gzGpuoVlyj9nviSRvXrMXPSQJpFhDEzNYsHPlzNJZ1b8OrNfdi8N4/v0/dz5+D2pyfRvHnqUr5P3895LSLYdegE0ycOYPHWAxQWl3Btn3haRoWfU1zePsFtycIErMLiEl77divvL8sk+9CJ0+VR9etx+ETh6e3GDeoxsmtLvty4j5y8Au68qD3REWGM6NqSuCYN/BG6CTDFJcrd761g3vo9PD+uJ6N7/rg8z3tLd/DwR+u47PwYvtq0F1W4a0gHHhjWCQF6/PlzrurZmnsvTWTsKz+QdfDHn+XI8BAeuSKJ82Ii6N2myVnF5veJBI3xt3rBQUy+JJHJlySSdfA42/cfZ9ehE7y/fCeJLSIoKCrhf6t2MaJLS568pjuPFZfwyOx1vPZtBgDvLtnB3HsvpEGo/Tcx52bhlhzmrd8DwEUdo8/Yd1P/thQUlvDYJxuoFyx0iI7glQVbOVlUwvh+bcgrKKJnfGNiGoXzzh39ufjvCwB467a+PDhrLb+dtYbEFhHM+9Vggn24bLH1LEydll9YTEiQEOJMf66qvLtkBwu37OfLjXsZ2zuOv1/Xw89Rmtpu8rQVLNycw5e/vogWkWVfNpq9MpuIsBAuS4rh0TnreeuH7SS3bULKjoN8cd/g0xNopu7IZV32ESYMSmDFzoO88NUWHh3VhbbNGp5VbHYZyphz9I/P0/jX1+lc3q0lj4/uytrsw7zwdTqPXtWFbnFR/g7P1CJDnvmGpNaNePmmPl7VLyou4Y63U/h2cw4AGX+9nCAf9RrsMpQx5+j/Lk2kWJVXv81g7to9p8tvfn0ps+++gHbNK/5L7rGPNxAZHsJ9Qzv6OlRTg50sKiHz4InTSwt7IyQ4iBdv7MW/vtpCTKNwnyWKyrBbZ40pR73gIH4zvDMvjO/FqdVdfzO8E0ECt725jJw8133ve4/ksyTjALsOnWBd9mEA1mUf5o1F23j+qy288NUWuy23jlFVtu8/Rn5hMZkHj1Ncoh7/uCgtMrweD1+RxC8ubO+jKCvHLkMZ44WSEj39113qjoPcPHUpJwqLGdC+KUsyflzEMSRI+PPoLjwzP42CwhL6tWvKt5tzCA0JIrltE/7982QahlmHPlCVlCgPfLja9cfD4Xw6xkQwqENz3vphOx/9chC9zvKOJV+qEYsficgIEUkTkXQRebCM/YNFZIWIFInItaX2FYvIKuc1x5dxGuOJ+2WAPm2bMOXnfahfL5j0fUfPqBcdGcbDH63j0PFCPrhzAG/e2pdJF3XgZFEJP2w9wL3TV9ksuQGquET59Yer+e/KbMJDg7njZ+3IPXaSt37YTttmDejcspG/QzwnPutZiEgwsBkYCmThWk97vKpucKuTADQCHgDmqOpMt31HVTXC28+znoWpbkXFJQQHCbsP59MsIpR12UdIjIngj7PX0aJROL+//PzTdfMLi/lgeSZ/mrOecX3jGd6lJRcmNj99F5ap/VZlHmLMS4u4pncsz17XAxHhaEERKdtz6RnfmMYNQv0dYplqwgB3PyBdVTOcgKYDo4HTyUJVtzv7bG5pU+uc+kXfunF9wNXjAHhuXK+f1A2vF8yEQQmk7zvKO0t2MH15Jhd1jOaNW/v69N54U3027T4CwL2XJp5++joiLIQhnVr4M6wq48s/a2IB97Uvs5wyb4WLSIqILBGRMWVVEJGJTp2UnJycc4nVmGrxyJVJ3HPJeVzVozXfbs7hzndSPE4kV1pxifK/Vdk89N81LM04UOkY9uXl26WwKlZcony6djf16wUTH6BP/dfkPnBbp2t0I/CciHQoXUFVp6hqsqomR0dH//QIxtQwoSFB3D+sE/8a15M/XZXE15tcE8ktyThw+u4qd0cLiti898yprF/9div3Tl/F+8syuWn+cqhOAAARs0lEQVTqUr7etBdw3YHzyZpd7HKb2qS0rzbuZcBfv2LsKz/8ZLzFnL2PVmbz3Zb9dIyJqBG3ufqCL5NFNhDvth3nlHlFVbOdrxnAAuCnfXtjaikR4bYL2vHUNd35bst+xk1ZwqgXv2fu2t2nk4aqcte7qYx4biGfrd19uu3Hq3fRuWUkKx8ZSudWkdz17gpWZx5iy76jTJ62kkFPfc3OA8fP+Ly8/EIu/vsC7ng7hRKF9H1HGTdlCTsOHKvW7ztQnboE9UwAP+3vy2SxHEgUkXYiEgqMA7y6q0lEmohImPO+OXABbmMdxgSK6/vG8+DIzoQGB7H7cD6/fG8Fw59byKrMQ3y6djffbdlP04Zh3DN9JdOW7uSud1PZtCeP65LjadIwlP/c3p/mEWGMfmkRw/658PRxx776w+lnPgBe/DqdbftdieGjXw7io18OoqikhLGvLGaj84vOnL2ducfpGBNBR2dKjkDk0+csRORy4DkgGHhDVZ8QkceAFFWdIyJ9gY+AJkA+sEdVu4jIIOA1oARXQntOVV+v6LPsbihT223em8eybblMWZjBgaMF1A8NJqZROO/9oj83v76Uddk//lJf9cehp++u2bTnCJOnrSR931Eu7hTNw1ecz43/Xsqh44VMuqg9Sa0bcdd7K7ghOZ5fD+tEdGQYAOn78rh56jIKi0v44M6BnNfC65sPTSkjnltIXJP6TJ3Q19+hVJrNDWVMLbXvSD63vL6MtL15/PeXg+jdpgm5x07y6xmrqB8azNCkGK7uFVdmW1VFRMg9dpIJbyxjrdO7iAgLYenvL/3JA4EZOUe5/rXF1AsO4sNJA21K9rOQX1hM50fmcdsFCfzpqi7+DqfSLFkYU4vl5Rey48Bxusae/YSFqkrG/mP8bd4mLu/W6ow1FNxt2HWEcVMW07RhKDMmDSx3VlRTtgdnrWH68kxeuak3I7u18nc4lWbJwhjjtdQdB7nl9aUIcGFiNLuP5DNhYFtaNgpna85R+rdvRseYSI4WFAGunopxGfLMN7Rp1pD/3N7P36GclZrwUJ4xppbo07YJ0ycO4I3vtzF71S4A7s88dHp/VP16PD22G099tomjBcW8dVvfc+r1BIpjBUXsyD1e7mXBQGLJwhgDQPe4xjw3rhfXJccTJEL2oRPUC3Y9M/D4JxuY9O6K03VvmrqU58f15KKO0aefVq6LNu/NQxXObxW4d0GdYsnCGHOGC85r/pOy/u2asXHPEXrENeZYQRHjpizh1jeX0yO+MXdd1J5hSS0D9mG0imzc7Xpg8vxWtXuSQG/U5Ce4jTE1RMuocC7u1IKmDUOJb9qA+fcN5qlrupFzJJ9J767gpqlLyTp4nIyco/x17kYOnyj0d8jVYtOeI0SEhRDrzA8WyKxnYYyptIiwEMb1a8O1feKYmZrF459sYMRz3xHXpD6b9uTx9aZ9vHpzb85rEbiXZ1SV1VmH6dwysk70qqxnYYw5ayHBQYzr14Z5vxpM19hGbNqTx5BO0Rw4WsDlz3/PS9+kU1gcmJNKP/nZJlZnHqJzHRivAOtZGGOqQHzTBkz7xQAWbsmhf7tmHC0o4k9z1vHM/DTmrt3N367tTpfWgXP31ImTxby/dCfRkWHcOfgnc5wGJOtZGGOqRFCQMKRTC+qHBhMdGcbLN/Xh1Zt7s/dIAaNfXMS7S3YExFrkK3YeZMxLi8grKOKlG3sT37RuPPVuycIY4zMjurbiy/sHM7BDM/4wex3XvbqYb9L21eqk8eai7aTtzeP3l3emX7um/g6n2thlKGOMTzVuEMrbt/VjRkomz3+1hdveXE6zhqFc2b0Vl3drRd+EprVmgFhVWbbtAKN6tGZiHbn8dIolC2OMzwUFCeP6teGa3nHMWb2LBWn7eGfJDt5evIOGzmWrAe2b8eDIzpVeqzr32EkOHj9JsAgtGoWxP+8kbZpV/aWh3GMneeHrLew9UlCnehSn2NxQxhi/OHT8JAvScpi7djffpO2jsFiJa1Kf3wzvRMeYSDrGRHpcn3zh5hx++d6K03NWnRLbuD4Xd45mdM9Y+iac+y/2khJl3L+XsGxbLq2iwpn3q8FE1a93zsetCWwiQWNMrXHiZDFrsw/zyOx1pDnLyEZHhtEjLor/uySRHvGNf9JmXfZhrnzhewDaRzfkpv5t2Xskn+AgYVvOMRZs3kd+YQnj+sZz72WJtIo6+wfnlm/P5bpXF3NtnzjuuSTRJz0Xf7GJBI0xtUb90GD6tWvKp/f8jCUZuezIPcaybbksSt/P6JcW0TehCcO7tCQ4SOgYE8nuw/n8de5GROCJMd24oW/8T3ohxwqKeGZ+Gu8s2cHHq3fxq8s68vNBbQkLCa50fJ+u2U1oSBCPjupSZ2fc9fVKeSOA53GtlDdVVZ8qtX8wrpX0ugPjVHWm274JwB+czb+o6tsVfZb1LIwJPIdPFDJjeSbTlu08vSysu19dlsivLutY4TEyc4/zx/+t45u0HGIb1+fhK87n8kquO3HJ3xfQplkD3rqtdk5DXhG/X4YSkWBgMzAUyMK1Jvd4Vd3gVicBaAQ8AMw5lSxEpCmQAiQDCqQCfVT1YHmfZ8nCmMBVUqJs3HOE0OAgMg8eZ++RAo4VFDG+X5ufrP5XnkXp+3nys42syz5CTKMwBidGM2FQAvFNGhDVoPzxh31H8un31694aGRn7rwo8O6AqgmXofoB6aqa4QQ0HRgNnE4Wqrrd2Vd6PoDhwBeqmuvs/wIYAbzvw3iNMTVUUJCcfgI8Mebspte44LzmzP7lBcxIyWLR1v38b/UuPkzNAiAyPISGoSFc0b0VA9o3o0vrRrR2Jgecs9q1vkf/9s2q4DupvXyZLGKBTLftLKD/ObQte01IY4zxUkhwEDf2b8ON/duw53A+S7cdYE3WYfblFVBQWMx/Fm/n9e+3AdAjLoroyHAWpO1jSKdoesQFznQlZ6NWj9SIyERgIkCbNm38HI0xpjZpGRXO6J6xZ6xNfqygiE178liUvp/vt+wnM/c4w7rE8PTY7nV6kSfwbbLIBuLdtuOcMm/bDinVdkHpSqo6BZgCrjGLswnSGGNOaRgWQp+2TejTtgn3XJro73BqFF/ODbUcSBSRdiISCowD5njZdj4wTESaiEgTYJhTZowxxg98lixUtQiYjOuX/EZghqquF5HHRGQUgIj0FZEs4DrgNRFZ77TNBR7HlXCWA4+dGuw2xhhT/ewJbmOMqcO8vXXWpig3xhjjkSULY4wxHlmyMMYY45ElC2OMMR5ZsjDGGONRwNwNJSI5wI4KqkQBhz0cxlOd8vaXVe5NWXNgv4eYqpo356Eq25/rea/svpp43s/1nFf2GN7WPZvz7m15WfVq23mv7p/1ivb78me9rapGe6jjWlO2LryAKedap7z9ZZV7Uwak1MTzUJXtz/W8V3ZfTTzv53rOK3sMb+uezXn3trycf4dadd6r+2e9sue9un/W69JlqI+roE55+8sq97asup1rDJVtf67nvbL7auJ5r4rPr8wxvK17Nufd23J/n3OofT/rFe33+896wFyGqo1EJEW9eBjGVC077/5h5736VeU5r0s9i5poir8DqKPsvPuHnffqV2Xn3HoWxhhjPLKehTHGGI8sWRhjjPHIkoUxxhiPLFnUYCLSUERSRORKf8dSV4jI+SLyqojMFJG7/B1PXSEiY0Tk3yLygYgM83c8dYGItBeR10Vkpjf1LVn4gIi8ISL7RGRdqfIRIpImIuki8qAXh/odMMM3UQaeqjjvqrpRVScB1wMX+DLeQFFF5322qv4/YBJwgy/jDQRVdM4zVPUOrz/T7oaqeiIyGDgK/EdVuzplwcBmYCiQhWsFwPFAMPBkqUPcDvQAmgHhwH5V/aR6oq+9quK8q+o+ZyXHu4B3VHVadcVfW1XVeXfaPQu8p6orqin8WqmKz/lMVb3W02eGVF345hRVXSgiCaWK+wHpqpoBICLTgdGq+iTwk8tMIjIEaAgkASdEZK6qlvgy7tquKs67c5w5wBwR+RSwZOFBFf28C/AU8JklCs+q6me9MixZVJ9YINNtOwvoX15lVX0YQERuxdWzsERxdip13p0kfQ0QBsz1aWSBrVLnHfg/4DIgSkTOU9VXfRlcgKrsz3oz4Amgl4g85CSVclmyqOFU9S1/x1CXqOoCYIGfw6hzVPVfwL/8HUddoqoHcI0RecUGuKtPNhDvth3nlBnfsvPuH3beq59Pz7kli+qzHEgUkXYiEgqMA+b4Oaa6wM67f9h5r34+PeeWLHxARN4HFgOdRCRLRO5Q1SJgMjAf2AjMUNX1/owz0Nh59w8779XPH+fcbp01xhjjkfUsjDHGeGTJwhhjjEeWLIwxxnhkycIYY4xHliyMMcZ4ZMnCGGOMR5YsjN+IyNFq+IxRXk4HX5WfOUREBp1Fu14i8rrz/lYRebHqo6s8EUkoPRV2GXWiRWRedcVkqp8lC1PrOVMzl0lV56jqUz74zIrmVRsCVDpZAL+nls6PpKo5wG4RsTVAApQlC1MjiMhvRGS5iKwRkT+7lc8WkVQRWS8iE93Kj4rIsyKyGhgoIttF5M8iskJE1opIZ6fe6b/QReQtEfmXiPwgIhkicq1THiQiL4vIJhH5QkTmntpXKsYFIvKciKQA94rIVSKyVERWisiXIhLjTBs9CbhPRFaJyIXOX92znO9veVm/UEUkEuiuqqvL2JcgIl875+YrEWnjlHcQkSXO9/uXsnpq4lpt8VMRWS0i60TkBqe8r3MeVovIMhGJdD7nO+ccriirdyQiwSLyjNu/1Z1uu2cDN5X5D2xqP1W1l7388gKOOl+HAVMAwfUHzCfAYGdfU+drfWAd0MzZVuB6t2NtB/7Pef9LYKrz/lbgRef9W8CHzmck4Zr7H+BaXNORBwEtgYPAtWXEuwB42W27CT/OgvAL4Fnn/aPAA271pgE/c963ATaWceyLgVlu2+5xfwxMcN7fDsx23n8CjHfeTzp1Pksddyzwb7ftKCAUyAD6OmWNcM1A3QAId8oSgRTnfQKwznk/EfiD8z4MSAHaOduxwFp//1zZyzcvm6Lc1ATDnNdKZzsC1y+rhcA9InK1Ux7vlB8AioFZpY7zX+drKq41KcoyW11rg2wQkRin7GfAh075HhH5poJYP3B7Hwd8ICKtcP0C3lZOm8uAJBE5td1IRCJU1b0n0ArIKaf9QLfv5x3gb27lY5z304C/l9F2LfCsiDwNfKKq34lIN2C3qi4HUNUj4OqFAC+KSE9c57djGccbBnR363lF4fo32QbsA1qX8z2YWs6ShakJBHhSVV87o9C1ENFlwEBVPS4iC3AtMwuQr6rFpY5T4Hwtpvyf7QK391JOnYocc3v/AvAPVZ3jxPpoOW2CgAGqml/BcU/w4/dWZVR1s4j0Bi4H/iIiXwEflVP9PmAvriV9g4Cy4hVcPbj5ZewLx/V9mABkYxamJpgP3C4iEQAiEisiLXD91XrQSRSdgQE++vxFwFhn7CIG1wC1N6L4cb2ACW7leUCk2/bnuFaCA8D5y720jcB55XzOD7immwbXmMB3zvsluC4z4bb/DCLSGjiuqu8CzwC9gTSglYj0depEOgP2Ubh6HCXALbjWbi5tPnCXiNRz2nZ0eiTg6olUeNeUqb0sWRi/U9XPcV1GWSwia4GZuH7ZzgNCRGQjrvWZl/gohFm4lqDcALwLrAAOe9HuUeBDEUkF9ruVfwxcfWqAG7gHSHYGhDdQxupkqroJ15KikaX34Uo0t4nIGly/xO91yn8F3O+Un1dOzN2AZSKyCvgT8BdVPQncALzg3CDwBa5ewcvABKesM2f2ok6Zius8rXBup32NH3txFwOfltHGBACbotwY4NQYgrjWJV4GXKCqe6o5hvuAPFWd6mX9BsAJVVURGYdrsHu0T4OsOJ6FwGhVPeivGIzv2JiFMS6fiEhjXAPVj1d3onC8AlxXifp9cA1IC3AI151SfiEi0bjGbyxRBCjrWRhjjPHIxiyMMcZ4ZMnCGGOMR5YsjDHGeGTJwhhjjEeWLIwxxnhkycIYY4xH/x+gyWgrlsLhAAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "m.sched.plot(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We choose a learning rate in the region just before the loss started to increase. Infact in this case, `1e-3` value we used would work fine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trying different learning rates and SGDR parameters (explained below)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "729a22b69f9c46a5b947fd03bba0c265",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   exp_rmspe                    \n",
      "    0      0.028962   0.023813   0.163325  \n",
      "    1      0.020587   0.018138   0.146005                     \n",
      "    2      0.017494   0.018654   0.135527                     \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.01865]), 0.13552661511774422]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.fit(lr, 3, metrics=[exp_rmspe])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5b2e5184afae4185b381b9acc50de257",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=8), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   exp_rmspe                    \n",
      "    0      0.015849   0.014832   0.126409  \n",
      "    1      0.014067   0.012542   0.119138                     \n",
      "    2      0.010901   0.011309   0.111057                     \n",
      "    3      0.009359   0.010918   0.109776                      \n",
      "    4      0.013122   0.013338   0.119559                     \n",
      "    5      0.010449   0.011476   0.11436                      \n",
      "    6      0.008599   0.010789   0.108697                      \n",
      "    7      0.007803   0.010351   0.106819                      \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.01035]), 0.10681856252429324]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.fit(lr, 2, metrics=[exp_rmspe], cycle_len=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stochastic Gradient Descent with Restarts\n",
    "We've used a technique called stochastic gradient descent with restarts (SGDR), a variant of learning rate annealing, which gradually decreases the learning rate as training progresses. This is helpful because as we get closer to the optimal weights, we want to take smaller steps.\n",
    "\n",
    "However, we may find ourselves in a part of the weight space that isn't very resilient - that is, small changes to the weights may result in big changes to the loss. We want to encourage our model to find parts of the weight space that are both accurate and stable. Therefore, from time to time we increase the learning rate (this is the 'restarts' in 'SGDR'), which will force the model to jump to a different part of the weight space if the current area is \"spikey\". Here's a picture of how that might look if we reset the learning rates 3 times (in this paper they call it a \"cyclic LR schedule\"): <img src=\"./data/learning_images/SGDR.png\">\n",
    "(From the paper [Snapshot Ensembles](https://arxiv.org/pdf/1704.00109.pdf)).\n",
    "\n",
    "The number of epochs between resetting the learning rate is set by cycle_len, and the number of times this happens is refered to as the number of cycles, and is what we're actually passing as the 2nd parameter to fit(). So here's what our actual learning rates looked like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8FHX+x/HXZzcd0gghlARSKaF3pShFBCxgwRMsP+yKYD29Q89TT48762FBLCeW81QEG1ERQRDEQu89IaCEltA7IfD9/bGDl8slZIGdzO7m83w89pHZ2ZnZ92LMZ78z3/l+xRiDUkop5WsupwMopZQKTlpglFJK2UILjFJKKVtogVFKKWULLTBKKaVsoQVGKaWULbTAKKWUsoUWGKWUUrbQAqOUUsoWIU4HcFLt2rVNamqq0zGUUiqgLFy4cIcxJrGy7ap1gUlNTWXBggVOx1BKqYAiIr94s52eIlNKKWULLTBKKaVsoQVGKaWULbTAKKWUsoUWGKWUUrawtcCISD8RWSsieSIyspzXw0XkI+v1uSKSWuq1h6z1a0Wkb6n1b4lIoYisKHOsWiIyTURyrZ/xdn42pZRSp2ZbgRERN/AK0B/IBoaISHaZzW4GdhtjMoHRwNPWvtnAYKA50A8Yax0P4B1rXVkjgenGmCxguvVcKaWUQ+y8D6YTkGeMyQcQkfHAQGBVqW0GAo9byx8DY0RErPXjjTFHgQ0ikmcd72djzPelWzpljtXDWn4XmAn80XcfR6mqsXnPYT5ZWEBkqJuaESEkx0eSVrsG9WMjcbnE6XhKec3OAtMA2FTqeQHQuaJtjDElIrIXSLDWzymzb4NK3i/JGLPVWt4GJJW3kYjcBtwG0LBhw8o/xRnI3b6fOtERxEaF2nJ8FdwmLtjEC9/m/s/6+KhQOqbWoktGAv1b1iMpJsKBdEp5Lyjv5DfGGBExFbz2BvAGQIcOHcrd5mzsO3KMPqO/B+DiVvUY3DGFbpm18TTMlKrciROeX8vlj1/IviMl/LrzEPk7DrDk1z3M3bCLqau285cvV9E5rRbXdG5E/xZ1CXVrfx3lf+wsMJuBlFLPk6115W1TICIhQCyw08t9y9ouIvWMMVtFpB5QeDbhz9SRY8cBCHULP+Xt4KtlW2leP4ZhPTK4qEU9PcWhvCIC0RGhREeE0iAuknMzEri2cyMA1hcd4MulW/lscQF3f7iYerER3NwtjevOaUREqLuSIytVdez82jMfyBKRNBEJw3PRPqfMNjnAUGt5EDDDGGOs9YOtXmZpQBYwr5L3K32socAkH3yGM/bYpc2Z83BvnrmyFYeLjzPig8UMfOVH5m/c5WQsFQQyEmtyzwVZzPh9D8YN7UCjhCj++tVqej8/i08XFfzWAlLKabYVGGNMCTAC+AZYDUwwxqwUkSdEZIC12TggwbqIfz9Wzy9jzEpgAp4OAVOA4caY4wAi8iHwM9BERApE5GbrWE8BfUQkF7jAeu6o8BA3v+uYwrT7z2f01a0p2n+Uq177mXvGL2b3wWKn46kA53IJvZslMf62c3n/ls7E1wjl/glLGfzPOeQXHXA6nlL2XoMxxkwGJpdZ92ip5SPAVRXsOwoYVc76IRVsvxPofTZ57eJ2CZe3TaZv87q8NiufV2fm8WPeTkZd3oK+zes6HU8Fga6ZtckZ3o2JCzcx6qvV9HtxNvf3acxt3dP1tKxyjF4ZrEJRYSHc36cxOSO6USc6nNvfW8jDny3naMlxp6OpIOByCVd3bMi3959PryZ1eOrrNQx9ex47Dhx1OpqqprTAOKBZvRgmjejK7een88HcX/ndaz+zec9hp2OpIFEnJoJXr2vH3y5vydwNu7joxdks0Gt/ygFaYBwS6nbxUP9mvH59e/KLDnLpyz+w6NfdTsdSQUJEuKZzQyYN70pUmJtr/jmXzxdX1hFTKd/SAuOwvs3rMmlEV2IiQhjyxhymrNjmdCQVRJrVi+Hz4V1p2zCOez9awuhp6/B01FTKflpg/EB6Yk0+GdaF7PoxDHt/If/6eaPTkVQQiYsK472bOzOofTIvTs/lkc9XaFdmVSW0wPiJhJrhfHjrOVzQLIlHJ63kzdn5TkdSQSQsxMWzg1oxrEcG78/9lQcmLqXk+AmnY6kgF5RDxQSqiFA3Y69tx73jl/DXr1Zz7LhhWI8Mp2OpICEi/KFvE6JC3Tw/bR1HSo7z4uC2OsyMso0WGD8T6nbx4uA2uF3C01PWIAJ3nK9FRvmGiHBX7ywiw9z89avVhIcs4/mrWuu9MsoWWmD8UIjbxeir22CAp75eQ1xkKIM72TPys6qebumezpFjx3lu6jpqhLt5cmALHZBV+ZwWGD/ldgnPX9WavYeP8fBny4mLCqVfi3pOx1JBZHjPTPYfLeH1WflER4Tyx35NnY6kgoyefPVjYSEuXruuHa1T4rj7wyX8vH6n05FUEBERRvZryjWdG/LqzPW8N+cXpyOpIKMFxs9FhYXw9g0daZgQxR3/XsiGHQedjqSCiIjw5MAW9Gpah8dzVjJzrSOzXKggpQUmAMRFhfHW0I64XcLN78xn76FjTkdSNqrqO1TcLuHlIW1pkhTNiA8Ws2bbvipOoIKVFhhfs+mvQ8OEKF67rj2bdh9i+AeLOKb3MAS1qr7cXiM8hHE3dKBGuJub3p6vA2Qqn9ACYxM7OuR0SqvF3y5vyQ95Oxj11Wrfv4Gq1urFRjJuaEd2Hizmrg8W642Y6qxpgQkwV3VI4ZZuabzz00Zylm5xOo4KMi0axDLq8pb8nL+TZ79Z63QcFeC0wASgP/ZvSsfUeEZ+sozc7fudjqOCzKD2yVx3TkNe/z6fycu3Oh1HBTAtMAEo1O1izDXtiApzM+z9RRw8WuJ0JBVkHr2kOW0bxvHgxKXkFer0y+rMaIEJUEkxEbw0pC35RQd46NPlOgS78qmwEBevXtue8FA3d324mCPHdNZVdfq0wASwLhm1PVMwL93CJ4t0MinlW3VjI3h2UCtWb93H01PWOB1HBSAtMAFuWI9MOqXV4rFJK/hlp96EqXyrd7MkbuiSyts/bmTGmu1Ox1EBRgtMgHO7hNFXt8HlEu4Zv0Tvj1E+N7J/U5rWjeaBicso3HfE6TgqgGiBCQIN4iL52+UtWbJpDy/PyHM6jgoyEaFuxlzTlkPFJfx+4lK93qe8pgUmSFzauj5XtGvAmBm5LPxll9NxVJDJrBPNny5qxuzcHYyfv8npOCpAaIEJIn8Z0Jz6cZE8OHGZ9vpRPndt50Z0yUjgr1+uYtOuQ07HUQFAC0wQiY4I5ekrW5G/4yD/mLbO6TgqyLhcwjODWgHwx0+WceKEnipTp6YFJsh0zazNkE4NeXN2Pot/3e10HBVkkuOjeOSSbH5av5P35+r8MerUtMAEoYcvakrdmAge/FhPlSnfG9wxhfMaJ/K3yWv4daeeKlMV0wIThKIjQvn7la3IKzzAS9NznY6jgoyI8PSVLXG7hD99rqNIqIppgQlS5zdO5Kr2ybz+fT4rNu91Oo4KMvViI/lDvybMzt3BpCU6qrcqn60FRkT6ichaEckTkZHlvB4uIh9Zr88VkdRSrz1krV8rIn0rO6aI9BaRRSKyRER+EJFMOz9bRfzpu9wjF2cTHxXKnz5bznG9IBswAqVBcG3nRrRJiePJL1ex51Cx03GUH7KtwIiIG3gF6A9kA0NEJLvMZjcDu40xmcBo4Glr32xgMNAc6AeMFRF3Jcd8FbjWGNMG+AB4xK7P5g2p8jkJ/1dsVCiPXJzN0oK9fDDvV6fjqNMgdsxY52Nul/D3K1qy9/Ax/jZZJ8BT/8vOFkwnIM8Yk2+MKQbGAwPLbDMQeNda/hjoLZ7/swYC440xR40xG4A863inOqYBYqzlWEDb7cDANvXpmpnAM1PWULhfh/lQvtWsXgy3dE9nwoICfl6/0+k4ys/YWWAaAKVv+S2w1pW7jTGmBNgLJJxi31Md8xZgsogUANcDT/nkUwQ4EeHJgS04euwEf/1Sv2Uq37undxYptSL502fLtdei+i/BdJH/PuAiY0wy8Dbwj/I2EpHbRGSBiCwoKiqq0oBOSU+sybAeGeQs3cLs3OrxmVXViQxzM+qyluTvOMibs/OdjqP8iJ0FZjOQUup5srWu3G1EJATPqa2dp9i33PUikgi0NsbMtdZ/BHQpL5Qx5g1jTAdjTIfExMQz+VwBaViPDNJq1+DPn6/Qb5nK585rnEi/5nUZ810em/ccdjqO8hN2Fpj5QJaIpIlIGJ6L9jlltskBhlrLg4AZxtOpPgcYbPUySwOygHmnOOZuIFZEGlvH6gPo+aBSIkLdPDmwBRt3HmLcDxucjqOC0COXNMMY+NtX+r+e8rCtwFjXVEYA3+D5Yz/BGLNSRJ4QkQHWZuOABBHJA+4HRlr7rgQmAKuAKcBwY8zxio5prb8V+EREluK5BvOgXZ8tUHXLqk3f5km88l0e2/bqBX/lW8nxUdzZI5Ovlm/lp7wdTsdRfsDWazDGmMnGmMbGmAxjzChr3aPGmBxr+Ygx5ipjTKYxppMxJr/UvqOs/ZoYY74+1TGt9Z8ZY1oaY1obY3qUPpb6j0cuzqbkhOGpr/VbpvK9289PJ6VWJI/lrNTJ71RQXeRXXkipFcVt3dP5fMkWnTdG+VxEqJs/X5xNbuEB/vWzDoZZ3WmBqYbu7JlB3ZgIHs9ZpUOuK5/rk53E+Y0TeWHaOor2H3U6jnKQFphqKCoshIcuasryzXuZuFBnJ1S+JSI8dmk2R0qO8/zUtU7HUQ7SAlNNDWhdnw6N4nn2m7XsO3LM6TgqyKQn1uT6c1KZsGATa7btczqOcogWmGpKRHh8QHN2HixmzIw8p+OoIHR370yiI0IZ9dVqHdK/mtICU421aBDLle2SeefHjTrHuvK5uKgw7u6dxezcHcxcpyNIVEdaYKq531/YGJcLnv1Gz5Ur37v+nEakJkTxt69WU6LdlqsdLTDVXL3YSG7ulkbO0i0sK9jjdBwVZMJCXIzs35TcwgN8tEA7lFQ3WmAUd5yfQUKNMD1XrmzRt3ldOqXWYvS0dezXDiXVihYYHwvEv8/REaHce0EWczfsYvrqQqfjVHvGr+ZFPXsiwiOXNGPHgWJenbne6TiqCmmBsUkATEj4XwZ3akh67Rr8/Ws9V+4PAuzXp1KtkuO4rE19xv2wgS062nK1oQVGARDqdvHH/k1ZX3RQz5UrWzzQtwnGwIvf5jodRVURLTDqNxdmJ9ExNZ7R09Zx4GiJ03FUkEmOj+K6cxoxceEm8goPOB1HVQEtMOo3IsLDF3nOlY+brXPGKN8b3jODyFA3z2m3+GpBC4z6L20bxtO3eRL/nJ3P7oPFTsdRQSahZji3npfOlJXbWLJJu8UHOy0w6n88cGETDhaX8Oos7fGjfO+W7ukk1Ajj6a/XaLf4IKcFRv2PrKRoLm/bgHd/2qgzXyqfqxkewohemfycv5PZuTrzZTDTAqPKdd8FjTlhDC/N0B4/yveu6dyQ5PhInp6yRuckCmJaYFS5UmpFcU2nhkyYv4mNOw46HUcFmfAQN/f3aczKLfv4avlWp+Mom2iBURUa3iuTULeLf0xb53QUFYQGtmlAk6Ronp+6lmN6c29Q0gKjKlQnOoIbu6aSs3QLq7bopFHKt9wu4Q/9mrBx5yEmLihwOo6ygRYYdUq3n5dBTESITn2rbNGraR3apMQxZkYuR0uOOx1H+ZgWGHVKsVGh3NEjg+lrClmwcZfTcVSQERHu79OYLXuP8NF8HaIo2HhdYEQkys4gyn/d0CWVxOhwntNWjLJB96zadEyNZ8yMPI4c01ZMMKm0wIhIFxFZBayxnrcWkbG2J1N+IyoshDt7ZDAnfxc/rdf7FpRveVoxTSjcf5R/z/nF6TjKh7xpwYwG+gI7AYwxS4Hz7Ayl/M+QTg1JignnhWm5eve18rlzMxLokpHAa7PWc6hYB1oNFl6dIjPGlD05qu3YCgTbZFEnRYS6Gd4zk3kbd/HT+p1Ox1FB6P4+jdlxoJh//aytmGDhTYHZJCJdACMioSLyALDa5lzKD13dMYV6sRH8Y9o6bcXYqLr+03ZIrcV5jRN5fdZ6nS4iSHhTYO4AhgMNgM1AG+BOO0MFg2CbkRA8d18P75nJwl926xhSNgu0GVF95f4+jdl96Bhv/6DTRQQDbwpME2PMtcaYJGNMHWPMdUAzu4Mp//S7Dik0iIvUVoyyRZuUOC5oVod/zs5n7+FjTsdRZ8mbAvOyl+tUNRAW4mJ4z0yWbNrDzLVFTsdRQejeCxqz70gJ42bnOx1FnaUKC4yInCsivwcSReT+Uo/HAbc3BxeRfiKyVkTyRGRkOa+Hi8hH1utzRSS11GsPWevXikjfyo4pHqNEZJ2IrBaRu736F1CnbVD7ZJLjIxn9rbZilO+1aBBLv+Z1eevHjTrpXYA7VQsmDKgJhADRpR77gEGVHVhE3MArQH8gGxgiItllNrsZ2G2MycTTHfppa99sYDDQHOgHjBURdyXHvAFIAZoaY5oB4yvLqM5MWIiLu3plsqxgL9NXFzodRwWh+/o05mBxCf/UVkxAC6noBWPMLGCWiLxjjDmTfoOdgDxjTD6AiIwHBgKrSm0zEHjcWv4YGCMiYq0fb4w5CmwQkTzreJzimMOAa4wxJ6z8+pfPRle0S+aV79Yz+tt19G5WB6muV6WVLZrUjeaiFvV496eN3No9nfgaYU5HUmfAm2swh0TkWRGZLCIzTj682K8BUPr+mQJrXbnbGGNKgL1Awin2PdUxM4CrRWSBiHwtIlnlhRKR26xtFhQV6TWEMxXq9rRiVm7Zx9RV252Oo4LQXb0zOVh8nLd+1B5lgcqbAvM+nmFi0oC/ABuB+TZmOlPhwBFjTAfgn8Bb5W1kjHnDGNPBGNMhMTGxSgMGm8vbNiA1IYoXvs3VWQmVzzWtG0P/FnV558eN7D2kPcoCkTcFJsEYMw44ZoyZZYy5CejlxX6b8VwTOSnZWlfuNiISAsTiGZKmon1PdcwC4FNr+TOglRcZ1VkIcbu4u3cWq7fuY+qqbU7HUUHorl5Z7D9awjhtxQQkbwrMya8OW0XkYhFpC9TyYr/5QJaIpIlIGJ6L9jlltskBhlrLg4AZxtMtKQcYbPUySwOygHmVHPNzoKe1fD6g0zBWgQGt65NWuwYvTc/THmXK57Lrx3BhdhJv/7hB74sJQN4UmL+KSCzwe+AB4E3gvsp2sq6pjAC+wTO0zARjzEoReUJEBlibjQMSrIv49wMjrX1XAhPwXLyfAgw3xhyv6JjWsZ4CrhSR5cDfgVu8+GzqLIW4PffFrNq6T3uUKVvc3TuL/UdKeOfHjU5HUaepwl5k8FtX4yxjzJd4LsD3PNX2ZRljJgOTy6x7tNTyEeCqCvYdBYzy5pjW+j3AxaeTT/nGwDb1eWl6Li/NyNUeZcrnWjSI5YJmSYz7IZ8bu6USExHqdCTlpVO2YIwxx4EhVZRFBahQt4vhPTNYVrCXmeu0Z57yvXt6Z7HvSAnvaismoHhziuxHERkjIt1FpN3Jh+3JVEC5vG0yDeIiefFbnS9G+V7L5Fh6Na3Dmz9s0JGWA4g3BaYNnjvqnwCetx7P2RlKBZ6wEBd39sxgyaY9OtKyssU9vbPYe/gY7/600ekoykuVFhhjTM9yHt50U1bVzKD2ydSLjeDF6dqKUb7XOiWOHk0SeXN2Pge1FRMQvJrRUnmvOv9dDQ9xM6xHBgt/2c3POuulssHdvbPYfegY783RWS8DgRYYm1TXjlS/65BCnehwXpye63SUgFWNv6NUql3DeLpn1eaf3+dzqFhbMf5OC4zyqYhQN3ecn8HcDbuYk6+tmDMlQTknqm/ce0EWOw8W829txfi9SguMiFxRzqO3iNSpioAq8Azp1JDaNcN5eYa2YpTvtW9Ui26ZtXnj+3wOFx93Oo46BW9aMDfjuXv/WuvxT+CPeLovX29jNhWgIsPc3H5eOj/m7WTBxl1Ox1FB6J4LsthxoJj352orxp95U2BCgGbGmCuNMVfimejLAJ3xFBql/se15zSkVo0wXpqR53QUFYQ6ptbi3PQEXv8+nyPHtBXjr7wpMCnGmNITfhRa63bxn4EwlfovUWEh3No9ne/XFbH4191Ox1FB6N4Lsijaf5QP5v7qdBRVAW8KzEwR+VJEhorIUGCSta4GsMfeeCqQXX9uI+KiQnlZWzHKBp3TE+icVovXZq3XVoyf8qbADAfewXNHfxvgX3hGNz5ojDmtwS9V9VIzPIRbuqUxY00hywv2Oh1HBaF7LsiicP9RPpq/qfKNVZXz5k5+Y4z52Bhzn/X42Oht2spLQ7ukEhMRwkvao0zZ4Nz0BDqmxvPqzPUcLdFWjL/xtptyrojsFZF9IrJfRPZVRTgV+KIjQrm5WzrTVm1n5RZtxSjfEhHu6d2YbfuOMGFBgdNxVBnenCJ7BhhgjIk1xsQYY6KNMTF2B1PB44auqUSHhzBGr8UoG3TNTKB9o3he/S5PWzF+xpsCs90Ys9r2JCpoxUaGcmPXVL5esY212/Y7HUcFGRHh7t5ZbNl7hI8XaivGn3hTYBaIyEciMqT03fy2J1NB5aZuadQIc+u1GGWL87Jq0yYljrHfrae45ITTcZTFmwITAxwCLgQutR6X2BlKBZ+4qDCGdkll8vKt5G7XVozyLRHhnguy2LznMJ8u0laMv/CmF9mN5TxuqopwKrjc0j2dyFC33hejbNGjcSKtkmMZ810ex45rK8YfVFhgROQP1s+XReSlso+qi6iCRa0aYfzfual8sWwLeYUHnI6jgoynR1kWBbsP89nizU7HUZy6BXPywv4CYGE5D6VO263d04gIcetIy8oWvZrWoUWDGF75Lo8SbcU4rsICY4z5wvr5bnmPqosYWPQO1FNLqBnO/53biC+WbmF9kbZilG+JCHf3yuKXnYf4fMkWp+NUe97caNlYRN4QkakiMuPkoyrCBTKdMKpit56XTniIW++LqYCOk3F2+mQnkV1PWzH+wJteZBOBxcAjwIOlHkqdkdo1w7nunIZMWrKZfG3FlE+/n5yxk/fFbNhxkC+WaSvGSd4UmBJjzKvGmHnGmIUnH7YnU0HttvMyCAtxMeY7bcUo37swO4mmdaN5eUYex09ok9Ap3hSYL0TkThGpJyK1Tj5sT6aCWmJ0ONd2bsSkJVvYuOOg03FUkHG5PK2Y/KKDfKmtGMd4U2CG4jkl9hP/6UG2wM5Qqnq4/fx0QlyirRhli37N69I4qaa2Yhx0ygIjIi7gOmNMWplHehXlU0GsTnQE13RuyGeLN/PLTm3FKN9yuYS7emWRV3iAycu3Oh2nWjplgTHGnADGVFEWVQ3dcX4GbpfwirZilA0ualmPzDo1eXlGLie0FVPlvDlFNl1ErhSR0+7XIiL9RGStiOSJyMhyXg+3BtLME5G5IpJa6rWHrPVrRaTvaRzzJRHRrkkBIikmgms6NeTTRZvZtOuQ03FUkHG7hLt6ZbJu+wGmrNzmdJxqx5sCczuerspHT2fCMRFxA68A/YFsYIiIZJfZ7GZgtzEmExgNPG3tmw0MBpoD/YCxIuKu7Jgi0gGI9+IzKT8yrEcGLm3FKJtc0qo+6Yk1eGm6tmKqmjeDXUYbY1zGmLDTnHCsE5BnjMk3xhQD44GBZbYZCJwcFeBjoLfVUhoIjDfGHDXGbADyrONVeEyr+DwL/MGLbMqPJMVEMKRjCh8vLNBWjPK5k62YNdv2M3WVtmKqkjctGEQkXkQ6ich5Jx9e7NYA2FTqeYG1rtxtjDElwF4g4RT7nuqYI4AcY4xezQtAd/TIwCXC2JnailG+d2mr+qTVrsGL0/MwOlRClfFmqJhbgO+Bb4C/WD8ftzfW6RGR+sBVwMtebHubiCwQkQVFRUX2h1NeqRcbydUdU5i4oICC3dqKUb4V4nYxvGcmq7fuY9qq7U7HqTa8acHcA3QEfjHG9ATaAnu82G8zkFLqebK1rtxtRCQEiAV2nmLfita3BTKBPBHZCESJSLlfhY0xbxhjOhhjOiQmJnrxMVRVGdYjAxEYO3O901FUELqsTX0aJUTx4vRcbcVUEW8KzBFjzBHw9PoyxqwBmnix33wgS0TSRCQMz0X7nDLb5OC5kRNgEDDDeP7L5wCDrV5maUAWMK+iYxpjvjLG1DXGpBpjUoFDVscBFUDqx0Xyuw4pTFywic17DjsdRwWZk62YlVv2MWNNodNxqgVvCkyBiMQBnwPTRGQS8EtlO1nXVEbgOaW2GphgjFkpIk+IyABrs3FAgtXauB8Yae27EpgArAKmAMONMccrOqb3H1f5uzt7er4XvKrXYpQNLm/bgJRakdqKqSIhlW1gjLncWnxcRL7DcxprijcHN8ZMBiaXWfdoqeUjeK6dlLfvKGCUN8csZ5ua3uRT/qdBXCSD2qcwYX4Bd/bIpH5cpNORVBAJdbsY3iOTkZ8uZ+baIno2reN0pKDmbS+ybiJyozFmFvAz/9sbTFn0W9HZG94zA4PhZZ0vRtnginbJJMdH8o9p6/T/V5t504vsMeCPwEPWqlDg33aGUtVbcnwUQzo1ZOKCTTpGmfK5sBAXd/fOYvnmvUzVHmW28qYFczkwADgIYIzZAkTbGSoo6IRRZ2VEz0zcLuHF6blOR6lyRifett0VbRuQXrsG/5i6TkdatpE3BabY6tllAESkhr2RlII6MREM7ZLK54s3k1e43+k4VU6/n9grxO3i3j6NWbt9v84XYyNvCswEEXkdiBORW4FvgX/aG0spuP28dCJD3Yz+tvq1YpT9LmlZj6Z1o3nh21xKjp9wOk5Q8mYssufwjBP2CZ77Xx41xlR6x7xSZyuhZjg3dUvjq2VbWbllr9NxVJBxuYT7+zRmw46DfLqo7D3gyhe86kVmjJlmjHnQGPOAMWaa3aGUOumW7unERIQweto6p6OoINQnO4nWybG8OD2XoyXHnY4TdCosMCeH5S/n4dVw/Ur5QmxkKLefn8G3qwtZ/Otup+OoICMi/P7CJmzec5gJ8zdVvoM6LRUWmJPD8pfz8Ha4fqV84oYuqSTUCOMf2opRNuieVZtOqbV4eUYeh4u1FeNLXp0iU8pJNcJDGNYjg9m5O5iTv9MsUcZ+AAAWfUlEQVTpOCrIeFoxjSncf5R/z6l0FCx1GrTAqIBw3TmNSIoJ5/mpa/Xua+VzndMT6J5Vm1dnrefA0RKn4wQNLTAqIESEuhnRK4v5G3cza53O46N87/cXNmHXwWLe/mGD01GChhYYFTCu7pBCSq1Inv1mrc6trnyuTUocfbKTeOP7fHYdLHY6TlDQAqMCRliIiwcubMLKLfvIWap3Xyvf+0PfJhwsLmGMDrTqE1pgVEC5tFV9mteP4bmpa/W+BeVzWUnRXNU+hffmbGTTLp26+2xpgVEBxeUSRvZvSsHuw/x7zq9Ox1FB6L4+jXG7hOenrnU6SsDTAqMCTvesRLpn1WbMjFz2HTnmdBwVZOrGRnBT1zQ+X7KFFZt1iKKzoQVGBaQ/9mvK7kPHeH3WeqejqCB0R48M4qJCeXrKGqejBDQtMD6mt2hUjRYNYhnYpj7jftjAtr1HnI6jgkxMRCgjemYyO3cHs3O1W/yZ0gJjE53Pw34PXNiE4ycML3yrQ8go37v+3EYkx0fy1NdrtFv8GdICowJWSq0orjunERMWbAquScn0b5lfCA9x/9Yt/gudlOyMaIFRAW1Ez0yiwkJ46uvg6vEj2gT2CwNae7rFP/uNdos/E1pgVEBLqBnOnT0z+Hb1dn7M2+F0HBVkXC7h4YuaUbD7MG//uNHpOAFHC4wKeDd1TSOlViRPfLFKp75VPtc1szYXNEtizIw8ivYfdTpOQNECowJeRKibh/s3Y+32/YzXSaOUDf50cTOOlhznuW+C61Ss3bTAqKDQr0VdOqXV4h/T1rH3sN58qXwrrXYNbuiSyoSFm/Tmy9OgBUYFBRHh0Uuy2X2omDEzcp2Oo4LQiF5ZxEeF8cSXq3ROIi9pgVFBo0WDWH7XPoV3ftrIhh0HnY6jgkxsZCi/v7Ax8zbsYsqKbU7HCQhaYFRQeaBvE8JD3Iz6arXTUVQQurpDCk3rRjNq8mqOHNNuy5XRAqOCSmJ0OMN7ZvLt6u06xIfyuRC3i0cvyaZg92HG6cyXldICo4LOTd1SSU2I4rGclXpznPK5Lpm16dvc0225YLfOGXMqthYYEeknImtFJE9ERpbzeriIfGS9PldEUku99pC1fq2I9K3smCLyvrV+hYi8JSKhdn425b/CQ9w8PqA5+UUHeXO2fstUvvfnS7IBeOKLVQ4n8W+2FRgRcQOvAP2BbGCIiGSX2exmYLcxJhMYDTxt7ZsNDAaaA/2AsSLiruSY7wNNgZZAJHCLXZ9N+b8eTepwUcu6vDQ9V2cmVD6XHB/FXb0zmbpqOzPWbHc6jt+yswXTCcgzxuQbY4qB8cDAMtsMBN61lj8GeouIWOvHG2OOGmM2AHnW8So8pjFmsrEA84BkGz+bCgB/viQbt0v4yxcrnY6igtAt3dLJSKzBYzkr9YJ/BewsMA2A0rdVF1jryt3GGFMC7AUSTrFvpce0To1dD0wpL5SI3CYiC0RkQVGRXgQOZvViI7n3giy+XV3ItFX6LVP5VliIiycva8GmXYcZ+12e03H8UjBe5B8LfG+MmV3ei8aYN4wxHYwxHRITE6s4mqpqN3ZNo3FSTR7PWcnhYv2WqXyrS0ZtLmtTn9dm5eu9V+Wws8BsBlJKPU+21pW7jYiEALHAzlPse8pjishjQCJwv08+gQp4oW4XTw5sweY9h3lZ7/BXNnj44maEh7h4dNIKvcO/DDsLzHwgS0TSRCQMz0X7nDLb5ABDreVBwAzrGkoOMNjqZZYGZOG5rlLhMUXkFqAvMMQY4/iQuqITeviNzukJDGqfzOvf57Nyi/+PI6V/ogJLnegIHuzXhNm5O/hkUdnv0NWbbQXGuqYyAvgGWA1MMMasFJEnRGSAtdk4IEFE8vC0OkZa+64EJgCr8FxLGW6MOV7RMa1jvQYkAT+LyBIRedSuz6YCzyMXNyM+Kow/fLwsIIb0F510O6Bc17kRHRrF8+SXqyjcf8TpOH7D1mswVs+uxsaYDGPMKGvdo8aYHGv5iDHmKmNMpjGmkzEmv9S+o6z9mhhjvj7VMa31Ida6NtbjCTs/mwoscVFhPDmwOSu37OON2fmV76DUaXC5hKcHteLwseM8Nkl7LZ4UjBf5lSpX/5b16N+iLi98m8v6ogNOx1FBJiOxJvdekMXXK7bx9fKtTsfxC1pgVLXyl4HNiQhxMfKTZZw4oVc7lG/d2j2d5vVj+POklew5VOx0HMdpgVHVSp3oCP58STbzN+7mXz9vdDqOCjKhbhfPDGrFnkPFPPGlDiOjBUZVO4PaJ9OjSSJ//3oNeYV6qkz5VvP6sdzZI4NPF21myorqfapMC4yqdkSEZ65sRVSYm/s+WkJxif/3KlOB5a7eWbRsEMtDny6ncF/17VWmBUZVS3ViIvj7FS1ZvnkvL03XGzCVb4W6XYy+ug2Hjx3nwY+XVdsbMLXAqGqrX4t6XNU+mbEz81iwcZfTcVSQyaxTkz9d1IxZ64p4b84vTsdxhBYYVa09NqA5DeIjuW/CEvYfOeZ0HBVkrjunET2aJDLqq9XkFe53Ok6V0wKjqrWa4SGM/l0bNu8+zMhPllfbUxnKHiLCM4NaUSM8hDvfX1TtBlzVAqOqvQ6ptXigbxO+Wr612p7KUPapEx3BC1e3IbfwAI98Xr0GxNQCoxRwx3kZ9Gpahye/XMXSTXucjqOCzHmNE7mrVxafLCpg4oICp+NUGS0wSuEZS+r5q1pTJzqC4R8sYu8hvR6jfOue3ll0zUzgz5NWsGrLPqfjVAktMEpZ4muEMeaatmzfd4R7P1rMcR1KRvmQ2yW8cHVbYiNDufP9hdXiS4wWGB+rRqdXg1LbhvE8dmlzvltbxDNT1jgdRwWZxOhwxl7bjs17DjP8g0UBMXXE2dACo1QZ153TiOvPacTr3+fz8cLqc75cVY0OqbUYdVlLfsjbwV+/Wu10HFuFOB0gWOl0UYHt0UuzWV90gIc/XU5a7SjaN6pVZe9dnXoZVVe/65jCuu37efOHDWQl1eTazo2cjmQLbcEoVY5Qt4ux17ajXlwEt/1rIRt2HKzS99cZt4PfQxc14/zGiTw2aSUz1xY6HccWWmCUqkBcVBhv3dARA1w/bm61HrRQ+Z7bJbx8TVuykqIZ9u9FLP51t9ORfE4LjFKnkJFYk7dv6Miug8X831vz2Hs4+Hv+qKoTExHKuzd1JDE6nJvemR90w8logVGqEq1T4nj9+vasLzrALe/O51BxidORVBCpEx3Bezd3wu1ycf24eRTsPuR0JJ/RAqOUF7pnJTL66jYs/GU3N7w9n4NHtcgo32mUUIN3b+rIwaMlXP36HDbtCo4iowVGKS9d0qo+LwxuaxWZeRzQIqN8qHn9WD649RwOHC1h8Btz+GVn1XYssYMWGKVOw4DW9XlpcFsW/bqH/xs3lz2Hip2OpIJIiwaxfHBrZw4Ve4pMoF+T0QKj1Gm6uFU9XrmmHSs27+OKV38KmtMZyj+cbMkcO264YuxPzNsQuJPhaYFR6gz0a1GXf9/SmZ0Hirl87E8sL9jrdCQVRJrVi+GzO7uQGB3OdW/O5YulW5yOdEa0wCh1hjql1eKTYecSHuJi0Gs/MXHBJqcjqSCSUiuKT4Z1oU1KHHd9uJinvl4TcGOXaYFR6ixk1olm0oiutG8Uz4MfL+OhT5dx5Fj1mrVQ2ScuKoz3bunEtZ0b8tqs9Vw3bi6F+wPnhl8tMEqdpdo1w3nv5s4M75nBh/M2cenLP+ikZcpnwkPcjLq8Jc9f1Zolm/bQ/4XZTF6+1elYXtECo5QPuF3Cg32b8u5NnThwtITLx/7I01PWVLs52JV9rmyfTM6IbtSPi+TO9xcx/P1Fft+a0QKjlA+d3ziRb+47j6vap/DqzPX0fG4mnyws4IROXqZ8oHFSNJ/d2YUH+zZh2qrt9Hx2JmNm5PrtaVlbC4yI9BORtSKSJyIjy3k9XEQ+sl6fKyKppV57yFq/VkT6VnZMEUmzjpFnHTPMzs+mVEViIkJ5elArJt5xLkkx4fx+4lIuemk2ny0u4FiAXaRV/ifE7WJ4z0ym3nce3bMSeW7qOs5/9jve+H49+4/411h5ts0HIyJu4BWgD1AAzBeRHGPMqlKb3QzsNsZkishg4GngahHJBgYDzYH6wLci0tjap6JjPg2MNsaMF5HXrGO/asdne2zSCuZvLH/k02L9A6IsHVNr8dmdXfli2RZe+S6P+z5ayjNT1nJlu2Qua1ufzDrRTkdUASy1dg1eu749c/N38tKMXP42eQ0vT89jQJv6XN62Ae0bxSMOz/tg54RjnYA8Y0w+gIiMBwYCpQvMQOBxa/ljYIx4/kUGAuONMUeBDSKSZx2P8o4pIquBXsA11jbvWse1pcDUqhFO/bjICl9vUjeazulVN0GV8l8ulzCwTQMGtK7PzLVFvPPTRsbOzGPMd3lkJNagS0ZtOqXVonFSNKm1owgPcTsdWQWYzukJvJ+ewIrNexn3wwY+XbSZ9+f+SlJMOF0zanNOegJN60WTkViTGuFVO8ekne/WACh9Y0AB0LmibYwxJSKyF0iw1s8ps28Da7m8YyYAe4wxJeVs73P3XJBl16FVkBIRejatQ8+mdSjcf4Svlm1l1roiPllUwHtzfgHAJRATGcqeQ8cIdeuMY+r0tGgQy+ir2/DkZSVMXbmNGWsKmbWuiE8Xb/5tm+jwEGIiQ4kMczNuaAcaJdSwNVO1mzJZRG4DbgNo2LChw2lUdVQnOoIbu6ZxY9c0jh0/wdpt+1lfdID1RQfZc6iYfYePkZWkp8/UmakZHsIV7ZK5ol0yJ04Y8nccJK9wP+uLDrLjwFH2HS7h8LGSKmkt21lgNgMppZ4nW+vK26ZAREKAWGBnJfuWt34nECciIVYrprz3AsAY8wbwBkCHDh20a49yVKjbRYsGsbRoEOt0FBWEXC4hs05NMuvUdOb9bTz2fCDL6t0VhueifU6ZbXKAodbyIGCGMcZY6wdbvczSgCxgXkXHtPb5zjoG1jEn2fjZlFJKVcK2Fox1TWUE8A3gBt4yxqwUkSeABcaYHGAc8J51EX8XnoKBtd0EPB0CSoDhxpjjAOUd03rLPwLjReSvwGLr2EoppRwini//1VOHDh3MggULnI6hlFIBRUQWGmM6VLad3smvlFLKFlpglFJK2UILjFJKKVtogVFKKWULLTBKKaVsUa17kYlIEfDLGe5eG9jhwzh2CqSsEFh5Nas9AikrBFZeX2RtZIxJrGyjal1gzoaILPCmm54/CKSsEFh5Nas9AikrBFbeqsyqp8iUUkrZQguMUkopW2iBOXNvOB3gNARSVgisvJrVHoGUFQIrb5Vl1WswSimlbKEtGKWUUrbQAnOaRKSfiKwVkTwRGelgjrdEpFBEVpRaV0tEpolIrvUz3lovIvKSlXmZiLQrtc9Qa/tcERla3nv5IGuKiHwnIqtEZKWI3OOveUUkQkTmichSK+tfrPVpIjLXyvSRNV0E1pQSH1nr54pIaqljPWStXysifX2dtdT7uEVksYh8GQBZN4rIchFZIiILrHV+93tgvUeciHwsImtEZLWInOuPWUWkifXvefKxT0Tu9Yusxhh9ePnAM0XAeiAdCAOWAtkOZTkPaAesKLXuGWCktTwSeNpavgj4GhDgHGCutb4WkG/9jLeW423IWg9oZy1HA+uAbH/Ma71nTWs5FJhrZZgADLbWvwYMs5bvBF6zlgcDH1nL2dbvRziQZv3euG36Xbgf+AD40nruz1k3ArXLrPO73wPrfd4FbrGWw4A4f81aKrMb2AY08oestnzIYH0A5wLflHr+EPCQg3lS+e8CsxaoZy3XA9Zay68DQ8puBwwBXi+1/r+2szH3JKCPv+cFooBFQGc8N6aFlP09wDM30bnWcoi1nZT93Si9nY8zJgPTgV7Al9Z7+2VW69gb+d8C43e/B3hm192AdZ3an7OWyXch8KO/ZNVTZKenAbCp1PMCa52/SDLGbLWWtwFJ1nJFuav881inZdriaRn4ZV7rlNMSoBCYhucb/R7jmY677Pv+lsl6fS+QUFVZgReAPwAnrOcJfpwVwABTRWShiNxmrfPH34M0oAh42zr9+KaI1PDTrKUNBj60lh3PqgUmSBnPVxC/6iIoIjWBT4B7jTH7Sr/mT3mNMceNMW3wtA46AU0djlQuEbkEKDTGLHQ6y2noZoxpB/QHhovIeaVf9KPfgxA8p6BfNca0BQ7iOc30Gz/KCoB1rW0AMLHsa05l1QJzejYDKaWeJ1vr/MV2EakHYP0stNZXlLvKPo+IhOIpLu8bYz7197wAxpg9wHd4TjPFicjJKcZLv+9vmazXY4GdVZS1KzBARDYC4/GcJnvRT7MCYIzZbP0sBD7DU8D98fegACgwxsy1nn+Mp+D4Y9aT+gOLjDHbreeOZ9UCc3rmA1lWL50wPM3RHIczlZYDnOz5MRTPtY6T6//P6j1yDrDXajp/A1woIvFWD5MLrXU+JSICjANWG2P+4c95RSRRROKs5Ug814pW4yk0gyrIevIzDAJmWN8Wc4DBVs+tNCALmOfLrMaYh4wxycaYVDy/izOMMdf6Y1YAEakhItEnl/H891uBH/4eGGO2AZtEpIm1qjewyh+zljKE/5weO5nJ2ax2XWwK1geeHhjr8JyX/5ODOT4EtgLH8HzbuhnP+fTpQC7wLVDL2laAV6zMy4EOpY5zE5BnPW60KWs3PM3zZcAS63GRP+YFWgGLrawrgEet9el4/ujm4TkFEW6tj7Ce51mvp5c61p+sz7AW6G/z70MP/tOLzC+zWrmWWo+VJ///8cffA+s92gALrN+Fz/H0rPLXrDXwtEZjS61zPKveya+UUsoWeopMKaWULbTAKKWUsoUWGKWUUrbQAqOUUsoWWmCUUkrZQguMUmdIRH6yfqaKyDU+PvbD5b2XUoFEuykrdZZEpAfwgDHmktPYJ8T8Z7yw8l4/YIyp6Yt8SjlFWzBKnSEROWAtPgV0t+biuM8aLPNZEZlvzbdxu7V9DxGZLSI5eO4KR0Q+twZ+XHly8EcReQqItI73fun3su6+flZEVohnXpWrSx17pvxn/pL3rREUEJGnxDMXzzIRea4q/41U9RZS+SZKqUqMpFQLxioUe40xHUUkHPhRRKZa27YDWhhjNljPbzLG7LKGpZkvIp8YY0aKyAjjGXCzrCvw3GHeGqht7fO99VpboDmwBfgR6Coiq4HLgabGGHNyGBylqoK2YJTyvQvxjPW0BM+0BAl4xvcCmFequADcLSJLgTl4BhrM4tS6AR8az4jP24FZQMdSxy4wxpzAMxxPKp4h+Y8A40TkCuDQWX86pbykBUYp3xPgLmNMG+uRZow52YI5+NtGnms3F+CZ3Ks1njHQIs7ifY+WWj6OZ9KxEjwjFn8MXAJMOYvjK3VatMAodfb245kK+qRvgGHWFAWISGNr9OCyYoHdxphDItIUz/S1Jx07uX8Zs4Grres8iXimzq5w5GPxzMETa4yZDNyH59SaUlVCr8EodfaWAcetU13v4JmTJRVYZF1oLwIuK2e/KcAd1nWStXhOk530BrBMRBYZzxD8J32GZ36apXhGqP6DMWabVaDKEw1MEpEIPC2r+8/sIyp1+rSbslJKKVvoKTKllFK20AKjlFLKFlpglFJK2UILjFJKKVtogVFKKWULLTBKKaVsoQVGKaWULbTAKKWUssX/AxVFnYximp5LAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "m.sched.plot_lr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train all:\n",
    "Since we now performed learning rate finder to determine the optimal parameters, we will now use all the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "samp_size = n\n",
    "joined_samp = joined.set_index(\"Date\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "844338"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(joined_samp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "df, y, nas, mapper = proc_df(joined_samp, 'Sales', do_scale=True)\n",
    "yl = np.log(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "633253\n"
     ]
    }
   ],
   "source": [
    "train_ratio = 0.75\n",
    "# train_ratio = 0.9\n",
    "train_size = int(samp_size * train_ratio); print(train_size)\n",
    "val_idx = np.flatnonzero(\n",
    "    (df.index<=datetime.datetime(2014,9,17)) & (df.index>=datetime.datetime(2014,8,1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "((val_df, trn_df), (val_y, trn_y)) = split_by_idx(val_idx, df, yl.astype(np.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "md = ColumnarModelData.from_data_frame(PATH, val_idx, df, yl.astype(np.float32), cat_flds=cat_vars, bs=128, \n",
    "                                  test_df=df_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars),\n",
    "                   0.04, 1, [1000,500], [0.001,0.01], y_range=y_range)\n",
    "lr = 1e-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "763f039b70f34e2e88ab850e18a12a39",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   exp_rmspe                      \n",
      "    0      0.014368   0.020363   0.131199  \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.02036]), 0.1311989278763488]"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.fit(lr, 1, metrics=[exp_rmspe])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "30a3d2a31c2e499fa386eab81b18a168",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   exp_rmspe                      \n",
      "    0      0.011016   0.013891   0.112283  \n",
      "    1      0.0093     0.01356    0.110362                        \n",
      "    2      0.008994   0.013076   0.108532                        \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.01308]), 0.10853249288017815]"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.fit(lr, 3, metrics=[exp_rmspe])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "daa12cd6040041aa874a0599473a2433",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   exp_rmspe                       \n",
      "    0      0.006908   0.011916   0.102187  \n",
      "    1      0.006924   0.011808   0.101475                        \n",
      "    2      0.006774   0.011602   0.101375                        \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.0116]), 0.10137538797671516]"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.fit(lr, 3, metrics=[exp_rmspe], cycle_len=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "m.save('val0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "m.load('val0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "x,y=m.predict_with_targs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1056749042159721"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exp_rmspe(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_test=m.predict(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_test = np.exp(pred_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "joined_test['Sales']=pred_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MixedInputModel(\n",
       "  (embs): ModuleList(\n",
       "    (0): Embedding(1116, 50)\n",
       "    (1): Embedding(8, 4)\n",
       "    (2): Embedding(4, 2)\n",
       "    (3): Embedding(13, 7)\n",
       "    (4): Embedding(32, 16)\n",
       "    (5): Embedding(3, 2)\n",
       "    (6): Embedding(26, 13)\n",
       "    (7): Embedding(27, 14)\n",
       "    (8): Embedding(5, 3)\n",
       "    (9): Embedding(4, 2)\n",
       "    (10): Embedding(4, 2)\n",
       "    (11): Embedding(24, 12)\n",
       "    (12): Embedding(9, 5)\n",
       "    (13): Embedding(13, 7)\n",
       "    (14): Embedding(53, 27)\n",
       "    (15): Embedding(22, 11)\n",
       "    (16): Embedding(7, 4)\n",
       "    (17): Embedding(7, 4)\n",
       "    (18): Embedding(4, 2)\n",
       "    (19): Embedding(4, 2)\n",
       "    (20): Embedding(9, 5)\n",
       "    (21): Embedding(9, 5)\n",
       "  )\n",
       "  (lins): ModuleList(\n",
       "    (0): Linear(in_features=217, out_features=1000, bias=True)\n",
       "    (1): Linear(in_features=1000, out_features=500, bias=True)\n",
       "  )\n",
       "  (bns): ModuleList(\n",
       "    (0): BatchNorm1d(1000, eps=1e-05, momentum=0.1, affine=True)\n",
       "    (1): BatchNorm1d(500, eps=1e-05, momentum=0.1, affine=True)\n",
       "  )\n",
       "  (outp): Linear(in_features=500, out_features=1, bias=True)\n",
       "  (emb_drop): Dropout(p=0.04)\n",
       "  (drops): ModuleList(\n",
       "    (0): Dropout(p=0.001)\n",
       "    (1): Dropout(p=0.01)\n",
       "  )\n",
       "  (bn): BatchNorm1d(18, eps=1e-05, momentum=0.1, affine=True)\n",
       ")"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison to Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "((val,trn), (y_val,y_trn)) = split_by_idx(val_idx, df.values, yl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = RandomForestRegressor(n_estimators=40, max_features=0.99, min_samples_leaf=2,\n",
    "                          n_jobs=-1, oob_score=True)\n",
    "m.fit(trn, y_trn);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.9821290849380468,\n",
       " 0.9319712460889528,\n",
       " 0.9241907128768685,\n",
       " 0.10859555196861052)"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = m.predict(val)\n",
    "m.score(trn, y_trn), m.score(val, y_val), m.oob_score_, exp_rmspe(preds, y_val)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}