04-sklearn集成学习--GBRT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.11.4 梯度提升树（GradientTree Boosting)\n",
    "GBRT是一个准确率高、有效的算法，可以解决回归和分类问题。该模型被广泛应用到各个领域，包括网络搜索排名。\n",
    "\n",
    "▲GBRT的优点包括：\n",
    "\n",
    "可以对混合数据类型进行自然处理（非均值特征）\n",
    "预测能力\n",
    "对输出空间中的异常数据具有鲁棒性（通过鲁棒损失函数）\n",
    "▲GBRT的缺点包括：\n",
    "\n",
    "由于其序列性，很难进行并行运算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.11.4.1分类（classification)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_hastie_10_2\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = make_hastie_10_2(random_state=0)\n",
    "X_train, X_test = X[:2000], X[2000:]\n",
    "y_train, y_test = y[:2000], y[2000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2000, 10) (10000, 10)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape, X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "              learning_rate=1.0, loss='deviance', max_depth=1,\n",
       "              max_features=None, max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "              n_iter_no_change=None, presort='auto', random_state=0,\n",
       "              subsample=1.0, tol=0.0001, validation_fraction=0.1,\n",
       "              verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 在数据量大的时候，建议使用RandomForestClassifier代替GradientBoostingClassifier\n",
    "clf = GradientBoostingClassifier(n_estimators=100, \n",
    "                                 learning_rate=1.0,\n",
    "                                 max_depth=1,\n",
    "                                 random_state=0)  # 100个弱学习器数量，最大深度控制树的大小，学习率控制过拟合．\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.913"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.11.4.2 回归（Regression)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.datasets import make_friedman1\n",
    "from sklearn.ensemble import GradientBoostingRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(200, 10) (1000, 10)\n"
     ]
    }
   ],
   "source": [
    "X, y = make_friedman1(n_samples=1200, random_state=0, noise=1.0)\n",
    "X_train, X_test = X[:200], X[200:]\n",
    "y_train, y_test = y[:200], y[200:]\n",
    "print(X_train.shape, X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse', init=None,\n",
       "             learning_rate=0.1, loss='ls', max_depth=1, max_features=None,\n",
       "             max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "             min_impurity_split=None, min_samples_leaf=1,\n",
       "             min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "             n_estimators=100, n_iter_no_change=None, presort='auto',\n",
       "             random_state=0, subsample=1.0, tol=0.0001,\n",
       "             validation_fraction=0.1, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "est = GradientBoostingRegressor(n_estimators=100, \n",
    "                                 learning_rate=0.1,\n",
    "                                 max_depth=1, \n",
    "                                 random_state=0,\n",
    "                                 loss='ls')  # 最小平方损失,默认\n",
    "est.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8058912110043196"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "est.score(X_test, y_test)  # 准确率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.009154859960321"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_squared_error(y_test, est.predict(X_test))  # 均方差"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来展示使用最小平方损失、500个基学习器的梯度提升回归对波士顿房价的拟合。每次迭代的训练样本的错误率保存在模型的train_score_中。测试样本的错误率可以通过staged_predict方法获得，该方法返回一个生成器，用于生成每个阶段的预测。像这样的曲线可以通过提前停止来确定树的最优数量，即参数：n_estimators。特征的重要程度，可通过fearture_importances_property获得。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn import ensemble\n",
    "from sklearn import datasets\n",
    "from sklearn.utils import shuffle\n",
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 加载数据\n",
    "boston = datasets.load_boston()\n",
    "X, y = shuffle(boston.data, boston.target, random_state=13)\n",
    "X = X.astype(np.float32)\n",
    "offset = int(X.shape[0]*0.9)\n",
    "\n",
    "X_train, y_train = X[:offset], y[:offset]\n",
    "X_test, y_test = X[offset:], y[offset:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(455, 13) (51, 13)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape, X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse', init=None,\n",
       "             learning_rate=0.01, loss='ls', max_depth=4, max_features=None,\n",
       "             max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "             min_impurity_split=None, min_samples_leaf=1,\n",
       "             min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "             n_estimators=500, n_iter_no_change=None, presort='auto',\n",
       "             random_state=None, subsample=1.0, tol=0.0001,\n",
       "             validation_fraction=0.1, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 拟合回归模型\n",
    "params = {'n_estimators': 500,\n",
    "          'max_depth': 4, \n",
    "          'min_samples_split': 2, \n",
    "          'learning_rate': 0.01, \n",
    "          'loss': 'ls'}\n",
    "clf = ensemble.GradientBoostingRegressor(**params)\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE: 6.4696\n"
     ]
    }
   ],
   "source": [
    "mse = mean_squared_error(y_test, clf.predict(X_test))\n",
    "print(\"MSE: %.4f\"%(mse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 下降曲线\n",
    "test_score = np.zeros((params['n_estimators'], ), dtype=np.float64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, y_pred in enumerate(clf.staged_predict(X_test)):\n",
    "    test_score[i] = clf.loss_(y_test, y_pred)  # LS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Deviance')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd8VGX2+PHPSYAAGmoQFKRID5ACAWkKKNJEsYBYQET3x+pawPJ1cXVF0fWLq/t1LVgQWcSGHbGhUlwFVASkigg2OoTeA4Hz++O5EwZIyMwwk0lmzvv1uq+ZuTP35lyMc/I8z33OI6qKMcaY+JUQ7QCMMcZElyUCY4yJc5YIjDEmzlkiMMaYOGeJwBhj4pwlAmOMiXOWCIwJkIjUFpHdIpIY7ViMCSdLBCbmiMjvIrJPRHaJyHYRmS0iN4rISf2+q+oqVT1VVQ+FK1ZjigNLBCZWXaSqyUAdYBTwV+Cl6IZkTPFkicDENFXdoaqTgf7AIBFpLiJJIvK4iKwSkY0i8ryIlAMQkWUi0tt3vIiUEpFsEWkpInVFREWklPfeYO/zu0TkVxH5s99xnUVkjYjcKSKbRGS9iAz2e7+ciPxLRP4QkR0iMtMvhrZeK2a7iCwUkc5F9M9l4pQlAhMXVHUOsAY4B9dCaARkAA2AmsD93kffAK7yO7Q7sFlV5+dz2k1Ab6ACMBh4QkRa+r1fA6jonf8GYLSIVPbeexxoBbQHqgB3A4dFpCbwMfCwt/8u4F0RqRbyxRtTCEsEJp6sw325DgFuV9WtqroLeAS40vvM68DFIlLee301LjkcR1U/VtVf1Pkv8Dku0fgcBEaq6kFV/QTYDTT2xiquB4aq6lpVPaSqs1U1BxgAfKKqn6jqYVX9ApgL9Arjv4MxRykV7QCMKUI1cb/z5YF5IuLbL0AigKquFJFlwEUi8iFwMZCZ38lEpCcwAte6SPDOu9jvI1tUNdfv9V7gVCAFKAv8ks9p6wD9ROQiv32lgRmBX6YxwbFEYOKCiLTGJYJJuIHjZqq6toCP+7qHEoAfVXVlPudLAt4FrgU+UNWDIjIJl1QKsxnYD9QHFh7z3mrgFVX9fwGcx5iwsK4hE9NEpII3+DsReFVVFwIv4vrzT/M+U1NEuvsdNhHoBtyE6yrKTxkgCcgGcr3WQbdAYlLVw8A44P9E5AwRSRSRdl5yeRXXGunu7S/rDTzXCvrijQmQJQITqz4UkV24v7DvBf4PN6ALrkWwEvhWRHYCU4HGvgNVdT3wDW4g9838Tu6NLdwGvAVsw40lTA4ivrtw3UjfA1uBR4EEVV0N9AH+hksyq4H/wf5fNREktjCNMcbEN/srwxhj4pwlAmOMiXOWCIwxJs5ZIjDGmDhXIuYRpKSkaN26daMdhjHGlCjz5s3brKqFlicpEYmgbt26zJ07N9phGGNMiSIifwTyOesaMsaYOGeJwBhj4pwlAmOMiXMlYozAmFh18OBB1qxZw/79+6MdiinBypYtS61atShdunRIx1siMCaK1qxZQ3JyMnXr1sWvLLYxAVNVtmzZwpo1a6hXr15I57CuIWOiaP/+/VStWtWSgAmZiFC1atWTalVaIjAmyiwJmJN1sr9DMZ0IXn8dnn8+2lEYY0zxFtOJ4N134Yknoh2FMcXXli1byMjIICMjgxo1alCzZs281wcOHAjoHIMHD2b58uUn/Mzo0aN57bXXwhEyH3zwARkZGaSnp5OamsrYsWNP+Pnp06fz7bff5vve2LFjqVatGpmZmTRs2JAePXoU+NlA3HvvvcyYUfJWFY3pweIWLWDSJNi3D8qVi3Y0xhQ/VatWZcGCBQA88MADnHrqqdx1111HfUZVUVUSEvL/u/E///lPoT/n5ptvPvlggZycHG666Sbmzp3LGWecQU5ODn/8ceLJs9OnTyclJYW2bdvm+/4111zDv//9bwCmTp1Knz59+Prrr2nUqFHQ8f3jH/8I+pjiIKZbBM2bw+HD8OOP0Y7EmJJl5cqVpKamcs0119CsWTPWr1/PkCFDyMrKolmzZowcOTLvsx07dmTBggXk5uZSqVIlhg8fTnp6Ou3atWPTpk0A3HfffXlfth07dmT48OG0adOGxo0bM3v2bAD27NnD5ZdfTmpqKn379iUrKysvSfns2LEDVaVKlSoAJCUl5X1hb9y4kcsuu4ysrCzatGnDt99+yy+//MLYsWN57LHHyMjIyPtZBenatSs33HADL774IgArVqyge/futGrVinPPPZeff/6ZrVu3Uq9ePXyLeu3atYvatWuTm5vLgAEDmDRpEgAjRoygdevWNG/enBtvvDHv8wVdf25uLrfffjvNmzcnLS2NZ599FoDvv/+eTp060apVK3r27MnGjRtD+U96QjHfIgBYsgRatYpuLMYUZtgwOOZ776RlZID3/Ru0n376iQkTJpCVlQXAqFGjqFKlCrm5uXTp0oW+ffuSmpp61DE7duygU6dOjBo1ijvuuINx48YxfPjw486tqsyZM4fJkyczcuRIpkyZwtNPP02NGjV49913WbhwIS1btjzuuNNOO43u3btTp04dzj//fC666CL69+9PQkICt912G3fffTdt27bl999/p3fv3ixZsoQ//elPpKSkMGzYsICuu2XLlrz88ssADBkyhLFjx1K/fn1mzZrFLbfcwueff05qaiozZ87knHPOYfLkyfTq1YtSpY7+Oh06dCgPPvggqsrVV1/NlClT6NmzZ4HX/9xzz7Fu3ToWLlxIYmIiW7duJScnh6FDhzJ58mRSUlJ47bXX+Pvf/86YMWMCupZAxXQiaNAAkpJg8eJoR2JMyVO/fv28JADwxhtv8NJLL5Gbm8u6dev48ccfj0sE5cqVy/uya9WqFV9//XW+577sssvyPvP7778DMHPmTP76178CkJ6eTrNmzfI9dvz48SxatIipU6cyatQopk2bxtixY5k6depRYxXbtm1j3759QV+37y/37du38+2333L55ZfnvZebmwtA//79efPNNznnnHOYOHEid9xxx3HnmTZtGo899hj79+9n8+bNeX/RF3T9U6dOZdiwYSQmJgJQpUoVFixYwNKlS+natSsAhw4dolatWkFfU2FiOhEkJkJqqiUCUzKE+pd7pJxyyil5z1esWMGTTz7JnDlzqFSpEgMGDMj3vvUyZcrkPU9MTMz74jxWUlJSoZ85kbS0NNLS0rj66qtp2rQpY8eOzfsr2z+GUPzwww80bdoUVSUlJeW47imASy65hBEjRvDggw+yePFiOnXqdNT7e/fu5ZZbbmH+/PnUrFmT++6776h/r0CvX1VJS0srMKGGS0yPEYDrHlqyJNpRGFOy7dy5k+TkZCpUqMD69ev57LPPwv4zOnTowFtvvQXA4sWL+TGfwb2dO3fy1Vdf5b1esGABderUAVz//ujRo496DyA5OZldu3YFFMOMGTMYN24cN9xwA5UrV+b000/n/fffB+Dw4cMsXLgQgAoVKpCRkcGwYcO4+OKLjxtI37dvHwkJCaSkpLBr1y7efffdQn/2BRdcwPPPP8+hQ4cA2Lp1K6mpqaxdu5Y5c+YAcODAAZYuXRrQtQQjLhLBunWwdWu0IzGm5GrZsiWpqak0adKEa6+9lg4dOoT9Z9x6662sXbuW1NRUHnzwQVJTU6lYseJRn1FV/vd//5fGjRuTkZHBww8/zLhx4wB3i+qsWbNIS0sjNTU1b8C3T58+vPXWW2RmZuY7WPzaa6+RkZFBo0aNePTRR5k0aRKNGzcGYOLEiTz//PN5XVUfffRR3nH9+/fn1VdfpX///seds2rVqgwaNIjU1FR69uzJ2WefXej1//nPf6ZGjRqkpaWRnp7OW2+9RVJSEu+88w533HEHaWlpZGZm8t133wX+jxog8fWHFWdZWVka6sI0U6ZAz57w5ZdwTOvNmKhbtmwZTZs2jXYYxUJubi65ubmULVuWFStW0K1bN1asWHHcIKzJX36/SyIyT1WzCjgkT8z/C/vuHFq82BKBMcXZ7t27Of/888nNzUVVeeGFFywJFJGY/1c+4wyoXNnGCYwp7ipVqsS8efOiHUZciugYgYjcLiJLRWSJiLwhImVFpJ6IfCciK0XkTRE5uSH+QmNwE8vsziFjjMlfxBKBiNQEbgOyVLU5kAhcCTwKPKGqDYBtwA2RisHHd+dQCRgOMcaYIhfpu4ZKAeVEpBRQHlgPnAe8473/MnBJhGOgRQvYuRNWr470TzLGmJInYolAVdcCjwOrcAlgBzAP2K6qvhkUa4Ca+R0vIkNEZK6IzM3Ozj6pWHwDxt4twMYYY/xEsmuoMtAHqAecAZwC9Aj0eFUdo6pZqppVrVq1k4olLc2NFYS7josxJV04ylADjBs3jg0bNuT73qxZszj77LPJyMigadOmPPTQQyc81/z585kyZUq+702dOpWKFSuSmZlJo0aN6NSpE5988knAcR4rnOWxS7JI3jXUFfhNVbMBROQ9oANQSURKea2CWsDaCMYAQHKyqztkicCYowVShjoQ48aNo2XLltSoUeO49wYNGsSkSZNo3rw5hw4dKnTtgvnz57NkyRJ69Mj/78YuXbrkVficP38+l156KRMmTDiuzEMgwlUeu6SL5BjBKqCtiJQXt47a+cCPwAygr/eZQcAHEYwhT0YG/PBDUfwkY2LDyy+/TJs2bcjIyOAvf/kLhw8fJjc3l4EDB9KiRQuaN2/OU089xZtvvsmCBQvo379/vi2J7OzsvASRmJiYV6hu9+7dXHfddbRp04bMzEw+/PBD9u3bx8iRI/Nm+77zzjvHxeWvZcuW3HvvvTzzzDNA/qWoDx06RJ06ddi5cyfgZiefddZZbN68+ajy2M8//zytW7cmPT2dfv365RWsGzBgAEOHDqV9+/acddZZeSUnAB555BFatGhBeno69957L5B/6eriLmItAlX9TkTeAeYDucAPwBjgY2CiiDzs7XspUjH4y8yEt9+G7duhUqWi+InGBKkY1aFesmQJ77//PrNnz6ZUqVIMGTKEiRMnUr9+fTZv3sxi737s7du3U6lSJZ5++mmeeeYZMjIyjjvXsGHDaNiwIV26dKFnz55ce+21JCUlMXLkSHr06MH48ePZtm0bZ599NosWLeL+++9nyZIleV/QhWnZsiVPP/00QIGlqHv37s0HH3zAwIEDmT17No0aNSIlJeWo8/Tr148bb7wRgOHDhzN+/HhuuukmADZt2sSsWbNYvHgxV1xxBZdeeikffvghn376KXPmzKFcuXJs9erYFFS6ujiL6IQyVR0BjDhm969Am0j+3Pz4fj8XLrQZxsYUZurUqXz//fd5Zaj37dvHmWeeSffu3Vm+fDm33XYbF154Id26dSv0XA8++CADBw7k888/Z8KECbz55ptMnTqVzz//nE8//ZRRo0YBsH//flatWhV0rP5lcgoqRd2/f3/++c9/MnDgQCZOnJhvfSBfEtq+fTu7du2id+/eee9dcskliAhpaWmsXbs272ddf/31lPOWP6xSpcoJS1cXZzE/s9gnM9M9LlhgicAUU8WoDrWqcv311+c7sLto0SI+/fRTRo8ezbvvvhvQIikNGjSgQYMGeYvE+FYamzRpEvXr1z/qs/7VRQPhKxvtizu/UtTnnHMO1113HVu2bGHy5Mn5Xte1117Lp59+SvPmzRk7duxRaxf7ykb7fkZBTlS6ujiL+eqjPjVqQPXqNmBsTCC6du3KW2+9xebNmwF3d9GqVavIzs5GVenXrx8jR45k/vz5wIlLPX/88cd5X54rVqwgKSmJ5ORkunfvntelA+4LvbBzHWvBggU88sgjeYO+BZWiFhH69OnDsGHDSE9Pp1I+/cN79uyhRo0aHDx4kNdff73Qn33BBRcwbty4vLGErVu3nrB0dXEWN4kAbMDYmEC1aNGCESNG0LVrV9LS0ujWrRsbN25k9erVnHvuuWRkZDB48GAeeeQRAAYPHsyf/vSnfAeLx48fT5MmTcjIyOC6667j9ddfJyEhgREjRrBnzx5atGhBs2bNeOCBBwA477zzWLhwIZmZmfkOFs+YMYPMzEwaN27MbbfdxrPPPpt3x1BBpajhxGWjAUaOHEnr1q3p0KHDcSuv5ad379706NGDrKwsMjIyeOKJJ4ATl64urmK+DLW/e+6Bf/0Ldu+Gk1zEyJiwsDLUJlxOpgx13LUIDh6ECCzwY4wxJVZcJQL/AWNjjDFOXCWC+vXhlFNsnMAULyWhe9YUbyf7OxRXiSAxEdLTwbvRwZioK1u2LFu2bLFkYEKmqmzZsoWyZcuGfI64mUfg07o1jBkDublgq+CZaKtVqxZr1qzhZCvsmvhWtmxZatWqFfLxcfdVmJUFTz4Jy5YdKU9tTLSULl2aevXqRTsME+fiqmsIXIsA4PvvoxuHMcYUF3GXCBo2hAoVIAzTEowxJibEXSJISIBWraxFYIwxPnGXCMCNEyxcCDk50Y7EGGOiLy4TQevWboaxV1LdGGPiWlwmAq/Euo0TGGMMcZoI6taFqlVtnMAYYyBOE4GIaxVYi8AYY+I0EYAbJ1i6FPbujXYkxhgTXXGbCNq0gUOHrFVgjDFxmwjatnWP33wT3TiMMSba4jYRVKvmZhlbIjDGxLu4TQQA7dq5RGAVgI0x8SzuE8GmTfDrr9GOxBhjoieuE0H79u7RuoeMMfEsrhNBs2aQnGyJwBgT3+I6ESQmuttILREYY+JZXCcCcN1DCxfC7t3RjsQYY6Ij7hNBu3Zw+LDVHTLGxK+4TwQ2scwYE+/iPhFUrgxNm8Ls2dGOxBhjoiPuEwG47qFvv7WJZcaY+GSJADdgvGUL/PxztCMxxpiiZ4kAOOcc9/jVV9GNwxhjosESAa74XPXq8N//RjsSY4wpepYIcCuWderkEoGNExhj4o0lAk+nTrBmDfz2W7QjMcaYomWJwNOpk3u0cQJjTLyJaCIQkUoi8o6I/CQiy0SknYhUEZEvRGSF91g5kjEEqmlTqFrVxgmMMfEn0i2CJ4EpqtoESAeWAcOBaaraEJjmvY66hAQ491xLBMaY+BOxRCAiFYFzgZcAVPWAqm4H+gAvex97GbgkUjEEq1MnN0awenW0IzHGmKITyRZBPSAb+I+I/CAiY0XkFKC6qq73PrMBqJ7fwSIyRETmisjc7OzsCIZ5hI0TGGPiUSQTQSmgJfCcqmYCezimG0hVFcj3hk1VHaOqWaqaVa1atQiGeUSLFlCpEnz5ZZH8OGOMKRYimQjWAGtU9Tvv9Tu4xLBRRE4H8B43RTCGoCQmQufOMG1atCMxxpiiE7FEoKobgNUi0tjbdT7wIzAZGOTtGwR8EKkYQtG1qxsnsAXtjTHxItJ3Dd0KvCYii4AM4BFgFHCBiKwAunqvi42uXd3j1KnRjcMYY4pKqUieXFUXAFn5vHV+JH/uyWjUCGrVcolgyJBoR2OMMZFnM4uPIQLnnw/Tp7slLI0xJtZZIshH165ufYKFC6MdiTHGRF5sJ4JVq0Jalf58r+PKxgmMMfEgthPB4MEhdfSffjo0a2aJwBgTH2I7EXTu7Pp3tm0L+tCuXd0M4/37wx+WMcYUJ7GdCDp1civNfP110Id27+6SgJWbMMbEuthOBG3aQFJSSCVFO3eGsmXhk0/CH5YxxhQnsZ0IypaFtm1DSgTlykGXLpYIjDGxL7YTAbg/7X/4AXbsCPrQXr1gxQpYuTL8YRljTHER+4mgUyc3M2zmzKAP7dXLPX76aZhjMsaYYiT2E0HbtlCmTEi1pc86Cxo3tu4hY0xsCzgRiEg5v0qiJUe5ctC+PXzxRUiH9+oFM2bA3r1hjssYY4qJgBKBiFwELACmeK8zRGRyJAMLq+7d3XyCDRuCPrRnT8jJccnAGGNiUaAtggeANsB2yKsqWi9CMYVf9+7u8fPPgz703HOhfHkbJzDGxK5AE8FBVT32tpt8l5gsltLT4bTT4LPPgj40KcnNMv74Yzc3zRhjYk2giWCpiFwNJIpIQxF5GpgdwbjCKyEBunVzLYIQakv36gW//w4//hj+0IwxJtoCTQS3As2AHOB1YAcwLFJBRUT37rB5s5tTEKSLL3brFLz/fgTiMsaYKAsoEajqXlW9V1Vbe9t9qlqyyrF16+YeQ+geOv10dxeqJQJjTCwK9K6hL0Skkt/ryiIS/DdqNJ12GmRmhpQIAC69FObPhz/+CHNcxhgTZYF2DaWo6nbfC1XdBpwWmZAiqHt3mD0bdu4M+tBLL3WP1iowxsSaQBPBYRGp7XshInUoSXcN+XTvDrm5IU0KaNAAWrSwRGCMiT2BJoJ7gZki8oqIvAp8BdwTubAipH17OPXUk+oemjkTsrPDHJcxxkRRoIPFU4CWwJvARKCVqpasMQJwNYe6dIEpU0KaFHDppe7u08klZ061McYUKpiic0nAVmAnkCoi50YmpAjr3h1++y2k2tLp6VC3rnUPGWNiS6lAPiQijwL9gaWAb0aW4rqIShZfuYnPPoOGDYM6VMS1CkaPduPNFSpEID5jjCligbYILgEaq+qFqnqRt10cycAipkEDV186xHGCvn3hwAH44IMwx2WMMVESaCL4FSgdyUCKVI8eMH26W50+SO3aQZ068MYbEYjLGGOiINBEsBdYICIviMhTvi2SgUXUxRe7BQamTQv6UBG46ipXtsjuHjLGxIJAE8Fk4CFcobl5flvJ1Lmzu400xNt/rroKDh2Cd94Jb1jGGBMNoiWgtnJWVpbOnTs3vCft1w9mzYI1a1x10iCousllVarAVyVvuNwYEydEZJ6qZhX2uUBrDTUUkXdE5EcR+dW3nXyYUdSnD6xfD/OCb9j4uoe+/hpWrYpAbMYYU4QC/VP4P8BzQC7QBZgAvBqpoIpEr16QmBjy7T9XXuke33wzjDEZY0wUBJoIyqnqNFxX0h+q+gBwYeTCKgJVqrh1KN9+O6RZxvXrw9lnw+uvRyA2Y4wpQoEmghwRSQBWiMgtInIpcGoE4yoaV18NP/8cUvcQuO6hBQts5TJjTMkWaCIYCpQHbgNaAQOBQZEKqsj07esWJX7llZAOv/JKKFUKxo8Pb1jGGFOUAi06972q7lbVNao6WFUvU9VvIx1cxFWqBBdd5GaHHTwY9OHVq8OFF8KECSEdbowxxcIJE4GI/Nt7/FBEJh+7FU2IETZggJsZNnVqSIdffz1s3OgKmhpjTElUWNE5X5/J45EOJGp69nQDx6+84p6HcHj16jBunGtcGGNMSXPCFoGq+kZRqwLfqup//bdAfoCIJIrIDyLykfe6noh8JyIrReRNESlzcpdwksqUgf79YdIk2LUr6MNLl4aBA+Gjj2DTpgjEZ4wxERboYPFFwM/eCmW9RSSg8tWeocAyv9ePAk+oagNgG3BDEOeKjAEDYN8+eO+9kA4fPNitgBnimLMxxkRVoIPFg4EGwNvAVcAvIjK2sONEpBZuvsFY77UA5wG+Kj0v40pcR1e7dlCvHrwa2hy51FR3ijFjQpqSYIwxURVwkR1VPQh8iluqch6BfYH/G7ibI4vZVAW2q2qu93oNUDO/A0VkiIjMFZG52ZEu8yniWgXTpsG6dSGd4qab3JSE6dPDHJsxxkRYoLWGeorIeGAFcDnuL/wahRzTG9jkN84QFFUdo6pZqppVrVq1UE4RnAED3J/zIU4V7tcPUlLc6mXGGFOSBNoiuBaYhFul7DpV/cTvr/qCdAAuFpHfca2I84AngUp+Ywy1gLXBhx0BjRpBmzYhd/SXLQs33OBKF61ZE+bYjDEmggIdI7gK+AE4B0BEyolIciHH3KOqtVS1LnAlMF1VrwFmAH29jw0Cis+ij4MGwaJF8N13IR3+5z+7RsWYMWGOyxhjIijQrqH/hxvgfcHbVQvXQgjFX4E7RGQlbszgpRDPE34DB0JyMjwV2uJr9eq5oqYvvujWNTbGmJIg0K6hm3FdPTsBVHUFcFqgP0RVv1TV3t7zX1W1jao2UNV+qpoTbNARk5zspgq/9ZZbqyAEf/kLbNjgpiUYY0xJEHD1UVXN+xvX6+OPzRslb7nFrUP5wguFfzYf3bu7lsGTT4Y5LmOMiZBAE8F/ReRvQDkRuQA3n+DDyIUVRQ0auP6d55+HnOAbK4mJcPvtMHu224wxprgLNBEMB7KBxcCfgU+A+yIVVNTdequrJPf22yEdfv31rnzRY4+FOS5jjImAQO8aOowbHP6LqvZV1Re1JKx6H6oLLoDGjeHpp0M6/JRT4Oab3a2ky5eHOTZjjAmzwspQi4g8ICKbgeXAchHJFpH7iya8KElIcK2COXNCvpX0llvcmjf/+leYYzPGmDArrEVwO+5uodaqWkVVqwBnAx1E5PaIRxdN114LFSrAE0+EdPhpp8F117lFazZsCG9oxhgTToUlgoHAVar6m2+Hqv4KDMDNNo5dycluhtjbb8PKlSGd4o473HyCEHuYjDGmSBSWCEqr6uZjd6pqNlA6MiEVI7ff7hYcCHHUt2FDuPxyeOYZ2LYtzLEZY0yYFJYITjQ/Nvbnzp5+uuvfGT8+5Kqkf/877NwJ//53WCMzxpiwKSwRpIvIzny2XUCLoggw6v7nf9yqMyGOFaSlwWWXuQlm27eHOTZjjAmDwpaqTFTVCvlsyaoa+11DAPXru6Usn38+5P6d+++HHTusVWCMKZ4CXpgmrg0fDrt3u87+EKSnw6WXukRgYwXGmOLGEkEg0tLgwgtd/86ePSGdYsQIN1bw6KNhjs0YY06SJYJA/e1vsGVLyEuQpafDNde4XLJ6dZhjM8aYk2CJIFDt20PPnjBqVMijvg89BIcPu9aBMcYUF5YIgvHII66T//HHQzq8bl1XeuLll2HJkvCGZowxobJEEIyMDLjySncraYgL1/ztb27S8vDhYY7NGGNCZIkgWA8/DAcPupliIahaFe65Bz7+GKZPD3NsxhgTAksEwapf31UmHTcOFi4M6RRDh8JZZ7nTHDwY5viMMSZIlghCcd99ULky3HUXhLAsQ9mybk7Bjz9aQTpjTPRZIghF5cru1p+pU+GTT0I6Re/ebkXMBx4IebjBGGPCwhJBqG66CRo1chVKQ1jbWMTNKcjJgbvvjkB8xhgTIEsEoSpdGp56ClasCHkZsgYNXE0tcrlBAAAVr0lEQVS7V1+FL78Mb3jGGBMoKQlLD2dlZencuXOjHUb+Lr8cPv0UfvoJatcO+vC9e6FFC7c65qJFUK5cBGI0xsQlEZmnqlmFfc5aBCfLV5769tBW7ixfHsaMcYugPfhgGOMyxpgAWSI4WbVru7uI3nsPPvsspFOcfz5cf72bsDx/fpjjM8aYQljXUDjk5LgKpTk5rn+nQoWgT7FtG6SmQrVq8P33kJQUgTiNMXHFuoaKUlKSW85y9eqQu4gqV4axY2HxYtfAMMaYomKJIFzatYO//tXNOJ48OaRTXHihuyv1X/+y8hPGmKJjXUPhdOAAtGnjZogtXAg1agR9ir17oWVLtyDa4sWupWCMMaGwrqFoKFPGTQrYtQuuvhoOHQr6FOXLu1Ns3AhDhoRUwcIYY4JiiSDcmjd3q5jNmAEjR4Z0iqws+Mc/4J13Ql4m2RhjAmaJIBIGD4brrnNLkn3+eUinuOsuV4/ozjvhu+/CG54xxvizRBApo0dDs2YwYACsXRv04QkJbiWzmjWhXz/Izo5AjMYYgyWCyClfHt5+243+XnFFSIXpqlRx3UPZ2a6SxYEDEYjTGBP3LBFEUpMm7nbS2bPdYsUhjPy2agX/+Q98/TXceKMNHhtjwq9UtAOIeVdc4W4lfeQRN5A8dGjQp7jySreIzUMPud6mO++MQJzGmLhliaAoPPSQq046bJhbtHjAgKBP8cADsGyZK1vduLEbSDbGmHCIWNeQiJwpIjNE5EcRWSoiQ739VUTkCxFZ4T3G/pSphAR47TU47zx3N9GHH4Z0ipdfdpPNrrzS7iQyxoRPJMcIcoE7VTUVaAvcLCKpwHBgmqo2BKZ5r2Nf2bIwaZL7Ju/XL6SVaMqXh48+gurVXTmKZcvCH6YxJv5ELBGo6npVne893wUsA2oCfYCXvY+9DFwSqRiKneRkt4hN/fpw0UUwZ07Qp6hRw01NKFUKunVzde6MMeZkFMldQyJSF8gEvgOqq6pvufYNQPUCjhkiInNFZG52LN1EX7Wq+yY/7TTo3h0WLAj6FPXrw5QpsHMnXHCBK21kjDGhingiEJFTgXeBYaq60/89dRXv8r0hUlXHqGqWqmZVq1Yt0mEWrZo1Ydo010K44AJ3S1CQMjLg449hzRro0sWSgTEmdBFNBCJSGpcEXlPV97zdG0XkdO/904FNkYyh2Kpb1yWD0qXdEmUhJIOOHV3LYO1a6NwZ1q0Le5TGmDgQybuGBHgJWKaq/+f31mRgkPd8EPBBpGIo9ho2hKlT3fOOHWHWrKBP0bGjWyFz/XqXDEKoZmGMiXORbBF0AAYC54nIAm/rBYwCLhCRFUBX73X8Sk11M49TUqBr15AWtWnf3iWDDRugQwc3ZcEYYwIVybuGZqqqqGqaqmZ42yequkVVz1fVhqraVVW3RiqGEqNePdcaaNECLr0UXnwx6FO0a+fuSN2/3yWDb74Jf5jGmNhktYaKi2rV3PqU3bu7FWlGjgy6sFDLlq5xUaWKG3b4IH473YwxQbBEUJyceqr79h40CEaMgBtucNVLg3DWWa5x0by5a1w89pgVqjPGnJglguKmdGlXbvS++9xjVpYrWheE005z3UR9+8Ldd7t8YiWsjTEFsURQHIm4QnVffAHbt0Pr1vDoo0GtgVy+PEycCH//u8snXbrYLGRjTP4sERRnXbvCokVw8cUwfLi7P/S33wI+PCHBDTVMnOhOk5np5h0YY4w/SwTFXUqKW+lswgT3bZ6W5u4qCqLjv39/mDsXzjgDevaEe++F3NwIxmyMKVEsEZQEIjBwoEsErVu7u4pat3alSANMCI0bu9LVf/qTWyOnfXubb2CMcSwRlCR16riZyOPGwbZtroJp587w7bcBHV6unGtMvPkm/Pqr6yp64gk4fDiyYRtjijdLBCVNQgIMHuz+nH/2WVi+3M0m69vXPQ/AFVfAkiWujPUdd7hcsnJlZMM2xhRflghKqtKl4aab3Df4gw+6GhOpqXDNNbB0aaGH16jh1skZP97dndqsmbvDKMhpC8aYGGCJoKQ79VS4/36XEO68001Ia94cLrus0PUsRdzctZ9+coumPfwwNG0K771nk9CMiSeWCGJF9erwz3/CH3+4P+2nT4e2baFNG3jlFcjJKfDQ00+HV1+F//4XKlaEyy933UZBzmMzxpRQlghiTdWqbvLA6tXwzDOwaxdcey2ceaabrbxmTYGHnnsuzJ8PTz3lHjMz4brrbCKaMbHOEkGsSk6Gm292C9588YUbUH7kEXfnUdeu7s6j7duPO6xUKbj1VtfTdNdd8MYb0KCBG474/feivwxjTORZIoh1Iu6L/4MP4Jdf3GyyP/5wBYiqV3eV6d5667hR4sqVXU/T8uWuVfDSSy4hXHddwDcnGWNKCNESMCqYlZWlc+fOjXYYsUPVTTV+4w1Xf2L9ejfofPHFLjGcf77LBH7WrIHHH4cxY9yaB337upySnh6lazDGFEpE5qlqVqGfs0QQ5w4dgq++gtdfh/ffhy1bXCuiZUvo1MmthdmhgytpCmza5CahjR7thh+6dXM9UBdeCImJUb4WY8xRLBGY4OXmulnK06bBjBnuue9uowYNoFUrdxfS2WezrU4Go8efwnPPwbp1ULu2q3wxaBDUqhXdyzDGOJYIzMnLyXG3D82c6da+nDcPVq1y7yUkQJMmHE7PZGmpdF5f1Jz3F9bjD+rS4fyyDBzopjIkJ0f3EoyJZ5YITGRs2ABz5rgEMW8eLFhw1C2pirAh8Qx+O1SbPxLrU77xmZzVvgaNe9SjTGoDt4RaUlIUL8CY+GGJwBSdLVtg2TK3VsJvv6G//cbORX9w6OeVVNi9jlIcWVBHRdAza5NQt46byVajhnv036pXd4PVpUpF8aKMKfkCTQT2f5o5eVWrukHljh0BEKCi99bBnMPM+GAL37z2K6unr6D67pU0WbOC5rtWU2vZPCrsWU/C3j35n7dSJahWzfUvJSVBmTLuMTnZ3eVUurTbV7as25KToUIFt1WseOS5/74yZYrkn8SYksRaBKbIHDzoblD66CP48EM3rQGgddPd9Gmznk6N1pNRfT2n7suGzZtdSyM7G/bsceMVOTnu3tXdu92Wm+sWY87JcfMgAlnKMynp+ORQoYJLIqVLuy0pySWW0qVdq8T3OinJ3Xp7+LC7syopyW2JiW7MpFSpgjffuY7dEhOPbAUd6zu/MUGyriFTrKnCzz+7pPDJJzB7tvuOF3GlLTp3dussd+zoGgYBnTAnB3bsgJ07j2yFvfbt27XLJZaDB9159u078ro4KCzR+DZfwhBxzwt69E88kXyekHBk8/1s31amzJFWnv9zX0vP9+jb70vIZcq4c5lCWSIwJcr+/a5Y6pdfujtXv/nG/bEv4lZXa9PGLcrWpo2bxFZk482qR5KDL1MlJrpWga+Vcviwa40cOuSSh/928ODx+47d/I/N7xyBbgcPunj9t8OHj3/ML96TeR6NdU99icG/tXbsYzje809S+W3+Lb1imJwsEZgSbd8+N41h5kz4/nt3o9LGje690qXd+glpaW5LT3eP3pw3Ew2HD+efIPyTj29TdZ/zJVhf957v8eDBI9uBA0f2+5Jxfo8neq+gx3Dzb/GIHNn8Xx/bMiroc/7b9OluHk9IIdlgsSnBypVzXUNdurjXqu4u1TlzXGJYuNDV0psw4cgx1asfSQpNmkCjRtCwodtfDP9Yiy2+rp6SwtfSKyxZ+BLRibZjE59/C+xEzw8dKrgF57+dckrE/zmsRWBKtOxsWLzYJYZFi9y2dOnRf/AlJ7uE4EsMvseGDaFKlejFbkykWdeQiVu5uW4C9IoVbkB6xYojz3//3f3R5VO16tHJoUEDV6m7dm03xcHqJ5mSzBKBMfk4cAB+/fXoJOF7PHbNnlKloGZNt6ZP7dpHHmvWPHruW0nqETHxxcYIjMlHmTJu/KBJk+Pf27vXJYnVq12LYtWqI8+/+Qbefjv/u0mrVj1+crRvq1HDzYlLSXHdUNbCMMWRJQJjPOXLQ/PmbsvP4cOu1NK6de5x/fqjtw0b3KI969fnnzBEXDJISTl6q1zZ7a9c+ejnvseKFW0+mYksSwTGBCghAc44w20nogpbtx5JDps3uy07+8jzzZtd62POHNi2zd2kUhARN6nu2ERx7HNf5Y1TTnGP/s9POcW6sEzBLBEYE2YirruoatWCWxfH2rfPJYRt21wSOdHzrVvdaqO+54FU1gA3/6KgJFFQAilf/uj5Vv5bQfuSkux23ZLGEoExxUC5cm4rrLVxLFVXdmnbNlclY88e9/rYx/z2+R43bnR1n/z3nexk4WOTRDCJJBz7Spe2ZBQMSwTGlGAirkso3AsAHTjgEsLevUfmV/m2Y18Hs8+3f+fOE3/2ZG9mFAlPYvHVIQxm89UYDGQrLjcPWCIwxhynTBk3/hCNCXfHlncKVxI6dt+2bSf+XFEQKTxZfPgh1K8f2TgsERhjihWRIzXdorXUqaprFfmXPQpl89UCPJmtXLnIX29UEoGI9ACeBBKBsao6KhpxGGNMfvyXm4gHRX53sogkAqOBnkAqcJWIpBZ1HMYYY5xoTFNpA6xU1V9V9QAwEegThTiMMcYQnURQE1jt93qNt+8oIjJEROaKyNzs7OwiC84YY+JNsZ24rqpjVDVLVbOqVasW7XCMMSZmRSMRrAXO9Htdy9tnjDEmCqKRCL4HGopIPREpA1wJTI5CHMYYY4jC7aOqmisitwCf4W4fHaeqS4s6DmOMMU5U5hGo6ifAJ9H42cYYY45WIlYoE5Fs4I8QDk0BNoc5nOLOrjk+2DXHh5O95jqqWujdNiUiEYRKROYGskxbLLFrjg92zfGhqK652N4+aowxpmhYIjDGmDgX64lgTLQDiAK75vhg1xwfiuSaY3qMwBhjTOFivUVgjDGmEJYIjDEmzsVsIhCRHiKyXERWisjwaMcTLiIyTkQ2icgSv31VROQLEVnhPVb29ouIPOX9GywSkZbRizx0InKmiMwQkR9FZKmIDPX2x+x1i0hZEZkjIgu9a37Q219PRL7zru1Nr0wLIpLkvV7pvV83mvGHSkQSReQHEfnIex3T1wsgIr+LyGIRWSAic719Rfq7HZOJIMYXvxkP9Dhm33Bgmqo2BKZ5r8Fdf0NvGwI8V0QxhlsucKeqpgJtgZu9/56xfN05wHmqmg5kAD1EpC3wKPCEqjYAtgE3eJ+/Adjm7X/C+1xJNBRY5vc61q/Xp4uqZvjNGSja321VjbkNaAd85vf6HuCeaMcVxuurCyzxe70cON17fjqw3Hv+AnBVfp8ryRvwAXBBvFw3UB6YD5yNm2Vaytuf93uOq93VznteyvucRDv2IK+zFu5L7zzgI0Bi+Xr9rvt3IOWYfUX6ux2TLQICXPwmhlRX1fXe8w1Ade95zP07eF0AmcB3xPh1e90kC4BNwBfAL8B2Vc31PuJ/XXnX7L2/A6hatBGftH8DdwOHvddVie3r9VHgcxGZJyJDvH1F+rsdlaJzJnJUVUUkJu8JFpFTgXeBYaq6U0Ty3ovF61bVQ0CGiFQC3geaRDmkiBGR3sAmVZ0nIp2jHU8R66iqa0XkNOALEfnJ/82i+N2O1RZBvC1+s1FETgfwHjd5+2Pm30FESuOSwGuq+p63O+avG0BVtwMzcF0jlUTE9wec/3XlXbP3fkVgSxGHejI6ABeLyO+4dczPA54kdq83j6qu9R434RJ+G4r4dztWE0G8LX4zGRjkPR+E60P37b/Wu9OgLbDDr7lZYoj70/8lYJmq/p/fWzF73SJSzWsJICLlcGMiy3AJoa/3sWOv2fdv0ReYrl4nckmgqveoai1VrYv7/3W6ql5DjF6vj4icIiLJvudAN2AJRf27He2BkggOwPQCfsb1q94b7XjCeF1vAOuBg7j+wRtwfaPTgBXAVKCK91nB3T31C7AYyIp2/CFec0dcP+oiYIG39Yrl6wbSgB+8a14C3O/tPwuYA6wE3gaSvP1lvdcrvffPivY1nMS1dwY+iofr9a5vobct9X1XFfXvtpWYMMaYOBerXUPGGGMCZInAGGPinCUCY4yJc5YIjDEmzlkiMMaYOGeJwBQLInLIq764UETmi0j7MJ//b8e8nh2m83b2q5TZOZxxi0hdEbna73WWiDwVrvMb42OJwBQX+9RVX0zHFQn83zCf/6hEoKphTTSezkBQ5/WbNZufukBeIlDVuap6W0iRGXMClghMcVQBV3LYV3/9MRFZ4tVs71/I/tNF5CuvdbFERM4RkVFAOW/fa97ndnuPnUXkSxF5R0R+EpHXvJnMiEgvb988rwb8RwUF7BXDuxG43fs553izg98Vke+9rYP32QdE5BURmQW84v3l/7XXEvJvDY0CzvHOd/sxrY8qIjJJXE36b0Ukze/c47xr+lVEbvP2nyIiH3striW+fy9jwIrOmeKjnLhKm2VxZXfP8/ZfhqvHnw6kAN+LyFe4v7zz2381rlTxP8StS1FeVb8WkVtUNaOAn50JNAPWAbOADuIWCHkBOFdVfxORN04UvKr+LiLPA7tV9XEAEXkdV0t/pojUxpVObuodkoorNrZPRMoDF6jqfhFpiJs9noWrQX+Xqvb2ztfZ70c+CPygqpeIyHnABO/fA1xxui5AMrBcRJ7DrWGxTlUv9M5V8UTXY+KLJQJTXOzzfVGLSDtggog0x5WXeENdJc6NIvJfoPUJ9n8PjBNXpG6Sqi4I4GfPUdU13s9egOuS2Q38qqq/eZ95A7cQSDC6AqlypEpqBXEVVAEmq+o+73lp4BkRyQAOAY0COHdH4HIAVZ0uIlVFpIL33seqmgPkiMgmXAnjxcC/RORRXPmGr4O8FhPDrGvIFDuq+g3ur/xqIRz7FXAuriLjeBG5NoDDcvyeHyJ8fyAlAG29sY8MVa2pqru99/b4fe52YCOudZMFlDnJn3vc9ajqz0BLXEJ4WETuP8mfYWKIJQJT7IhIEyARV1b4a6C/uEVaquG+5OcUtF9E6gAbVfVFYCzuyw/goNdKCNRy4Cw5shZuIH3qu3DdMT6fA7f6XVdBXVMVgfWqehgYiLv2/M7n72vgGu+8nYHNqrqzoMBE5Axgr6q+CjzGkX8XY6xryBQbvjECcBUWB6nqIRF5H1eHfyGuAundqrrhBPsHAf8jIgdx3Tu+FsEYYJGIzFdX3viEvL77vwBTRGQPrsupMB8C74hIH1wCuA0YLSKLcP+vfYUbUD7Ws8C7XutlCkdaC4uAQyKyELdW9Q9+xzyA6wJbBOzlSMnigrQAHhORw7jKtTcFcD0mTlj1UWMKICKnqupu7y6i0cAKVX0i2nEZE27WNWRMwf6f10pZiuu+eSHK8RgTEdYiMMaYOGctAmOMiXOWCIwxJs5ZIjDGmDhnicAYY+KcJQJjjIlz/x+M7CCWFx0s9gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title('Deviance')\n",
    "plt.plot(np.arange(params['n_estimators'])+1, clf.train_score_, 'b-', label='Training Set Deviance')\n",
    "plt.plot(np.arange(params['n_estimators'])+1, test_score, 'r-', label='Test Set Deviance')\n",
    "plt.legend(loc='upper right')\n",
    "plt.xlabel('Boosting Iterations')\n",
    "plt.ylabel('Deviance')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Variable Importance')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xm8VXW9//HXW1AQUXBEROIk4axh8rP6XTUccjaHBjlXUxp+5s0yvWqZTWg5lYWappfKUG/OpqGV5lW5WjmBTAJqDjigKEriRCr4+f2xvicXm7MP+7DOHs7Z7+fjsR9nre/6rrU/62zYn/Nda+/PVxGBmZnZqlqt3gGYmVn35kRiZmaFOJGYmVkhTiRmZlaIE4mZmRXiRGJmZoU4kViPJukDkt6Q1KuCvqMlPdfB9omSftS1EZp1f04k1jAk3Srp9HbaD5K0QFLvzh4zIp6JiP4Rsaxrolw1kkLSh+oZQxtJ8yTtWe84rOdwIrFGchlwhCSVtH8e+G1ELO3MwVYl8fRk/n1YtTiRWCO5CVgf2KWtQdK6wAHA5Wl9f0nTJL0m6VlJ43J9W9Jf/l+S9AxwZ66td+rzBUlzJb0u6UlJXykNQtKpkl5Of7kfXi5YSQdImi7pVUl/k7R9JScpaZyk6yT9d4pjlqTNJX1b0kvpvPbK9Z8s6SxJD6Tz/r2k9XLbPyVpdopjsqStctvmSfqWpJnAm5KuAj4A3Jwu+X0z9bsujfoWS7pb0ja5Y0yUdJGkP6R475c0PLd9G0m3S1ok6UVJp6b21SSdIukJSa9IujYft/UcTiTWMCJiCXAtcGSu+XPAIxExI62/mbYPBPYH/kPSwSWH+gSwFbB3O0/zElliWgf4AjBe0kdy2zcGNgCGAEcBEyRtUXoQSTsAlwJfIUt+/wVMktSnwtM9ELgCWBeYBtxG9v9xCHB6Ol7ekcAXgcHAUuCCFMfmwFXA8cCGwB/JksQauX1byX5XAyOiFXgGODBd8vtx6vMnYASwEfAQ8NuS5x8DnJbifRw4Iz3/2sD/ALcCmwAfAu5I+3wdOJjs9dgE+AdwUYW/H+tOIsIPPxrmAewMvAr0Tet/BU7ooP95wPi03AIEsFlue1tb7zL73wR8Iy2PJnuTXiu3/Vrge2l5IvCjtHwx8MOSYz0KfKLM8wTwobQ8Drg9t+1A4A2gV1pfO/UfmNYnA2fn+m8NvAP0Ar4HXJvbthowHxid1ucBXyyJZR6wZwe/04Hp+QfkzvtXue37kSV3yJLUtDLHmQvskVsfDLxb7rXwo/s+PCKxhhIRfwFeBg5Ol092Aq5s2y7po5LukrRQ0mLgGLIRRN6z5Y4vaV9J96XLMK+SvSnm9/9HRLyZW3+a7K/pUsOAE9PlpFfTsYaW6dueF3PLS4CX4/0PBCxJP/vn+uTP6Wlg9RT3JmkdgIh4L/UdUmbfFUjqJensdAnqNbJEA8v/Xhbklt/KxTYUeKLMoYcBN+Z+P3OBZcCgjuKx7seJxBrR5WSXco4AbouI/JvulcAkYGhEDAAuAUpvzrdb0jpddroBOBcYFBEDyS4F5fdfV9JaufUPAM+3c7hngTMiYmDu0S8irqr4LDtnaElM75Il3OfJ3rABSB9UGEo2KmlT+vsoXf934CBgT2AA2SgOVvy9tudZYLMOtu1b8jvqGxHzy/S3bsqJxBrR5WRvav+P7JNceWsDiyLin5J2InsTrNQaQB9gIbBU0r7AXu30O03SGpJ2Ibufcl07fX4JHJNGSJK0VvogwNqdiKczjpC0taR+ZPdQrk8jmGuB/SXtIWl14ETgbeBvHRzrRZZ/81877fMK0A84sxNx3QIMlnS8pD6S1pb00bTtEuAMScMAJG0o6aBOHNu6CScSazgRMY/sjXAtstFH3leB0yW9Dnyf7I200uO+DhyX9vkHWRIqPf6CtO15shvOx0TEI+0cawpZorsw9X8cGFtpLKvgCrJ7FQuAvmTnQUQ8SjZy+znZCOVAshvp73RwrLOA76ZLTieRJe6nyUYxc4D7Kg0q/U4/mZ53AfB3YLe0+Xyy3++f0+t1H/DR9o5j3ZsiPLGVWSOTNBn474j4Vb1jMWuPRyRmZlaIE4mZmRXiS1tmZlaIRyRmZlZIUxRx22CDDaKlpaXeYZiZdRtTp059OSI2rKRvUySSlpYWpkyZUu8wzMy6DUlPr7xXxpe2zMysECcSMzMrxInEzMwKcSIxM7NCnEjMzKwQJxIzMyvEicTMzApxIjEzs0Ka4guJs+YvpuWUP9Q7DDOzmpl39v41ey6PSMzMrBAnEjMzK6TLE4mkN9pp20LSZEnTJc2VNEHS3ml9uqQ3JD2ali/P7XeepPmSVkvrX8jt846kWWn57K4+DzMzq0yt7pFcAIyPiN8DSNouImYBt6X1ycBJaR5sUttqwCHAs8AngLsi4jfAb9L2ecBuEfFyjc7BzMzaUatLW4OB59pWUhJZmdHAbOBioLU6YZmZWVG1SiTjgTsl/UnSCZIGVrBPK3AVcCOwv6TVO/OEko6WNEXSlGVvLV6FkM3MrBI1SSTpktRWwHVkI437JPUp11/SGsB+wE0R8RpwP7B3J59zQkSMiohRvfoNWOXYzcysYzX71FZEPB8Rl0bEQcBSYNsOuu8NDARmpXshO+PLW2ZmDakmiUTSPm2XpiRtDKwPzO9gl1bgyxHREhEtwAeBT0rqV/VgzcysU6rxqa1+kp7Lrf8M2BQ4X9I/U9vJEbGgvZ1TstgHOKatLSLelPQX4EDgmirEbGZmq6jLE0lElBvl/GcH+4zOLb8FrNdOn0NL1ltWLUIzM+tK/ma7mZkV0hRFG7cbMoApNSxgZmbWTDwiMTOzQpxIzMyskKa4tOX5SMyaSy3n4jCPSMzMrCAnEjMzK6ShEomkZWl+kYcl3dxW3FFSi6SQ9KNc3w0kvSvpwvpFbGZmDZVIgCURMTIitgUWAcfmtj0F5C98fpaszLyZmdVRoyWSvHuBIbn1t4C5kkal9cOAa2selZmZLachE4mkXsAewKSSTVcDYyQNBZYBz3dwDM9HYmZWA42WSNaUNB1YAAwCbi/ZfivwSWAMKyne6PlIzMxqo9ESyZKIGAkMA8Ty90iIiHeAqcCJwPW1D8/MzEo1WiIB/lUB+DjgREmlX5r8KfCtiFhU+8jMzKxUQyYSgIiYBsykZGbEiJgdEZfVJyozMyvVUCVSIqJ/yfqBudUVpuaNiInAxOpGZWZmHWmoRFItLiNvZlY9DXtpy8zMugcnEjMzK6QpLm319DLyLpltZvXkEYmZmRXiRGJmZoU0TCLJlZCfLWmGpBMlrZa2jZZ0S1oeJOmW1GeOpD/WN3Izs+bWSPdI2sqjIGkj4EpgHeAHJf1OB26PiPNT3+1rGqWZmS2nYUYkeRHxEnA08DVJKtk8GHgu13dmLWMzM7PlNWQiAYiIJ4FewEYlmy4Cfi3pLknfkbRJe/u7jLyZWW00bCIpJyJuAzYDfglsCUyTtGE7/VxG3sysBho2kUjajGzyqpdKt0XEooi4MiI+DzwI7Frr+MzMLNOQiSSNMC4BLoyIKNm2u6R+aXltYDjwTO2jNDMzaKxPbbXNjrg6sBS4AvhZO/12BC6UtJQsEf4qIh6sXZhmZpbXMIkkInp1sG0yMDkt/wT4SW2iMjOzlWmYRFJNLiNvZlY9DXmPxMzMug8nEjMzK8SJxMzMCmmKeyRdOR+J5/4wM1ueRyRmZlaIE4mZmRVS80QiKST9NLd+kqRxufWjJT2SHg9I2jm195I0VdKuub5/lvTZmp6AmZktpx4jkreBQyVtULpB0gHAV4CdI2JL4BjgSkkbR8Qy4Ktk32pfXVIr8F5EXFfL4M3MbHn1SCRLgQnACe1s+xZwckS8DBARDwGXAcem9fuBe4FxwJnA12oQr5mZdaBe90guAg6XVFrffRtgaknblNTe5tvA8cCVEfF4uSfwfCRmZrVRl0QSEa8BlwPHrcLuuwKLgW1X8hyej8TMrAbq+amt84AvAWvl2uaQVffN2xGYDSBpLeDHwO7ARpL2q0GcZmbWgbolkohYBFxLlkza/Bg4R9L6AJJGAmOBX6Tt3weujYhHyG68j5fUt2ZBm5nZCur9zfafkrthHhGTJA0B/iYpgNeBIyLiBUnbAIcAH059p0m6jewG/Wm1D93MzKAOiSQi+ueWXwT6lWy/GLi4nf1mA5uXtK3KPRYzM+tC9R6R1ITnIzEzqx6XSDEzs0KcSMzMrJCmuLTVVWXkXULezGxFHpGYmVkhTiRmZlZIzRKJpI0lXS3piVQO/o+SNpe0RNJ0SXMkXS5p9dR/tKRb0vLYVH5+z9zxDk5tn6nVOZiZ2YpqkkgkCbgRmBwRwyNiR7Lii4OAJyJiJLAdsCnwuTKHmQWMya23AjOqF7WZmVWiViOS3YB3I+KStoaImAE8m1tfBjwADClzjHuAndJcJP2BDwHTqxeymZlVolaJZFtWLA+/nFQz66PArWW6BPA/wN7AQcCklRzPZeTNzGqgEW62D5c0HXgReCEiZnbQ92qyy1tjgKs6OqjLyJuZ1UatEslsViwP36btHslwYEdJnyp3kIh4gOxeygYR8VjXh2lmZp1Vq0RyJ9BH0tFtDZK2B4a2rafpdU8huwnfkVOAU6sRpJmZdV5NEklEBFkJ+D3Tx39nA2cBC0q63gT0k7RLB8f6U0TcVb1ozcysM2pWIiUinqf9j/Zum+sTpPlGksmpfSIwsZ1jju3CEM3MbBU0Ra0tl5E3M6ueRvjUlpmZdWNOJGZmVkhTXNoqUkbepePNzDrmEYmZmRXiRGJmZoWsNJFIWpbKvD8s6TpJQ9L6dEkLJM3Pra9R0v9mSQNLjne8pH9KGpDW987t/4akR9Py5flS8qnvwZJmSporaZakg7v+V2JmZp1RyYhkSUSMjIhtgXeAw9L6SOASYHzbekS8U9J/EXBsyfFagQeBQwEi4rbc8aYAh6f1I/M7SfowcC5wUERsBXwKODd9Q97MzOqks5e27iEr316pe8mVhZc0HOgPfJcsoXTGScCZEfEUQPp5FnByJ49jZmZdqOJEIqk3sC/ZBFOV9O8F7MHy5d7HkFXwvQfYQtKgykNlG1YsRT8ltZuZWZ1UkkjWTGXepwDPAL+usP8CshkQb89tawWujoj3gBuAz3Y+5Mp4PhIzs9rozD2SkRHx9XQfZKX9gWGASPdIJG0HjABulzSPbHTSmctbc1ixFP2OZCXqV+D5SMzMaqNqH/+NiLeA44AT02WxVmBcRLSkxybAJpKGVXjIc4FvS2oBSD9PBX7axaGbmVknVPV7JBExDZhJlkTGADeWdLkxtVdyrOnAt4CbJT0C3Ax8M7WbmVmdKKvc3rP1GTwiBh913irt6xIpZtaMJE2NiFGV9PU3283MrJCmKNro+UjMzKrHIxIzMyvEicTMzAppiktblc5H4hvrZmad5xGJmZkV4kRiZmaF1C2RSFp/JfOaHCwpJG2Z22eUpNmS1kjrwyU9KWmdep2HmVmzq1siiYhXVjKvSSvwF3L1uCJiCvC/ZCXlAS4CvhMRr9U4fDMzSxryZruk/sDOwG5kpVB+kNt8KjBN0lKgd0RcVYcQzcwsachEAhwE3BoRj0l6RdKOETEVICJelXQ28Atg63IHkHQ0cDRAr3U2rEXMZmZNqVFvtreSTYBF+llabn5f4EU6SCQuI29mVhsNNyKRtB6wO7CdpAB6ASHp5IgISQcAA4C9gRsl3ZZK1puZWR004ojkM8AVETEszVsyFHgK2EXSmsDPgGMjYhbwe+A7dYzVzKzpNWIiaWXFeUtuSO3fA26MiDmpfRzQKmlE7cIzM7O8hri0FRHjcsu7tbP9gjL7vQ5sVr3IzMxsZRoikVSby8ibmVVPI17aMjOzbsSJxMzMCmmKS1srKyPv8vFmZqvOIxIzMyvEicTMzAqpayIpUyp+hKRbJD0haaqkuyTtmraNlbQwV25+uqSyZVLMzKz66j0iWa5UvKS+wB+ACRExPCJ2BL7O8t8VuSZXbn5k7suJZmZWB/Wc2KqtVPyXgDGp+XDg3oiY1NYvIh6OiIm1j9DMzCpRz09trVAqHtgGeGgl+x0maefc+scjYknVojQzsw7VM5G0Auen5fZKxSPpRmAE8FhEHJqar4mIr63s4J6PxMysNuqSSMqVigdOA3Zt6xcRh0gaBZzb2eeIiAnABIA+g0dEV8RtZmYrqtc9knKl4h8H/k3Sp3J9+9UlQjMzq0i9Lm21AueUtN1AdtP9AOBnks4jmwXxdeBHuX6l90i+GhF/q2awZmZWXl0SSQWl4vcrs99EYGJ1ojIzs1VR7++RmJlZN9cURRs9H4mZWfV4RGJmZoU4kZiZWSFNcWnL85GYmVWPRyRmZlaIE4mZmRXS7RKJpGVpHpIZkh6S9H/rHZOZWTPrjvdIlkTESABJewNnAZ+ob0hmZs2r241ISqwD/KPeQZiZNbPuOCJZU9J0oC8wmKyK8ApcRt7MrDa644hkSZpid0tgH+BySSrtFBETImJURIzq1W9A7aM0M2sS3TGR/EtE3AtsAHjIYWZWJ906kUjakmxSrFfqHYuZWbPqzvdIAAQcFRHL6hmQmVkz63aJJCJ61TsGMzN7X7dLJKvCZeTNzKqnW98jMTOz+nMiMTOzQpoikcyav7jeIZiZ9VhNkUjMzKx6nEjMzKyQuiSSXCn4hyXdLGlgyfbjJf1T0oBc22hJiyVNk/SopLslHVD76M3MLK9eI5K2elnbAouAY0u2twIPAoeWtN8TETtExBbAccCFkvaofrhmZlZOI1zauhcY0rYiaTjQH/guWUJpV0RMB04HvlbtAM3MrLy6JhJJvYA9gEm55jHA1cA9wBaSBnVwiIeALcsc+2hJUyRNWfaWP7VlZlYt9UokbfWyFgCDgNtz21qBqyPiPeAG4LMdHGeF8vFtXEbezKw26nqPBBhGlgyOBZC0HTACuF3SPLLRSdnLW8AOwNzqhmpmZh2p66WtiHiL7Kb5iZJ6kyWNcRHRkh6bAJtIGla6r6Ttge8BF9U0aDMzW07dizZGxDRJM8mSyBhgv5IuN6b2+4FdJE0D+gEvAcdFxB21jNfMzJaniKh3DFXXZ/CIePuFv9c7DDOzbkPS1IgYVUnfRvj4r5mZdWNNkUi2G+JPbZmZVUtTJBIzM6seJxIzMyukKRKJ5yMxM6uepkgkZmZWPU4kZmZWSJcmEklvpJ8tkkLS13PbLpQ0Ni1PlPSUpBmSHpN0uaRNS4+TWx8r6cK0vIWkyWk+k7mSJnTlOZiZWedUc0TyEvANSWuU2X5yRHwY2AKYBtzZQd+8C4DxaT6TrYCfd024Zma2KqqZSBYCdwBHddQpMuPJKgHvW8FxBwPP5fafVSRIMzMrptr3SM4BTkrzjqxM2blFSownG738SdIJpdP0tvF8JGZmtVHVRBIRT5IVW/z3CrqXnVuk7XDpmL8BtgKuA0YD90nq085zez4SM7MaqMWnts4EvsXKE0V+bpElJfdL1gNebluJiOcj4tKIOAhYCmzbhfGamVknVD2RRMQjwBzgwPa2K3Mc2b2PW1Pz/wJHpO1rAp8D7krr+0haPS1vDKwPzK/mOZiZWXm1+h7JGcCmJW0/kTQDeAz4P8BuEfFO2vYN4NA0He99wHURcXfathfwcNr3NrJPfy2o+hmYmVm7PB+JmZmtwPORlHAZeTOz6mmKRGJmZtXjRGJmZoU4kZiZWSFOJGZmVogTiZmZFdJwiUTSIalEfP7xnqT/6Kg0vZmZ1UfDJZKIuDGViB8ZESOBXwD3kH35cGWl6c3MrMYaLpHkSdoc+D7weeA9KixNb2ZmtdOwiSTV07oSODEinsltqqg0fb6M/MKFC6sZqplZU2vYRAL8EJgdEdfkGystTZ8vI7/hhhtWMUwzs+bWu94BtEfSaODTwEfKdDkTuJ6sSrCZmdVRw41IJK0L/AY4MiJeb6/PykrTm5lZ7TTiiOQYYCPgYmm5ubCuKul3BjCtVkGZmVn7Gi6RRMRZwFllNp+T6zeDBhxRmZk1G78Rm5lZIU4kZmZWiBOJmZkV4kRiZmaFOJGYmVkhTiRmZlaIE4mZmRVS9UQiaWNJV0t6QtJUSX+UtLmkh0v6jZN0Um69t6SFks4u6XeApGmSZkiaI+kr1T4HMzMrr6pfSFT21fQbgcsiYkxq+zAwqILdPwk8BnxW0rcjIlJF4AnAThHxnKQ+QEt1ojczs0pUe0SyG/BuRFzS1pC+kf5sBfu2AucDzwAfT21rkyW/V9Kx3o6IR7s0YjMz65RqJ5Jtgalltg3PT6dLVmMLAEl9gT2Bm8lqbLUCRMQiYBLwtKSrJB0uqd1z8HwkZma1Uc+b7U+UTKl7SW7bAcBdEbEEuAE4uG0iq4j4MrAH8ABwEnBpewf3fCRmZrVR7UQyG9hxFfZrBfaUNI9sRLM+sHvbxoiYFRHjye6jfLoL4jQzs1VU7URyJ9BH0tFtDZK2B4aW20HSOsAuwAcioiUiWoBjgVZJ/dOkV21GAk9XI3AzM6tMVRNJRARwCNno4glJs8lKxC/oYLdDgDsj4u1c2+/JJrHqBXxT0qPpvsppwNiqBG9mZhVR9l7fs40aNSqmTJlS7zDMzLoNSVMjYlQlff3NdjMzK8SJxMzMCnEiMTOzQpxIzMysECcSMzMrxInEzMwKcSIxM7NCnEjMzKwQJxIzMyvEicTMzAppihIpkl4HmnUCrA2Al+sdRB35/H3+zXr+Rc99WERUNAdHVafabSCPVlozpqeRNKVZzx18/j7/5j3/Wp67L22ZmVkhTiRmZlZIsySSCfUOoI6a+dzB5+/zb141O/emuNluZmbV0ywjEjMzqxInEjMzK6RHJxJJ+6T53R+XdEq946k2SUMl3SVpjqTZkr6R2teTdLukv6ef69Y71mqR1EvSNEm3pPUPSro//Ru4RtIa9Y6xWiQNlHS9pEckzZX08SZ77U9I/+4flnSVpL49+fWXdKmklyQ9nGtr9/VW5oL0e5gp6SNdGUuPTSSSegEXAfsCWwOtkraub1RVtxQ4MSK2Bj4GHJvO+RTgjogYAdyR1nuqbwBzc+vnAOMj4kPAP4Av1SWq2jgfuDUitgQ+TPZ7aIrXXtIQ4DhgVERsC/QCxtCzX/+JwD4lbeVe732BEelxNHBxVwbSYxMJsBPweEQ8GRHvAFcDB9U5pqqKiBci4qG0/DrZG8kQsvO+LHW7DDi4PhFWl6RNgf2BX6V1AbsD16cuPfncBwC7Ar8GiIh3IuJVmuS1T3oDa0rqDfQDXqAHv/4RcTewqKS53Ot9EHB5ZO4DBkoa3FWx9OREMgR4Nrf+XGprCpJagB2A+4FBEfFC2rQAGFSnsKrtPOCbwHtpfX3g1YhYmtZ78r+BDwILgd+kS3u/krQWTfLaR8R84FzgGbIEshiYSvO8/m3Kvd5VfT/syYmkaUnqD9wAHB8Rr+W3RfZ57x73mW9JBwAvRcTUesdSJ72BjwAXR8QOwJuUXMbqqa89QLoXcBBZQt0EWIsVL/s0lVq+3j05kcwHhubWN01tPZqk1cmSyG8j4nep+cW2YWz6+VK94quifwM+JWke2WXM3cnuGQxMlzqgZ/8beA54LiLuT+vXkyWWZnjtAfYEnoqIhRHxLvA7sn8TzfL6tyn3elf1/bAnJ5IHgRHpUxtrkN14m1TnmKoq3RP4NTA3In6W2zQJOCotHwX8vtaxVVtEfDsiNo2IFrLX+s6IOBy4C/hM6tYjzx0gIhYAz0raIjXtAcyhCV775BngY5L6pf8HbeffFK9/TrnXexJwZPr01seAxblLYIX16G+2S9qP7Lp5L+DSiDijziFVlaSdgXuAWbx/n+BUsvsk1wIfAJ4GPhcRpTfpegxJo4GTIuIASZuRjVDWA6YBR0TE2/WMr1okjST7oMEawJPAF8j+WGyK117SacBhZJ9enAZ8mew+QI98/SVdBYwmKxf/IvAD4Cbaeb1Tcr2Q7HLfW8AXImJKl8XSkxOJmZlVX0++tGVmZjXgRGJmZoU4kZiZWSFOJGZmVogTiZmZFeJEYt2WpGWSpqdqrzdLGljBPm+sZPtASV/NrW8i6fqO9qkw1pZ8ldZakDQyfQTerKqcSKw7WxIRI1O110XAsV1wzIHAvxJJRDwfEZ/poH9DSt/mHgk4kVjVOZFYT3EvuSJ0kk6W9GCae+G00s6S+ku6Q9JDkmZJaqsMfTYwPI10fpIfSUi6T9I2uWNMljRK0lppbogHUsHEDqtMSxor6aY0X8Q8SV+T9J9p3/skrZc7/vm5UddOqX29tP/M1H/71D5O0hWS/gpcAZwOHJb2P0zSTpLuTc/zt7Zvwad4fifpVmXzWPw4F+s+6Xc0Q9Idqa1T52tNICL88KNbPoA30s9ewHXAPml9L2ACILI/lm4Bdi3ZpzewTlreAHg89W8BHs49x7/WgROA09LyYODRtHwm2TemIRvRPAasVRJr/jhj0/OtDWxIVqn2mLRtPFmxTYDJwC/T8q65/X8O/CAt7w5MT8vjyCrerpl7ngtzMawD9E7LewI35Po9CQwA+pJ9I3poiu1Z4IOp33qVnq8fzfVoK2Zm1h2tKWk62UhkLnB7at8rPaal9f5kE/rcndtXwJmSdiUrJzOElZdYvxb4M1kpis/x/jwXe5EVjDwprfclK1Exd4UjvO+uyOaMeV3SYuDm1D4L2D7X7yrI5p6QtE66D7Qz8OnUfqek9SWtk/pPioglZZ5zAHCZpBFkVWFXz227IyIWA0iaAwwD1gXujoin0nO1lVZZlfO1HsyJxLqzJRExUlI/4DayeyQXkCWJsyLivzrY93Cyv7h3jIh3U9Xgvh09WUTMl/RKupR0GHBM2iTg0xHxaCdiz9d7ei+3/h7L/78srWG0sppGb3aw7YdkCewQZfPVTC4TzzI6fm9YlfO1Hsz3SKzbi4i3yKZZPTHdZL4N+KKyeVmQNETSRiW7DSCbv+RdSbuR/QUO8DrZJadyriGbPGtARMxMbbcBX0+F8ZC0Q1ecV3JYOubOZBVbF5MV5jw8tY8GXo6SeWeS0nMZwPulw8dW8Nz3AbtK+mB6rvVSezXP17ohJxLrESJiGjATaI2IPwNzH4bhAAAAqklEQVRXAvdKmkV2Cao0OfwWGJW2Hwk8ko7zCvDXdHP7J+081fVkZeqvzbX9kOwy0UxJs9N6V/mnpGnAJbw/3/g4YEdJM8k+HHBUmX3vArZuu9kO/Bg4Kx1vpVcjImIh2fzev5M0gyyJQnXP17ohV/81a1CSJpOVw++yct9m1eARiZmZFeIRiZmZFeIRiZmZFeJEYmZmhTiRmJlZIU4kZmZWiBOJmZkV8v8B1rJEbn5jfh0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 重要特征\n",
    "feature_importance = clf.feature_importances_\n",
    "feature_importance = 100.0*(feature_importance/feature_importance.max())\n",
    "sorted_idx = np.argsort(feature_importance)\n",
    "pos = np.arange(sorted_idx.shape[0])+0.5\n",
    "plt.barh(pos, feature_importance[sorted_idx], align='center')\n",
    "plt.yticks(pos, boston.feature_names[sorted_idx])\n",
    "plt.xlabel('Relative Importance')\n",
    "plt.title('Variable Importance')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}