CLA06-NB01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decision Trees in Practice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment we will explore various techniques for preventing overfitting in decision trees. We will extend the implementation of the binary decision trees that we implemented in the previous assignment. You will have to use your solutions from this previous assignment and extend them.\n",
    "\n",
    "In this assignment you will:\n",
    "\n",
    "* Implement binary decision trees with different early stopping methods.\n",
    "* Compare models with different stopping parameters.\n",
    "* Visualize the concept of overfitting in decision trees.\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fire up Turi Create"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure you have the latest version of Turi Create."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import turicreate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load LendingClub Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This assignment will use the [LendingClub](https://www.lendingclub.com/) dataset used in the previous two assignments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "loans = turicreate.SFrame('lending-club-data.sframe/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, we reassign the labels to have +1 for a safe loan, and -1 for a risky (bad) loan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)\n",
    "loans = loans.remove_column('bad_loans')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the same 4 categorical features as in the previous assignment: \n",
    "1. grade of the loan \n",
    "2. the length of the loan term\n",
    "3. the home ownership status: own, mortgage, rent\n",
    "4. number of years of employment.\n",
    "\n",
    "In the dataset, each of these features is a categorical feature. Since we are building a binary decision tree, we will have to convert this to binary data in a subsequent section using 1-hot encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = ['grade',              # grade of the loan\n",
    "            'term',               # the term of the loan\n",
    "            'home_ownership',     # home_ownership status: own, mortgage or rent\n",
    "            'emp_length',         # number of years of employment\n",
    "           ]\n",
    "target = 'safe_loans'\n",
    "loans = loans[features + [target]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subsample dataset to make sure classes are balanced"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as we did in the previous assignment, we will undersample the larger class (safe loans) in order to balance out our dataset. This means we are throwing away many data points. We used `seed = 1` so everyone gets the same results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of safe loans                 : 0.5022361744216048\n",
      "Percentage of risky loans                : 0.4977638255783951\n",
      "Total number of loans in our new dataset : 46508\n"
     ]
    }
   ],
   "source": [
    "safe_loans_raw = loans[loans[target] == 1]\n",
    "risky_loans_raw = loans[loans[target] == -1]\n",
    "\n",
    "# Since there are less risky loans than safe loans, find the ratio of the sizes\n",
    "# and use that percentage to undersample the safe loans.\n",
    "percentage = len(risky_loans_raw)/float(len(safe_loans_raw))\n",
    "safe_loans = safe_loans_raw.sample(percentage, seed = 1)\n",
    "risky_loans = risky_loans_raw\n",
    "loans_data = risky_loans.append(safe_loans)\n",
    "\n",
    "print(\"Percentage of safe loans                 :\", len(safe_loans) / float(len(loans_data)))\n",
    "print(\"Percentage of risky loans                :\", len(risky_loans) / float(len(loans_data)))\n",
    "print(\"Total number of loans in our new dataset :\", len(loans_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this [paper](http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=5128907&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F69%2F5173046%2F05128907.pdf%3Farnumber%3D5128907 ). For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transform categorical data into binary features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are implementing **binary decision trees**, we transform our categorical data into binary data using 1-hot encoding, just as in the previous assignment. Here is the summary of that discussion:\n",
    "\n",
    "For instance, the **home_ownership** feature represents the home ownership status of the loanee, which is either `own`, `mortgage` or `rent`. For example, if a data point has the feature \n",
    "```\n",
    "   {'home_ownership': 'RENT'}\n",
    "```\n",
    "we want to turn this into three features: \n",
    "```\n",
    " { \n",
    "   'home_ownership = OWN'      : 0, \n",
    "   'home_ownership = MORTGAGE' : 0, \n",
    "   'home_ownership = RENT'     : 1\n",
    " }\n",
    "```\n",
    "\n",
    "Since this code requires a few Python and Turi Create tricks, feel free to use this block of code as is. Refer to the API documentation for a deeper understanding.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "loans_data = risky_loans.append(safe_loans)\n",
    "for feature in features:\n",
    "    loans_data_one_hot_encoded = loans_data[feature].apply(lambda x: {x: 1})    \n",
    "    loans_data_unpacked = loans_data_one_hot_encoded.unpack(column_name_prefix=feature)\n",
    "    \n",
    "    # Change None's to 0's\n",
    "    for column in loans_data_unpacked.column_names():\n",
    "        loans_data_unpacked[column] = loans_data_unpacked[column].fillna(0)\n",
    "\n",
    "    loans_data = loans_data.remove_column(feature)\n",
    "    loans_data = loans_data.add_columns(loans_data_unpacked)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feature columns now look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['grade.A',\n",
       " 'grade.B',\n",
       " 'grade.C',\n",
       " 'grade.D',\n",
       " 'grade.E',\n",
       " 'grade.F',\n",
       " 'grade.G',\n",
       " 'term. 36 months',\n",
       " 'term. 60 months',\n",
       " 'home_ownership.MORTGAGE',\n",
       " 'home_ownership.OTHER',\n",
       " 'home_ownership.OWN',\n",
       " 'home_ownership.RENT',\n",
       " 'emp_length.1 year',\n",
       " 'emp_length.10+ years',\n",
       " 'emp_length.2 years',\n",
       " 'emp_length.3 years',\n",
       " 'emp_length.4 years',\n",
       " 'emp_length.5 years',\n",
       " 'emp_length.6 years',\n",
       " 'emp_length.7 years',\n",
       " 'emp_length.8 years',\n",
       " 'emp_length.9 years',\n",
       " 'emp_length.< 1 year',\n",
       " 'emp_length.n/a']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = loans_data.column_names()\n",
    "features.remove('safe_loans')  # Remove the response variable\n",
    "features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train-Validation split\n",
    "\n",
    "We split the data into a train-validation split with 80% of the data in the training set and 20% of the data in the validation set. We use `seed=1` so that everyone gets the same result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, validation_set = loans_data.random_split(.8, seed=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Early stopping methods for decision trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will extend the **binary tree implementation** from the previous assignment in order to handle some early stopping conditions. Recall the 3 early stopping methods that were discussed in lecture:\n",
    "\n",
    "1. Reached a **maximum depth**. (set by parameter `max_depth`).\n",
    "2. Reached a **minimum node size**. (set by parameter `min_node_size`).\n",
    "3. Don't split if the **gain in error reduction** is too small. (set by parameter `min_error_reduction`).\n",
    "\n",
    "For the rest of this assignment, we will refer to these three as **early stopping conditions 1, 2, and 3**.\n",
    "\n",
    "## Early stopping condition 1: Maximum depth\n",
    "\n",
    "Recall that we already implemented the maximum depth stopping condition in the previous assignment. In this assignment, we will experiment with this condition a bit more and also write code to implement the 2nd and 3rd early stopping conditions.\n",
    "\n",
    "We will be reusing code from the previous assignment and then building upon this.  We will **alert you** when you reach a function that was part of the previous assignment so that you can simply copy and past your previous code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Early stopping condition 2: Minimum node size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function **reached_minimum_node_size** takes 2 arguments:\n",
    "\n",
    "1. The `data` (from a node)\n",
    "2. The minimum number of data points that a node is allowed to split on, `min_node_size`.\n",
    "\n",
    "This function simply calculates whether the number of data points at a given node is less than or equal to the specified minimum node size. This function will be used to detect this early stopping condition in the **decision_tree_create** function.\n",
    "\n",
    "Fill in the parts of the function below where you find `## YOUR CODE HERE`.  There is **one** instance in the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reached_minimum_node_size(data, min_node_size):\n",
    "    # Return True if the number of data points is less than or equal to the minimum node size.\n",
    "    ## YOUR CODE HERE\n",
    "    if len(data) <= min_node_size:\n",
    "        return True\n",
    "    return False\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Given an intermediate node with 6 safe loans and 3 risky loans, if the `min_node_size` parameter is 10, what should the tree learning algorithm do next?\n",
    "\n",
    "**A:** Stop, not more splitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Early stopping condition 3: Minimum gain in error reduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function **error_reduction** takes 2 arguments:\n",
    "\n",
    "1. The error **before** a split, `error_before_split`.\n",
    "2. The error **after** a split, `error_after_split`.\n",
    "\n",
    "This function computes the gain in error reduction, i.e., the difference between the error before the split and that after the split. This function will be used to detect this early stopping condition in the **decision_tree_create** function.\n",
    "\n",
    "Fill in the parts of the function below where you find `## YOUR CODE HERE`.  There is **one** instance in the function below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def error_reduction(error_before_split, error_after_split):\n",
    "    # Return the error before the split minus the error after the split.\n",
    "    ## YOUR CODE HERE\n",
    "    return error_before_split - error_after_split\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Assume an intermediate node has 6 safe loans and 3 risky loans.  For each of 4 possible features to split on, the error reduction is 0.0, 0.05, 0.1, and 0.14, respectively. If the **minimum gain in error reduction** parameter is set to 0.2, what should the tree learning algorithm do next?\n",
    "\n",
    "**A:** \n",
    "\n",
    "| feature | error |\n",
    "| -  | ---- |\n",
    "| x1 | 0.0  |\n",
    "| x2 | 0.05 |\n",
    "| x3 | 0.1  |\n",
    "| x4 | 0.14 |\n",
    "\n",
    "Stop, since splitting on any node would have < minimum gain. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grabbing binary decision tree helper functions from past assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall from the previous assignment that we wrote a function `intermediate_node_num_mistakes` that calculates the number of **misclassified examples** when predicting the **majority class**. This is used to help determine which feature is best to split on at a given node of the tree.\n",
    "\n",
    "**Please copy and paste your code for `intermediate_node_num_mistakes` here**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def intermediate_node_num_mistakes(labels_in_node):\n",
    "    # Corner case: If labels_in_node is empty, return 0\n",
    "    if len(labels_in_node) == 0:\n",
    "        return 0\n",
    "    \n",
    "    # Count the number of 1's (safe loans)\n",
    "    ## YOUR CODE HERE\n",
    "    countPos = len(labels_in_node.filter(lambda x: x == 1))\n",
    "    \n",
    "    # Count the number of -1's (risky loans)\n",
    "    ## YOUR CODE HERE\n",
    "    countNeg = len(labels_in_node.filter(lambda x: x == -1))\n",
    "    \n",
    "    if countPos >= countNeg:\n",
    "        result = countNeg\n",
    "    else:    \n",
    "        result = countPos\n",
    "\n",
    "    #print(\"labels_in_node: \", labels_in_node)\n",
    "    #print(\"countPos: \", countPos)\n",
    "    #print(\"countNeg: \", countNeg)\n",
    "    #print(\"result: \", result)\n",
    "    \n",
    "    return result   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then wrote a function `best_splitting_feature` that finds the best feature to split on given the data and a list of features to consider.\n",
    "\n",
    "**Please copy and paste your `best_splitting_feature` code here**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def best_splitting_feature(data, features, target):\n",
    "    \n",
    "    best_feature = None # Keep track of the best feature \n",
    "    best_error = 10     # Keep track of the best error so far \n",
    "    # Note: Since error is always <= 1, we should intialize it with something larger than 1.\n",
    "\n",
    "    # Convert to float to make sure error gets computed correctly.\n",
    "    num_data_points = float(len(data))  \n",
    "    \n",
    "    # Loop through each feature to consider splitting on that feature\n",
    "    for feature in features:\n",
    "        \n",
    "        # The left split will have all data points where the feature value is 0\n",
    "        left_split = data[data[feature] == 0]\n",
    "        \n",
    "        # The right split will have all data points where the feature value is 1\n",
    "        ## YOUR CODE HERE\n",
    "        right_split =  data[data[feature] == 1]\n",
    "            \n",
    "        # Calculate the number of misclassified examples in the left split.\n",
    "        # Remember that we implemented a function for this! (It was called intermediate_node_num_mistakes)\n",
    "        # YOUR CODE HERE\n",
    "        left_mistakes = intermediate_node_num_mistakes(left_split[target])            \n",
    "\n",
    "        # Calculate the number of misclassified examples in the right split.\n",
    "        ## YOUR CODE HERE\n",
    "        right_mistakes = intermediate_node_num_mistakes(right_split[target]) \n",
    "            \n",
    "        # Compute the classification error of this split.\n",
    "        # Error = (# of mistakes (left) + # of mistakes (right)) / (# of data points)\n",
    "        ## YOUR CODE HERE\n",
    "        error = (left_mistakes + right_mistakes) / len(data)\n",
    "        \n",
    "        #print(\"left_mistakes\\t\", left_mistakes, \" right_mistakes\\t\", right_mistakes, \" error\\t\", error, \" best_feature\\t\", best_feature)\n",
    "\n",
    "        # If this is the best error we have found so far, store the feature as best_feature and the error as best_error\n",
    "        ## YOUR CODE HERE\n",
    "        if error < best_error:\n",
    "            best_feature = feature\n",
    "            best_error = error\n",
    "        \n",
    "    \n",
    "    return best_feature # Return the best feature we found"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, recall the function `create_leaf` from the previous assignment, which creates a leaf node given a set of target values.  \n",
    "\n",
    "**Please copy and paste your `create_leaf` code here**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_leaf(target_values):\n",
    "    \n",
    "    # Create a leaf node\n",
    "    leaf = {\n",
    "        'splitting_feature' : None,\n",
    "        'left' : None,\n",
    "        'right' : None,\n",
    "        'is_leaf': True    \n",
    "    }   ## YOUR CODE HERE\n",
    "    \n",
    "    # Count the number of data points that are +1 and -1 in this node.\n",
    "    num_ones = len(target_values[target_values == +1])\n",
    "    num_minus_ones = len(target_values[target_values == -1])\n",
    "    \n",
    "    # For the leaf node, set the prediction to be the majority class.\n",
    "    # Store the predicted class (1 or -1) in leaf['prediction']\n",
    "    if num_ones > num_minus_ones:\n",
    "        leaf['prediction'] = +1         ## YOUR CODE HERE\n",
    "    else:\n",
    "        leaf['prediction'] = -1         ## YOUR CODE HERE\n",
    "        \n",
    "    # Return the leaf node        \n",
    "    return leaf "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Incorporating new early stopping conditions in binary decision tree implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you will implement a function that builds a decision tree handling the three early stopping conditions described in this assignment.  In particular, you will write code to detect early stopping conditions 2 and 3.  You implemented above the functions needed to detect these conditions.  The 1st early stopping condition, **max_depth**, was implemented in the previous assigment and you will not need to reimplement this.  In addition to these early stopping conditions, the typical stopping conditions of having no mistakes or no more features to split on (which we denote by \"stopping conditions\" 1 and 2) are also included as in the previous assignment.\n",
    "\n",
    "**Implementing early stopping condition 2: minimum node size:**\n",
    "\n",
    "* **Step 1:** Use the function **reached_minimum_node_size** that you implemented earlier to write an if condition to detect whether we have hit the base case, i.e., the node does not have enough data points and should be turned into a leaf. Don't forget to use the `min_node_size` argument.\n",
    "* **Step 2:** Return a leaf. This line of code should be the same as the other (pre-implemented) stopping conditions.\n",
    "\n",
    "\n",
    "**Implementing early stopping condition 3: minimum error reduction:**\n",
    "\n",
    "**Note:** This has to come after finding the best splitting feature so we can calculate the error after splitting in order to calculate the error reduction.\n",
    "\n",
    "* **Step 1:** Calculate the **classification error before splitting**.  Recall that classification error is defined as:\n",
    "\n",
    "$$\n",
    "\\text{classification error} = \\frac{\\text{# mistakes}}{\\text{# total examples}}\n",
    "$$\n",
    "* **Step 2:** Calculate the **classification error after splitting**. This requires calculating the number of mistakes in the left and right splits, and then dividing by the total number of examples.\n",
    "* **Step 3:** Use the function **error_reduction** to that you implemented earlier to write an if condition to detect whether  the reduction in error is less than the constant provided (`min_error_reduction`). Don't forget to use that argument.\n",
    "* **Step 4:** Return a leaf. This line of code should be the same as the other (pre-implemented) stopping conditions.\n",
    "\n",
    "Fill in the places where you find `## YOUR CODE HERE`. There are **seven** places in this function for you to fill in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decision_tree_create(data, features, target, current_depth = 0, \n",
    "                         max_depth = 10, min_node_size=1, \n",
    "                         min_error_reduction=0.0):\n",
    "    \n",
    "    remaining_features = features[:] # Make a copy of the features.\n",
    "    \n",
    "    target_values = data[target]\n",
    "    print(\"--------------------------------------------------------------------\")\n",
    "    print(\"Subtree, depth = %s (%s data points).\" % (current_depth, len(target_values)))\n",
    "\n",
    "    \n",
    "    # Stopping condition 1: All nodes are of the same type.\n",
    "    if intermediate_node_num_mistakes(target_values) == 0:\n",
    "        print(\"Stopping condition 1 reached. All data points have the same target value.\")\n",
    "        return create_leaf(target_values)\n",
    "\n",
    "    # Stopping condition 2: No more features to split on.\n",
    "    if remaining_features == []:\n",
    "        print(\"Stopping condition 2 reached. No remaining features.\")\n",
    "        return create_leaf(target_values)\n",
    "    \n",
    "    # Early stopping condition 1: Reached max depth limit.\n",
    "    if current_depth >= max_depth:\n",
    "        print(\"Early stopping condition 1 reached. Reached maximum depth.\")\n",
    "        return create_leaf(target_values)\n",
    "    \n",
    "    # Early stopping condition 2: Reached the minimum node size.\n",
    "    # If the number of data points is less than or equal to the minimum size, return a leaf.\n",
    "    if len(data) <= min_node_size:\n",
    "        print(\"Early stopping condition 2 reached. Reached minimum node size.\")\n",
    "        return create_leaf(target_values)\n",
    "    \n",
    "    # Find the best splitting feature\n",
    "    splitting_feature = best_splitting_feature(data, features, target)\n",
    "    \n",
    "    # Split on the best feature that we found. \n",
    "    left_split = data[data[splitting_feature] == 0]\n",
    "    right_split = data[data[splitting_feature] == 1]\n",
    "    \n",
    "    # Early stopping condition 3: Minimum error reduction\n",
    "    # Calculate the error before splitting (number of misclassified examples \n",
    "    # divided by the total number of examples)\n",
    "    error_before_split = intermediate_node_num_mistakes(target_values) / float(len(data))\n",
    "    \n",
    "    # Calculate the error after splitting (number of misclassified examples \n",
    "    # in both groups divided by the total number of examples)\n",
    "    left_mistakes =  intermediate_node_num_mistakes(left_split[target]) ### CHECK THIS\n",
    "    right_mistakes = intermediate_node_num_mistakes(right_split[target]) ### CHECK THIS\n",
    "    error_after_split = (left_mistakes + right_mistakes) / float(len(data))\n",
    "    \n",
    "    # If the error reduction is LESS THAN OR EQUAL TO min_error_reduction, return a leaf.\n",
    "    if error_after_split <= min_error_reduction:\n",
    "        print(\"Early stopping condition 3 reached. Minimum error reduction.\")\n",
    "        return create_leaf(target_values)\n",
    "    \n",
    "    \n",
    "    remaining_features.remove(splitting_feature)\n",
    "    print(\"Split on feature %s. (%s, %s)\" % (\\\n",
    "                      splitting_feature, len(left_split), len(right_split)))\n",
    "\n",
    "    \n",
    "    # Repeat (recurse) on left and right subtrees\n",
    "    left_tree = decision_tree_create(left_split, remaining_features, target, \n",
    "                                     current_depth + 1, max_depth, min_node_size, min_error_reduction)        \n",
    "    \n",
    "    ## YOUR CODE HERE\n",
    "    right_tree = decision_tree_create(right_split, remaining_features, target, \n",
    "                                     current_depth + 1, max_depth, min_node_size, min_error_reduction)        \n",
    "    \n",
    "    \n",
    "    return {'is_leaf'          : False, \n",
    "            'prediction'       : None,\n",
    "            'splitting_feature': splitting_feature,\n",
    "            'left'             : left_tree, \n",
    "            'right'            : right_tree}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a function to count the nodes in your tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_nodes(tree):\n",
    "    if tree['is_leaf']:\n",
    "        return 1\n",
    "    return 1 + count_nodes(tree['left']) + count_nodes(tree['right'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following test code to check your implementation. Make sure you get **'Test passed'** before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (9223 data points).\n",
      "Split on feature grade.A. (9122, 101)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9122 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (101 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (28001 data points).\n",
      "Split on feature grade.D. (23300, 4701)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (23300 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (4701 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "Test passed!\n"
     ]
    }
   ],
   "source": [
    "small_decision_tree = decision_tree_create(train_data, features, 'safe_loans', max_depth = 2, \n",
    "                                        min_node_size = 10, min_error_reduction=0.0)\n",
    "if count_nodes(small_decision_tree) == 7:\n",
    "    print('Test passed!')\n",
    "else:\n",
    "    print('Test failed... try again!')\n",
    "    print('Number of nodes found                :', count_nodes(small_decision_tree))\n",
    "    print('Number of nodes that should be there : 7' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a tree!\n",
    "\n",
    "Now that your code is working, we will train a tree model on the **train_data** with\n",
    "* `max_depth = 6`\n",
    "* `min_node_size = 100`, \n",
    "* `min_error_reduction = 0.0`\n",
    "\n",
    "**Warning**: This code block may take a minute to learn. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (9223 data points).\n",
      "Split on feature grade.A. (9122, 101)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9122 data points).\n",
      "Split on feature grade.B. (8074, 1048)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (8074 data points).\n",
      "Split on feature grade.C. (5884, 2190)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5884 data points).\n",
      "Split on feature grade.D. (3826, 2058)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (3826 data points).\n",
      "Split on feature grade.E. (1693, 2133)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1693 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2133 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2058 data points).\n",
      "Split on feature grade.E. (2058, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2058 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (2190 data points).\n",
      "Split on feature grade.D. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2190 data points).\n",
      "Split on feature grade.E. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2190 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1048 data points).\n",
      "Split on feature emp_length.5 years. (969, 79)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (969 data points).\n",
      "Split on feature grade.C. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (969 data points).\n",
      "Split on feature grade.D. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (969 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (79 data points).\n",
      "Early stopping condition 2 reached. Reached minimum node size.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (101 data points).\n",
      "Split on feature emp_length.n/a. (96, 5)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (96 data points).\n",
      "Early stopping condition 2 reached. Reached minimum node size.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (5 data points).\n",
      "Early stopping condition 2 reached. Reached minimum node size.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (28001 data points).\n",
      "Split on feature grade.D. (23300, 4701)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (23300 data points).\n",
      "Split on feature grade.E. (22024, 1276)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (22024 data points).\n",
      "Split on feature grade.F. (21666, 358)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (21666 data points).\n",
      "Split on feature emp_length.n/a. (20734, 932)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (20734 data points).\n",
      "Split on feature grade.G. (20638, 96)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (20638 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (96 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (932 data points).\n",
      "Split on feature grade.A. (702, 230)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (702 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (230 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (358 data points).\n",
      "Split on feature emp_length.8 years. (347, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (347 data points).\n",
      "Split on feature grade.A. (347, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (347 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Early stopping condition 2 reached. Reached minimum node size.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1276 data points).\n",
      "Split on feature grade.A. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (1276 data points).\n",
      "Split on feature grade.B. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (1276 data points).\n",
      "Split on feature grade.C. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1276 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (4701 data points).\n",
      "Split on feature grade.A. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (4701 data points).\n",
      "Split on feature grade.B. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (4701 data points).\n",
      "Split on feature grade.C. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (4701 data points).\n",
      "Split on feature grade.E. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (4701 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n"
     ]
    }
   ],
   "source": [
    "my_decision_tree_new = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6, \n",
    "                                min_node_size = 100, min_error_reduction=0.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now train a tree model **ignoring early stopping conditions 2 and 3** so that we get the same tree as in the previous assignment.  To ignore these conditions, we set `min_node_size=0` and `min_error_reduction=-1` (a negative value)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (9223 data points).\n",
      "Split on feature grade.A. (9122, 101)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9122 data points).\n",
      "Split on feature grade.B. (8074, 1048)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (8074 data points).\n",
      "Split on feature grade.C. (5884, 2190)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5884 data points).\n",
      "Split on feature grade.D. (3826, 2058)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (3826 data points).\n",
      "Split on feature grade.E. (1693, 2133)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1693 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2133 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2058 data points).\n",
      "Split on feature grade.E. (2058, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2058 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (2190 data points).\n",
      "Split on feature grade.D. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2190 data points).\n",
      "Split on feature grade.E. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2190 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1048 data points).\n",
      "Split on feature emp_length.5 years. (969, 79)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (969 data points).\n",
      "Split on feature grade.C. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (969 data points).\n",
      "Split on feature grade.D. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (969 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (79 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (34, 45)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (34 data points).\n",
      "Split on feature grade.C. (34, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (34 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (45 data points).\n",
      "Split on feature grade.C. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (45 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (101 data points).\n",
      "Split on feature emp_length.n/a. (96, 5)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (96 data points).\n",
      "Split on feature emp_length.< 1 year. (85, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (85 data points).\n",
      "Split on feature grade.B. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (85 data points).\n",
      "Split on feature grade.C. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (85 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (11 data points).\n",
      "Split on feature grade.B. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature grade.C. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (11 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (5 data points).\n",
      "Split on feature grade.B. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5 data points).\n",
      "Split on feature grade.C. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (5 data points).\n",
      "Split on feature grade.D. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (5 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (28001 data points).\n",
      "Split on feature grade.D. (23300, 4701)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (23300 data points).\n",
      "Split on feature grade.E. (22024, 1276)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (22024 data points).\n",
      "Split on feature grade.F. (21666, 358)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (21666 data points).\n",
      "Split on feature emp_length.n/a. (20734, 932)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (20734 data points).\n",
      "Split on feature grade.G. (20638, 96)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (20638 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (96 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (932 data points).\n",
      "Split on feature grade.A. (702, 230)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (702 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (230 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (358 data points).\n",
      "Split on feature emp_length.8 years. (347, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (347 data points).\n",
      "Split on feature grade.A. (347, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (347 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature home_ownership.OWN. (9, 2)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (9 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1276 data points).\n",
      "Split on feature grade.A. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (1276 data points).\n",
      "Split on feature grade.B. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (1276 data points).\n",
      "Split on feature grade.C. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1276 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (4701 data points).\n",
      "Split on feature grade.A. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (4701 data points).\n",
      "Split on feature grade.B. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (4701 data points).\n",
      "Split on feature grade.C. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (4701 data points).\n",
      "Split on feature grade.E. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (4701 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n"
     ]
    }
   ],
   "source": [
    "my_decision_tree_old = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6, \n",
    "                                min_node_size = 0, min_error_reduction=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that in the previous assignment you implemented a function `classify` to classify a new point `x` using a given `tree`.\n",
    "\n",
    "**Please copy and paste your `classify` code here**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify(tree, x, annotate = False):   \n",
    "    # if the node is a leaf node.\n",
    "    if tree['is_leaf']:\n",
    "        if annotate: \n",
    "            print(\"At leaf, predicting %s\" % tree['prediction'])\n",
    "        return tree['prediction'] \n",
    "    else:\n",
    "        # split on feature.\n",
    "        split_feature_value = x[tree['splitting_feature']]\n",
    "        if annotate: \n",
    "            print(\"Split on %s = %s\" % (tree['splitting_feature'], split_feature_value))\n",
    "            \n",
    "        if split_feature_value == 0:\n",
    "            return classify(tree['left'], x, annotate)\n",
    "        else:\n",
    "            ### YOUR CODE HERE\n",
    "            return classify(tree['right'], x, annotate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's consider the first example of the validation set and see what the `my_decision_tree_new` model predicts for this data point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'safe_loans': -1,\n",
       " 'grade.A': 0,\n",
       " 'grade.B': 0,\n",
       " 'grade.C': 0,\n",
       " 'grade.D': 1,\n",
       " 'grade.E': 0,\n",
       " 'grade.F': 0,\n",
       " 'grade.G': 0,\n",
       " 'term. 36 months': 0,\n",
       " 'term. 60 months': 1,\n",
       " 'home_ownership.MORTGAGE': 0,\n",
       " 'home_ownership.OTHER': 0,\n",
       " 'home_ownership.OWN': 0,\n",
       " 'home_ownership.RENT': 1,\n",
       " 'emp_length.1 year': 0,\n",
       " 'emp_length.10+ years': 0,\n",
       " 'emp_length.2 years': 1,\n",
       " 'emp_length.3 years': 0,\n",
       " 'emp_length.4 years': 0,\n",
       " 'emp_length.5 years': 0,\n",
       " 'emp_length.6 years': 0,\n",
       " 'emp_length.7 years': 0,\n",
       " 'emp_length.8 years': 0,\n",
       " 'emp_length.9 years': 0,\n",
       " 'emp_length.< 1 year': 0,\n",
       " 'emp_length.n/a': 0}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validation_set[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted class: -1 \n"
     ]
    }
   ],
   "source": [
    "print('Predicted class: %s ' % classify(my_decision_tree_new, validation_set[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add some annotations to our prediction to see what the prediction path was that lead to this predicted class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Split on term. 36 months = 0\n",
      "Split on grade.A = 0\n",
      "Split on grade.B = 0\n",
      "Split on grade.C = 0\n",
      "Split on grade.D = 1\n",
      "Split on grade.E = 0\n",
      "At leaf, predicting -1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "-1"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classify(my_decision_tree_new, validation_set[0], annotate = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now recall the prediction path for the decision tree learned in the previous assignment, which we recreated here as `my_decision_tree_old`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Split on term. 36 months = 0\n",
      "Split on grade.A = 0\n",
      "Split on grade.B = 0\n",
      "Split on grade.C = 0\n",
      "Split on grade.D = 1\n",
      "Split on grade.E = 0\n",
      "At leaf, predicting -1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "-1"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classify(my_decision_tree_old, validation_set[0], annotate = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** For `my_decision_tree_new` trained with `max_depth = 6`, `min_node_size = 100`, `min_error_reduction=0.0`, is the prediction path for `validation_set[0]` shorter, longer, or the same as for `my_decision_tree_old` that ignored the early stopping conditions 2 and 3?\n",
    "\n",
    "**A:** Same"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** For `my_decision_tree_new` trained with `max_depth = 6`, `min_node_size = 100`, `min_error_reduction=0.0`, is the prediction path for **any point** always shorter, always longer, always the same, shorter or the same, or longer or the same as for `my_decision_tree_old` that ignored the early stopping conditions 2 and 3?\n",
    "\n",
    "**A:** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Split on term. 36 months = 1\n",
      "Split on grade.D = 0\n",
      "Split on grade.E = 0\n",
      "Split on grade.F = 0\n",
      "Split on emp_length.n/a = 0\n",
      "Split on grade.G = 0\n",
      "At leaf, predicting 1\n",
      "Split on term. 36 months = 1\n",
      "Split on grade.D = 0\n",
      "Split on grade.E = 0\n",
      "Split on grade.F = 0\n",
      "Split on emp_length.n/a = 0\n",
      "Split on grade.G = 0\n",
      "At leaf, predicting 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classify(my_decision_tree_new, validation_set[5000], annotate = True)\n",
    "classify(my_decision_tree_old, validation_set[5000], annotate = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** For a tree trained on **any** dataset using `max_depth = 6`, `min_node_size = 100`, `min_error_reduction=0.0`, what is the maximum number of splits encountered while making a single prediction?\n",
    "\n",
    "**A:** 6 is the max from root to lowest leaf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us evaluate the model that we have trained. You implemented this evaluation in the function `evaluate_classification_error` from the previous assignment.\n",
    "\n",
    "**Please copy and paste your `evaluate_classification_error` code here**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_classification_error(tree, data, target):\n",
    "    # Apply the classify(tree, x) to each row in your data\n",
    "    predictions = data.apply(lambda x: classify(tree, x))\n",
    "    \n",
    "    # Once you've made the predictions, calculate the classification error and return it\n",
    "    ## YOUR CODE HERE\n",
    "    mistakes = 0\n",
    "    for i, p in enumerate(predictions):\n",
    "        if p != data[i][target]:\n",
    "            mistakes += 1\n",
    "    #predictions['actual'] = data[target]\n",
    "    #mistakes = 0 # predictions.filter(,)\n",
    "    return mistakes / len(data)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's use this function to evaluate the classification error of `my_decision_tree_new` on the **validation_set**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.38410168031021114"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_classification_error(my_decision_tree_new, validation_set, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, evaluate the validation error using `my_decision_tree_old`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3837785437311504"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_classification_error(my_decision_tree_old, validation_set, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Is the validation error of the new decision tree (using early stopping conditions 2 and 3) lower than, higher than, or the same as that of the old decision tree from the previous assignment?\n",
    "\n",
    "**A:** Higher"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring the effect of max_depth\n",
    "\n",
    "We will compare three models trained with different values of the stopping criterion. We intentionally picked models at the extreme ends (**too small**, **just right**, and **too large**).\n",
    "\n",
    "Train three models with these parameters:\n",
    "\n",
    "1. **model_1**: max_depth = 2 (too small)\n",
    "2. **model_2**: max_depth = 6 (just right)\n",
    "3. **model_3**: max_depth = 14 (may be too large)\n",
    "\n",
    "For each of these three, we set `min_node_size = 0` and `min_error_reduction = -1`.\n",
    "\n",
    "** Note:** Each tree can take up to a few minutes to train. In particular, `model_3` will probably take the longest to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (9223 data points).\n",
      "Split on feature grade.A. (9122, 101)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9122 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (101 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (28001 data points).\n",
      "Split on feature grade.D. (23300, 4701)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (23300 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (4701 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (9223 data points).\n",
      "Split on feature grade.A. (9122, 101)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9122 data points).\n",
      "Split on feature grade.B. (8074, 1048)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (8074 data points).\n",
      "Split on feature grade.C. (5884, 2190)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5884 data points).\n",
      "Split on feature grade.D. (3826, 2058)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (3826 data points).\n",
      "Split on feature grade.E. (1693, 2133)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1693 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2133 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2058 data points).\n",
      "Split on feature grade.E. (2058, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2058 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (2190 data points).\n",
      "Split on feature grade.D. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2190 data points).\n",
      "Split on feature grade.E. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2190 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1048 data points).\n",
      "Split on feature emp_length.5 years. (969, 79)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (969 data points).\n",
      "Split on feature grade.C. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (969 data points).\n",
      "Split on feature grade.D. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (969 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (79 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (34, 45)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (34 data points).\n",
      "Split on feature grade.C. (34, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (34 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (45 data points).\n",
      "Split on feature grade.C. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (45 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (101 data points).\n",
      "Split on feature emp_length.n/a. (96, 5)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (96 data points).\n",
      "Split on feature emp_length.< 1 year. (85, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (85 data points).\n",
      "Split on feature grade.B. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (85 data points).\n",
      "Split on feature grade.C. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (85 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (11 data points).\n",
      "Split on feature grade.B. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature grade.C. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (11 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (5 data points).\n",
      "Split on feature grade.B. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5 data points).\n",
      "Split on feature grade.C. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (5 data points).\n",
      "Split on feature grade.D. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (5 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (28001 data points).\n",
      "Split on feature grade.D. (23300, 4701)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (23300 data points).\n",
      "Split on feature grade.E. (22024, 1276)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (22024 data points).\n",
      "Split on feature grade.F. (21666, 358)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (21666 data points).\n",
      "Split on feature emp_length.n/a. (20734, 932)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (20734 data points).\n",
      "Split on feature grade.G. (20638, 96)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (20638 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (96 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (932 data points).\n",
      "Split on feature grade.A. (702, 230)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (702 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (230 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (358 data points).\n",
      "Split on feature emp_length.8 years. (347, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (347 data points).\n",
      "Split on feature grade.A. (347, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (347 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature home_ownership.OWN. (9, 2)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (9 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1276 data points).\n",
      "Split on feature grade.A. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (1276 data points).\n",
      "Split on feature grade.B. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (1276 data points).\n",
      "Split on feature grade.C. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1276 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (4701 data points).\n",
      "Split on feature grade.A. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (4701 data points).\n",
      "Split on feature grade.B. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (4701 data points).\n",
      "Split on feature grade.C. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (4701 data points).\n",
      "Split on feature grade.E. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (4701 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (9223 data points).\n",
      "Split on feature grade.A. (9122, 101)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9122 data points).\n",
      "Split on feature grade.B. (8074, 1048)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (8074 data points).\n",
      "Split on feature grade.C. (5884, 2190)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5884 data points).\n",
      "Split on feature grade.D. (3826, 2058)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (3826 data points).\n",
      "Split on feature grade.E. (1693, 2133)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1693 data points).\n",
      "Split on feature home_ownership.OTHER. (1692, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (1692 data points).\n",
      "Split on feature grade.F. (339, 1353)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (339 data points).\n",
      "Split on feature grade.G. (0, 339)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (339 data points).\n",
      "Split on feature term. 60 months. (0, 339)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (339 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (175, 164)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (175 data points).\n",
      "Split on feature home_ownership.OWN. (142, 33)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (142 data points).\n",
      "Split on feature emp_length.6 years. (133, 9)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (133 data points).\n",
      "Split on feature home_ownership.RENT. (0, 133)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (133 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (9 data points).\n",
      "Split on feature home_ownership.RENT. (0, 9)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (9 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (33 data points).\n",
      "Split on feature emp_length.n/a. (31, 2)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (31 data points).\n",
      "Split on feature emp_length.2 years. (30, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (30 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (2 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (164 data points).\n",
      "Split on feature emp_length.2 years. (159, 5)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (159 data points).\n",
      "Split on feature emp_length.3 years. (148, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (148 data points).\n",
      "Split on feature home_ownership.OWN. (148, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (148 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (11 data points).\n",
      "Split on feature home_ownership.OWN. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (11 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (5 data points).\n",
      "Split on feature home_ownership.OWN. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (5 data points).\n",
      "Split on feature home_ownership.RENT. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (5 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (1353 data points).\n",
      "Split on feature grade.G. (1353, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (1353 data points).\n",
      "Split on feature term. 60 months. (0, 1353)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (1353 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (710, 643)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (710 data points).\n",
      "Split on feature home_ownership.OWN. (602, 108)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (602 data points).\n",
      "Split on feature home_ownership.RENT. (0, 602)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (602 data points).\n",
      "Split on feature emp_length.1 year. (565, 37)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (565 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (37 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (108 data points).\n",
      "Split on feature home_ownership.RENT. (108, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (108 data points).\n",
      "Split on feature emp_length.1 year. (100, 8)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (100 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (8 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (643 data points).\n",
      "Split on feature home_ownership.OWN. (643, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (643 data points).\n",
      "Split on feature home_ownership.RENT. (643, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (643 data points).\n",
      "Split on feature emp_length.1 year. (602, 41)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (602 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (41 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2133 data points).\n",
      "Split on feature grade.F. (2133, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (2133 data points).\n",
      "Split on feature grade.G. (2133, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (2133 data points).\n",
      "Split on feature term. 60 months. (0, 2133)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (2133 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (1045, 1088)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (1045 data points).\n",
      "Split on feature home_ownership.OTHER. (1044, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (1044 data points).\n",
      "Split on feature home_ownership.OWN. (879, 165)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (879 data points).\n",
      "Split on feature home_ownership.RENT. (0, 879)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (879 data points).\n",
      "Split on feature emp_length.1 year. (809, 70)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (809 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (70 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (165 data points).\n",
      "Split on feature emp_length.9 years. (157, 8)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (157 data points).\n",
      "Split on feature home_ownership.RENT. (157, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (157 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (8 data points).\n",
      "Split on feature home_ownership.RENT. (8, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (8 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (1088 data points).\n",
      "Split on feature home_ownership.OTHER. (1088, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (1088 data points).\n",
      "Split on feature home_ownership.OWN. (1088, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (1088 data points).\n",
      "Split on feature home_ownership.RENT. (1088, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (1088 data points).\n",
      "Split on feature emp_length.1 year. (1035, 53)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (1035 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (53 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2058 data points).\n",
      "Split on feature grade.E. (2058, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2058 data points).\n",
      "Split on feature grade.F. (2058, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (2058 data points).\n",
      "Split on feature grade.G. (2058, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (2058 data points).\n",
      "Split on feature term. 60 months. (0, 2058)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (2058 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (923, 1135)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (923 data points).\n",
      "Split on feature home_ownership.OTHER. (922, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (922 data points).\n",
      "Split on feature home_ownership.OWN. (762, 160)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (762 data points).\n",
      "Split on feature home_ownership.RENT. (0, 762)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (762 data points).\n",
      "Split on feature emp_length.1 year. (704, 58)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (704 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (58 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (160 data points).\n",
      "Split on feature home_ownership.RENT. (160, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (160 data points).\n",
      "Split on feature emp_length.1 year. (154, 6)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (154 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (6 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (1135 data points).\n",
      "Split on feature home_ownership.OTHER. (1135, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (1135 data points).\n",
      "Split on feature home_ownership.OWN. (1135, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (1135 data points).\n",
      "Split on feature home_ownership.RENT. (1135, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (1135 data points).\n",
      "Split on feature emp_length.1 year. (1096, 39)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (1096 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (39 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (2190 data points).\n",
      "Split on feature grade.D. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2190 data points).\n",
      "Split on feature grade.E. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2190 data points).\n",
      "Split on feature grade.F. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (2190 data points).\n",
      "Split on feature grade.G. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (2190 data points).\n",
      "Split on feature term. 60 months. (0, 2190)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (2190 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (803, 1387)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (803 data points).\n",
      "Split on feature emp_length.4 years. (746, 57)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (746 data points).\n",
      "Split on feature home_ownership.OTHER. (746, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (746 data points).\n",
      "Split on feature home_ownership.OWN. (598, 148)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (598 data points).\n",
      "Split on feature home_ownership.RENT. (0, 598)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (598 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (148 data points).\n",
      "Split on feature emp_length.< 1 year. (137, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (137 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (11 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (57 data points).\n",
      "Split on feature home_ownership.OTHER. (57, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (57 data points).\n",
      "Split on feature home_ownership.OWN. (49, 8)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (49 data points).\n",
      "Split on feature home_ownership.RENT. (0, 49)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (49 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (8 data points).\n",
      "Split on feature home_ownership.RENT. (8, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (8 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (1387 data points).\n",
      "Split on feature emp_length.6 years. (1313, 74)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (1313 data points).\n",
      "Split on feature home_ownership.OTHER. (1313, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (1313 data points).\n",
      "Split on feature home_ownership.OWN. (1313, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (1313 data points).\n",
      "Split on feature home_ownership.RENT. (1313, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (1313 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (74 data points).\n",
      "Split on feature home_ownership.OTHER. (74, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (74 data points).\n",
      "Split on feature home_ownership.OWN. (74, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (74 data points).\n",
      "Split on feature home_ownership.RENT. (74, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (74 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1048 data points).\n",
      "Split on feature emp_length.5 years. (969, 79)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (969 data points).\n",
      "Split on feature grade.C. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (969 data points).\n",
      "Split on feature grade.D. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (969 data points).\n",
      "Split on feature grade.E. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (969 data points).\n",
      "Split on feature grade.F. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (969 data points).\n",
      "Split on feature grade.G. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (969 data points).\n",
      "Split on feature term. 60 months. (0, 969)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (969 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (367, 602)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (367 data points).\n",
      "Split on feature home_ownership.OTHER. (367, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (367 data points).\n",
      "Split on feature home_ownership.OWN. (291, 76)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (291 data points).\n",
      "Split on feature home_ownership.RENT. (0, 291)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (291 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (76 data points).\n",
      "Split on feature emp_length.9 years. (71, 5)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (71 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (5 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (602 data points).\n",
      "Split on feature emp_length.9 years. (580, 22)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (580 data points).\n",
      "Split on feature emp_length.3 years. (545, 35)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (545 data points).\n",
      "Split on feature emp_length.4 years. (506, 39)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (506 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (39 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (35 data points).\n",
      "Split on feature home_ownership.OTHER. (35, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (35 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (22 data points).\n",
      "Split on feature home_ownership.OTHER. (22, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (22 data points).\n",
      "Split on feature home_ownership.OWN. (22, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (22 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (79 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (34, 45)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (34 data points).\n",
      "Split on feature grade.C. (34, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (34 data points).\n",
      "Split on feature grade.D. (34, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (34 data points).\n",
      "Split on feature grade.E. (34, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (34 data points).\n",
      "Split on feature grade.F. (34, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (34 data points).\n",
      "Split on feature grade.G. (34, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (34 data points).\n",
      "Split on feature term. 60 months. (0, 34)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (34 data points).\n",
      "Split on feature home_ownership.OTHER. (34, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (34 data points).\n",
      "Split on feature home_ownership.OWN. (25, 9)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (25 data points).\n",
      "Split on feature home_ownership.RENT. (0, 25)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (25 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (9 data points).\n",
      "Split on feature home_ownership.RENT. (9, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (9 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (45 data points).\n",
      "Split on feature grade.C. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (45 data points).\n",
      "Split on feature grade.D. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (45 data points).\n",
      "Split on feature grade.E. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (45 data points).\n",
      "Split on feature grade.F. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (45 data points).\n",
      "Split on feature grade.G. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (45 data points).\n",
      "Split on feature term. 60 months. (0, 45)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (45 data points).\n",
      "Split on feature home_ownership.OTHER. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (45 data points).\n",
      "Split on feature home_ownership.OWN. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (45 data points).\n",
      "Split on feature home_ownership.RENT. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (45 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (101 data points).\n",
      "Split on feature emp_length.n/a. (96, 5)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (96 data points).\n",
      "Split on feature emp_length.< 1 year. (85, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (85 data points).\n",
      "Split on feature grade.B. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (85 data points).\n",
      "Split on feature grade.C. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (85 data points).\n",
      "Split on feature grade.D. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (85 data points).\n",
      "Split on feature grade.E. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (85 data points).\n",
      "Split on feature grade.F. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (85 data points).\n",
      "Split on feature grade.G. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (85 data points).\n",
      "Split on feature term. 60 months. (0, 85)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (85 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (26, 59)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (26 data points).\n",
      "Split on feature emp_length.3 years. (24, 2)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (24 data points).\n",
      "Split on feature home_ownership.OTHER. (24, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (24 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (2 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (59 data points).\n",
      "Split on feature home_ownership.OTHER. (59, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (59 data points).\n",
      "Split on feature home_ownership.OWN. (59, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (59 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (11 data points).\n",
      "Split on feature grade.B. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature grade.C. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (11 data points).\n",
      "Split on feature grade.D. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (11 data points).\n",
      "Split on feature grade.E. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (11 data points).\n",
      "Split on feature grade.F. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (11 data points).\n",
      "Split on feature grade.G. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (11 data points).\n",
      "Split on feature term. 60 months. (0, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (11 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (8, 3)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (8 data points).\n",
      "Split on feature home_ownership.OTHER. (8, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (8 data points).\n",
      "Split on feature home_ownership.OWN. (6, 2)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (6 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (2 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (3 data points).\n",
      "Split on feature home_ownership.OTHER. (3, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (3 data points).\n",
      "Split on feature home_ownership.OWN. (3, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (3 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (5 data points).\n",
      "Split on feature grade.B. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5 data points).\n",
      "Split on feature grade.C. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (5 data points).\n",
      "Split on feature grade.D. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (5 data points).\n",
      "Split on feature grade.E. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (5 data points).\n",
      "Split on feature grade.F. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (5 data points).\n",
      "Split on feature grade.G. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (5 data points).\n",
      "Split on feature term. 60 months. (0, 5)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (5 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (2, 3)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (2 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (3 data points).\n",
      "Split on feature home_ownership.OTHER. (3, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (3 data points).\n",
      "Split on feature home_ownership.OWN. (3, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (3 data points).\n",
      "Split on feature home_ownership.RENT. (3, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (3 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (28001 data points).\n",
      "Split on feature grade.D. (23300, 4701)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (23300 data points).\n",
      "Split on feature grade.E. (22024, 1276)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (22024 data points).\n",
      "Split on feature grade.F. (21666, 358)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (21666 data points).\n",
      "Split on feature emp_length.n/a. (20734, 932)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (20734 data points).\n",
      "Split on feature grade.G. (20638, 96)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (20638 data points).\n",
      "Split on feature grade.A. (15839, 4799)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (15839 data points).\n",
      "Split on feature home_ownership.OTHER. (15811, 28)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (15811 data points).\n",
      "Split on feature grade.B. (6894, 8917)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (6894 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (4102, 2792)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (4102 data points).\n",
      "Split on feature emp_length.4 years. (3768, 334)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (3768 data points).\n",
      "Split on feature emp_length.9 years. (3639, 129)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (3639 data points).\n",
      "Split on feature emp_length.2 years. (3123, 516)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (3123 data points).\n",
      "Split on feature grade.C. (0, 3123)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (3123 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (516 data points).\n",
      "Split on feature home_ownership.OWN. (458, 58)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (458 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (58 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (129 data points).\n",
      "Split on feature home_ownership.OWN. (113, 16)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (113 data points).\n",
      "Split on feature grade.C. (0, 113)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (113 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (16 data points).\n",
      "Split on feature grade.C. (0, 16)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (16 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (334 data points).\n",
      "Split on feature grade.C. (0, 334)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (334 data points).\n",
      "Split on feature term. 60 months. (334, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (334 data points).\n",
      "Split on feature home_ownership.OWN. (286, 48)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (286 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (48 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (2792 data points).\n",
      "Split on feature emp_length.2 years. (2562, 230)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (2562 data points).\n",
      "Split on feature emp_length.5 years. (2335, 227)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (2335 data points).\n",
      "Split on feature grade.C. (0, 2335)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (2335 data points).\n",
      "Split on feature term. 60 months. (2335, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (2335 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (227 data points).\n",
      "Split on feature grade.C. (0, 227)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (227 data points).\n",
      "Split on feature term. 60 months. (227, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (227 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (230 data points).\n",
      "Split on feature grade.C. (0, 230)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (230 data points).\n",
      "Split on feature term. 60 months. (230, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (230 data points).\n",
      "Split on feature home_ownership.OWN. (230, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (230 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (8917 data points).\n",
      "Split on feature grade.C. (8917, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (8917 data points).\n",
      "Split on feature term. 60 months. (8917, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (8917 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (4748, 4169)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (4748 data points).\n",
      "Split on feature home_ownership.OWN. (4089, 659)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (4089 data points).\n",
      "Split on feature home_ownership.RENT. (0, 4089)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (4089 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (659 data points).\n",
      "Split on feature home_ownership.RENT. (659, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (659 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (4169 data points).\n",
      "Split on feature home_ownership.OWN. (4169, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (4169 data points).\n",
      "Split on feature home_ownership.RENT. (4169, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (4169 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (28 data points).\n",
      "Split on feature grade.B. (11, 17)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (11 data points).\n",
      "Split on feature emp_length.6 years. (10, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (10 data points).\n",
      "Split on feature grade.C. (0, 10)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (10 data points).\n",
      "Split on feature term. 60 months. (10, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (10 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (10, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (10 data points).\n",
      "Split on feature home_ownership.OWN. (10, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (10 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (17 data points).\n",
      "Split on feature emp_length.1 year. (16, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (16 data points).\n",
      "Split on feature emp_length.3 years. (15, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (15 data points).\n",
      "Split on feature emp_length.4 years. (14, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (14 data points).\n",
      "Split on feature emp_length.< 1 year. (13, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (13 data points).\n",
      "Split on feature grade.C. (13, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (13 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (4799 data points).\n",
      "Split on feature grade.B. (4799, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (4799 data points).\n",
      "Split on feature grade.C. (4799, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (4799 data points).\n",
      "Split on feature term. 60 months. (4799, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (4799 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (2163, 2636)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (2163 data points).\n",
      "Split on feature home_ownership.OTHER. (2154, 9)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (2154 data points).\n",
      "Split on feature home_ownership.OWN. (1753, 401)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (1753 data points).\n",
      "Split on feature home_ownership.RENT. (0, 1753)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (1753 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (401 data points).\n",
      "Split on feature home_ownership.RENT. (401, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (401 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (9 data points).\n",
      "Split on feature emp_length.3 years. (8, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (8 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (2636 data points).\n",
      "Split on feature home_ownership.OTHER. (2636, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (2636 data points).\n",
      "Split on feature home_ownership.OWN. (2636, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (2636 data points).\n",
      "Split on feature home_ownership.RENT. (2636, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (2636 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (96 data points).\n",
      "Split on feature grade.A. (96, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (96 data points).\n",
      "Split on feature grade.B. (96, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (96 data points).\n",
      "Split on feature grade.C. (96, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (96 data points).\n",
      "Split on feature term. 60 months. (96, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (96 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (44, 52)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (44 data points).\n",
      "Split on feature emp_length.3 years. (43, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (43 data points).\n",
      "Split on feature emp_length.7 years. (42, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (42 data points).\n",
      "Split on feature emp_length.8 years. (41, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (41 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (52 data points).\n",
      "Split on feature emp_length.2 years. (47, 5)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (47 data points).\n",
      "Split on feature home_ownership.OTHER. (47, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (47 data points).\n",
      "Split on feature home_ownership.OWN. (47, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (47 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (5 data points).\n",
      "Split on feature home_ownership.OTHER. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (5 data points).\n",
      "Split on feature home_ownership.OWN. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (5 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (932 data points).\n",
      "Split on feature grade.A. (702, 230)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (702 data points).\n",
      "Split on feature home_ownership.OTHER. (701, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (701 data points).\n",
      "Split on feature grade.B. (317, 384)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (317 data points).\n",
      "Split on feature grade.C. (1, 316)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (316 data points).\n",
      "Split on feature grade.G. (316, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (316 data points).\n",
      "Split on feature term. 60 months. (316, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (316 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (189, 127)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (189 data points).\n",
      "Split on feature home_ownership.OWN. (139, 50)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (139 data points).\n",
      "Split on feature home_ownership.RENT. (0, 139)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (139 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (50 data points).\n",
      "Split on feature home_ownership.RENT. (50, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (50 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (127 data points).\n",
      "Split on feature home_ownership.OWN. (127, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (127 data points).\n",
      "Split on feature home_ownership.RENT. (127, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (127 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (384 data points).\n",
      "Split on feature grade.C. (384, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (384 data points).\n",
      "Split on feature grade.G. (384, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (384 data points).\n",
      "Split on feature term. 60 months. (384, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (384 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (210, 174)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (210 data points).\n",
      "Split on feature home_ownership.OWN. (148, 62)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (148 data points).\n",
      "Split on feature home_ownership.RENT. (0, 148)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (148 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (62 data points).\n",
      "Split on feature home_ownership.RENT. (62, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (62 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (174 data points).\n",
      "Split on feature home_ownership.OWN. (174, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (174 data points).\n",
      "Split on feature home_ownership.RENT. (174, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (174 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (230 data points).\n",
      "Split on feature grade.B. (230, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (230 data points).\n",
      "Split on feature grade.C. (230, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (230 data points).\n",
      "Split on feature grade.G. (230, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (230 data points).\n",
      "Split on feature term. 60 months. (230, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (230 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (119, 111)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (119 data points).\n",
      "Split on feature home_ownership.OTHER. (119, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (119 data points).\n",
      "Split on feature home_ownership.OWN. (71, 48)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (71 data points).\n",
      "Split on feature home_ownership.RENT. (0, 71)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (71 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (48 data points).\n",
      "Split on feature home_ownership.RENT. (48, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (48 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (111 data points).\n",
      "Split on feature home_ownership.OTHER. (111, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (111 data points).\n",
      "Split on feature home_ownership.OWN. (111, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (111 data points).\n",
      "Split on feature home_ownership.RENT. (111, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (111 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (358 data points).\n",
      "Split on feature emp_length.8 years. (347, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (347 data points).\n",
      "Split on feature grade.A. (347, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (347 data points).\n",
      "Split on feature grade.B. (347, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (347 data points).\n",
      "Split on feature grade.C. (347, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (347 data points).\n",
      "Split on feature grade.G. (347, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (347 data points).\n",
      "Split on feature term. 60 months. (347, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (347 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (237, 110)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (237 data points).\n",
      "Split on feature home_ownership.OTHER. (235, 2)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (235 data points).\n",
      "Split on feature home_ownership.OWN. (203, 32)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (203 data points).\n",
      "Split on feature home_ownership.RENT. (0, 203)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (203 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (32 data points).\n",
      "Split on feature home_ownership.RENT. (32, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (32 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (2 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (110 data points).\n",
      "Split on feature home_ownership.OTHER. (110, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (110 data points).\n",
      "Split on feature home_ownership.OWN. (110, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (110 data points).\n",
      "Split on feature home_ownership.RENT. (110, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (110 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature home_ownership.OWN. (9, 2)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (9 data points).\n",
      "Split on feature grade.A. (9, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (9 data points).\n",
      "Split on feature grade.B. (9, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (9 data points).\n",
      "Split on feature grade.C. (9, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (9 data points).\n",
      "Split on feature grade.G. (9, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (9 data points).\n",
      "Split on feature term. 60 months. (9, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (9 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (6, 3)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (6 data points).\n",
      "Split on feature home_ownership.OTHER. (6, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (6 data points).\n",
      "Split on feature home_ownership.RENT. (0, 6)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (6 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (3 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1276 data points).\n",
      "Split on feature grade.A. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (1276 data points).\n",
      "Split on feature grade.B. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (1276 data points).\n",
      "Split on feature grade.C. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1276 data points).\n",
      "Split on feature grade.F. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (1276 data points).\n",
      "Split on feature grade.G. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (1276 data points).\n",
      "Split on feature term. 60 months. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (1276 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (855, 421)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (855 data points).\n",
      "Split on feature home_ownership.OTHER. (849, 6)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (849 data points).\n",
      "Split on feature home_ownership.OWN. (737, 112)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (737 data points).\n",
      "Split on feature home_ownership.RENT. (0, 737)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (737 data points).\n",
      "Split on feature emp_length.1 year. (670, 67)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (670 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (67 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (112 data points).\n",
      "Split on feature home_ownership.RENT. (112, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (112 data points).\n",
      "Split on feature emp_length.1 year. (102, 10)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (102 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (10 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (6 data points).\n",
      "Split on feature home_ownership.OWN. (6, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (6 data points).\n",
      "Split on feature home_ownership.RENT. (6, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (6 data points).\n",
      "Split on feature emp_length.1 year. (6, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (6 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (421 data points).\n",
      "Split on feature emp_length.6 years. (408, 13)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (408 data points).\n",
      "Split on feature home_ownership.OTHER. (408, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (408 data points).\n",
      "Split on feature home_ownership.OWN. (408, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (408 data points).\n",
      "Split on feature home_ownership.RENT. (408, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (408 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (13 data points).\n",
      "Split on feature home_ownership.OTHER. (13, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (13 data points).\n",
      "Split on feature home_ownership.OWN. (13, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (13 data points).\n",
      "Split on feature home_ownership.RENT. (13, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (13 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (4701 data points).\n",
      "Split on feature grade.A. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (4701 data points).\n",
      "Split on feature grade.B. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (4701 data points).\n",
      "Split on feature grade.C. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (4701 data points).\n",
      "Split on feature grade.E. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (4701 data points).\n",
      "Split on feature grade.F. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (4701 data points).\n",
      "Split on feature grade.G. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (4701 data points).\n",
      "Split on feature term. 60 months. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (4701 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (3047, 1654)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (3047 data points).\n",
      "Split on feature home_ownership.OTHER. (3037, 10)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (3037 data points).\n",
      "Split on feature home_ownership.OWN. (2633, 404)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (2633 data points).\n",
      "Split on feature home_ownership.RENT. (0, 2633)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (2633 data points).\n",
      "Split on feature emp_length.1 year. (2392, 241)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (2392 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (241 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (404 data points).\n",
      "Split on feature home_ownership.RENT. (404, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (404 data points).\n",
      "Split on feature emp_length.1 year. (374, 30)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (374 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (30 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (10 data points).\n",
      "Split on feature home_ownership.OWN. (10, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (10 data points).\n",
      "Split on feature home_ownership.RENT. (10, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (10 data points).\n",
      "Split on feature emp_length.1 year. (9, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (9 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (1 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 10 (1654 data points).\n",
      "Split on feature emp_length.5 years. (1532, 122)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (1532 data points).\n",
      "Split on feature emp_length.3 years. (1414, 118)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (1414 data points).\n",
      "Split on feature emp_length.9 years. (1351, 63)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (1351 data points).\n",
      "Split on feature home_ownership.OTHER. (1351, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (1351 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (63 data points).\n",
      "Split on feature home_ownership.OTHER. (63, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (63 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (118 data points).\n",
      "Split on feature home_ownership.OTHER. (118, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (118 data points).\n",
      "Split on feature home_ownership.OWN. (118, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (118 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 11 (122 data points).\n",
      "Split on feature home_ownership.OTHER. (122, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (122 data points).\n",
      "Split on feature home_ownership.OWN. (122, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (122 data points).\n",
      "Split on feature home_ownership.RENT. (122, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (122 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 14 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 13 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 12 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 9 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 8 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 7 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n"
     ]
    }
   ],
   "source": [
    "model_1 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 2, min_node_size = 0, min_error_reduction=-1)\n",
    "model_2 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6, min_node_size = 0, min_error_reduction=-1)\n",
    "model_3 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 14, min_node_size = 0, min_error_reduction=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the models\n",
    "\n",
    "Let us evaluate the models on the **train** and **validation** data. Let us start by evaluating the classification error on the training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data, classification error (model 1): 0.40003761014399314\n",
      "Training data, classification error (model 2): 0.38185041908446166\n",
      "Training data, classification error (model 3): 0.37446271222866967\n"
     ]
    }
   ],
   "source": [
    "print(\"Training data, classification error (model 1):\", evaluate_classification_error(model_1, train_data, target))\n",
    "print(\"Training data, classification error (model 2):\", evaluate_classification_error(model_2, train_data, target))\n",
    "print(\"Training data, classification error (model 3):\", evaluate_classification_error(model_3, train_data, target))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now evaluate the classification error on the validation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data, classification error (model 1): 0.3981042654028436\n",
      "Training data, classification error (model 2): 0.3837785437311504\n",
      "Training data, classification error (model 3): 0.38000861697544164\n"
     ]
    }
   ],
   "source": [
    "print(\"Training data, classification error (model 1):\", evaluate_classification_error(model_1, validation_set, target))\n",
    "print(\"Training data, classification error (model 2):\", evaluate_classification_error(model_2, validation_set, target))\n",
    "print(\"Training data, classification error (model 3):\", evaluate_classification_error(model_3, validation_set, target))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Which tree has the smallest error on the validation data?\n",
    "\n",
    "**A:** model_3\n",
    "\n",
    "**Quiz Question:** Does the tree with the smallest error in the training data also have the smallest error in the validation data?\n",
    "\n",
    "**A:** yes\n",
    "\n",
    "**Quiz Question:** Is it always true that the tree with the lowest classification error on the **training** set will result in the lowest classification error in the **validation** set?\n",
    "\n",
    "**A:** No\n",
    "\n",
    "\n",
    "\n",
    "### Measuring the complexity of the tree\n",
    "\n",
    "Recall in the lecture that we talked about deeper trees being more complex. We will measure the complexity of the tree as\n",
    "\n",
    "```\n",
    "  complexity(T) = number of leaves in the tree T\n",
    "```\n",
    "\n",
    "Here, we provide a function `count_leaves` that counts the number of leaves in a tree. Using this implementation, compute the number of nodes in `model_1`, `model_2`, and `model_3`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_leaves(tree):\n",
    "    if tree['is_leaf']:\n",
    "        return 1\n",
    "    return count_leaves(tree['left']) + count_leaves(tree['right'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the number of nodes in `model_1`, `model_2`, and `model_3`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "41\n",
      "341\n"
     ]
    }
   ],
   "source": [
    "print(count_leaves(model_1))\n",
    "print(count_leaves(model_2))\n",
    "print(count_leaves(model_3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Which tree has the largest complexity?\n",
    "\n",
    "**A:** model_3\n",
    "\n",
    "**Quiz Question:** Is it always true that the most complex tree will result in the lowest classification error in the **validation_set**?\n",
    "\n",
    "**A:** No\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring the effect of min_error\n",
    "\n",
    "We will compare three models trained with different values of the stopping criterion. We intentionally picked models at the extreme ends (**negative**, **just right**, and **too positive**).\n",
    "\n",
    "Train three models with these parameters:\n",
    "1. **model_4**: `min_error_reduction = -1` (ignoring this early stopping condition)\n",
    "2. **model_5**: `min_error_reduction = 0` (just right)\n",
    "3. **model_6**: `min_error_reduction = 5` (too positive)\n",
    "\n",
    "For each of these three, we set `max_depth = 6`, and `min_node_size = 0`.\n",
    "\n",
    "** Note:** Each tree can take up to 30 seconds to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (9223 data points).\n",
      "Split on feature grade.A. (9122, 101)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9122 data points).\n",
      "Split on feature grade.B. (8074, 1048)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (8074 data points).\n",
      "Split on feature grade.C. (5884, 2190)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5884 data points).\n",
      "Split on feature grade.D. (3826, 2058)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (3826 data points).\n",
      "Split on feature grade.E. (1693, 2133)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1693 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2133 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2058 data points).\n",
      "Split on feature grade.E. (2058, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2058 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (2190 data points).\n",
      "Split on feature grade.D. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2190 data points).\n",
      "Split on feature grade.E. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2190 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1048 data points).\n",
      "Split on feature emp_length.5 years. (969, 79)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (969 data points).\n",
      "Split on feature grade.C. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (969 data points).\n",
      "Split on feature grade.D. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (969 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (79 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (34, 45)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (34 data points).\n",
      "Split on feature grade.C. (34, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (34 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (45 data points).\n",
      "Split on feature grade.C. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (45 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (101 data points).\n",
      "Split on feature emp_length.n/a. (96, 5)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (96 data points).\n",
      "Split on feature emp_length.< 1 year. (85, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (85 data points).\n",
      "Split on feature grade.B. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (85 data points).\n",
      "Split on feature grade.C. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (85 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (11 data points).\n",
      "Split on feature grade.B. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature grade.C. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (11 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (5 data points).\n",
      "Split on feature grade.B. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5 data points).\n",
      "Split on feature grade.C. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (5 data points).\n",
      "Split on feature grade.D. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (5 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (28001 data points).\n",
      "Split on feature grade.D. (23300, 4701)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (23300 data points).\n",
      "Split on feature grade.E. (22024, 1276)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (22024 data points).\n",
      "Split on feature grade.F. (21666, 358)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (21666 data points).\n",
      "Split on feature emp_length.n/a. (20734, 932)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (20734 data points).\n",
      "Split on feature grade.G. (20638, 96)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (20638 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (96 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (932 data points).\n",
      "Split on feature grade.A. (702, 230)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (702 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (230 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (358 data points).\n",
      "Split on feature emp_length.8 years. (347, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (347 data points).\n",
      "Split on feature grade.A. (347, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (347 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature home_ownership.OWN. (9, 2)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (9 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1276 data points).\n",
      "Split on feature grade.A. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (1276 data points).\n",
      "Split on feature grade.B. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (1276 data points).\n",
      "Split on feature grade.C. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1276 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (4701 data points).\n",
      "Split on feature grade.A. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (4701 data points).\n",
      "Split on feature grade.B. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (4701 data points).\n",
      "Split on feature grade.C. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (4701 data points).\n",
      "Split on feature grade.E. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (4701 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (9223 data points).\n",
      "Split on feature grade.A. (9122, 101)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9122 data points).\n",
      "Split on feature grade.B. (8074, 1048)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (8074 data points).\n",
      "Split on feature grade.C. (5884, 2190)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5884 data points).\n",
      "Split on feature grade.D. (3826, 2058)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (3826 data points).\n",
      "Split on feature grade.E. (1693, 2133)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1693 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2133 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2058 data points).\n",
      "Split on feature grade.E. (2058, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2058 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (2190 data points).\n",
      "Split on feature grade.D. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2190 data points).\n",
      "Split on feature grade.E. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2190 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1048 data points).\n",
      "Split on feature emp_length.5 years. (969, 79)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (969 data points).\n",
      "Split on feature grade.C. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (969 data points).\n",
      "Split on feature grade.D. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (969 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (79 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (34, 45)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (34 data points).\n",
      "Split on feature grade.C. (34, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (34 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (45 data points).\n",
      "Split on feature grade.C. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (45 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (101 data points).\n",
      "Split on feature emp_length.n/a. (96, 5)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (96 data points).\n",
      "Split on feature emp_length.< 1 year. (85, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (85 data points).\n",
      "Split on feature grade.B. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (85 data points).\n",
      "Split on feature grade.C. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (85 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (11 data points).\n",
      "Split on feature grade.B. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature grade.C. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (11 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (5 data points).\n",
      "Split on feature grade.B. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5 data points).\n",
      "Split on feature grade.C. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (5 data points).\n",
      "Split on feature grade.D. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (5 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (28001 data points).\n",
      "Split on feature grade.D. (23300, 4701)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (23300 data points).\n",
      "Split on feature grade.E. (22024, 1276)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (22024 data points).\n",
      "Split on feature grade.F. (21666, 358)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (21666 data points).\n",
      "Split on feature emp_length.n/a. (20734, 932)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (20734 data points).\n",
      "Split on feature grade.G. (20638, 96)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (20638 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (96 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (932 data points).\n",
      "Split on feature grade.A. (702, 230)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (702 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (230 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (358 data points).\n",
      "Split on feature emp_length.8 years. (347, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (347 data points).\n",
      "Split on feature grade.A. (347, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (347 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature home_ownership.OWN. (9, 2)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (9 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1276 data points).\n",
      "Split on feature grade.A. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (1276 data points).\n",
      "Split on feature grade.B. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (1276 data points).\n",
      "Split on feature grade.C. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1276 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (4701 data points).\n",
      "Split on feature grade.A. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (4701 data points).\n",
      "Split on feature grade.B. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (4701 data points).\n",
      "Split on feature grade.C. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (4701 data points).\n",
      "Split on feature grade.E. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (4701 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Early stopping condition 3 reached. Minimum error reduction.\n"
     ]
    }
   ],
   "source": [
    "model_4 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6, min_node_size = 0, min_error_reduction=-1)\n",
    "model_5 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6, min_node_size = 0, min_error_reduction=0)\n",
    "model_6 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6, min_node_size = 0, min_error_reduction=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the accuracy of each model (**model_4**, **model_5**, or **model_6**) on the validation set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation data, classification error (model 4): 0.3837785437311504\n",
      "Validation data, classification error (model 5): 0.3837785437311504\n",
      "Validation data, classification error (model 6): 0.503446790176648\n"
     ]
    }
   ],
   "source": [
    "print(\"Validation data, classification error (model 4):\", evaluate_classification_error(model_4, validation_set, target))\n",
    "print(\"Validation data, classification error (model 5):\", evaluate_classification_error(model_5, validation_set, target))\n",
    "print(\"Validation data, classification error (model 6):\", evaluate_classification_error(model_6, validation_set, target))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `count_leaves` function, compute the number of leaves in each of each models in (**model_4**, **model_5**, and **model_6**). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "41\n",
      "41\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(count_leaves(model_4))\n",
    "print(count_leaves(model_5))\n",
    "print(count_leaves(model_6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Using the complexity definition above, which model (**model_4**, **model_5**, or **model_6**) has the largest complexity?\n",
    "\n",
    "Did this match your expectation?\n",
    "\n",
    "**A:** 4 and 5\n",
    "\n",
    "**Quiz Question:** **model_4** and **model_5** have similar classification error on the validation set but **model_5** has lower complexity. Should you pick **model_5** over **model_4**?\n",
    "\n",
    "**A:** choose model_5, if it has lower complexity\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring the effect of min_node_size\n",
    "\n",
    "We will compare three models trained with different values of the stopping criterion. Again, intentionally picked models at the extreme ends (**too small**, **just right**, and **just right**).\n",
    "\n",
    "Train three models with these parameters:\n",
    "1. **model_7**: min_node_size = 0 (too small)\n",
    "2. **model_8**: min_node_size = 2000 (just right)\n",
    "3. **model_9**: min_node_size = 50000 (too large)\n",
    "\n",
    "For each of these three, we set `max_depth = 6`, and `min_error_reduction = -1`.\n",
    "\n",
    "** Note:** Each tree can take up to 30 seconds to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (9223 data points).\n",
      "Split on feature grade.A. (9122, 101)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9122 data points).\n",
      "Split on feature grade.B. (8074, 1048)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (8074 data points).\n",
      "Split on feature grade.C. (5884, 2190)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5884 data points).\n",
      "Split on feature grade.D. (3826, 2058)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (3826 data points).\n",
      "Split on feature grade.E. (1693, 2133)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1693 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2133 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2058 data points).\n",
      "Split on feature grade.E. (2058, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2058 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (2190 data points).\n",
      "Split on feature grade.D. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2190 data points).\n",
      "Split on feature grade.E. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2190 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1048 data points).\n",
      "Split on feature emp_length.5 years. (969, 79)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (969 data points).\n",
      "Split on feature grade.C. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (969 data points).\n",
      "Split on feature grade.D. (969, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (969 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (79 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (34, 45)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (34 data points).\n",
      "Split on feature grade.C. (34, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (34 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (45 data points).\n",
      "Split on feature grade.C. (45, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (45 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (101 data points).\n",
      "Split on feature emp_length.n/a. (96, 5)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (96 data points).\n",
      "Split on feature emp_length.< 1 year. (85, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (85 data points).\n",
      "Split on feature grade.B. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (85 data points).\n",
      "Split on feature grade.C. (85, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (85 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (11 data points).\n",
      "Split on feature grade.B. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature grade.C. (11, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (11 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (5 data points).\n",
      "Split on feature grade.B. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5 data points).\n",
      "Split on feature grade.C. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (5 data points).\n",
      "Split on feature grade.D. (5, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (5 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (28001 data points).\n",
      "Split on feature grade.D. (23300, 4701)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (23300 data points).\n",
      "Split on feature grade.E. (22024, 1276)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (22024 data points).\n",
      "Split on feature grade.F. (21666, 358)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (21666 data points).\n",
      "Split on feature emp_length.n/a. (20734, 932)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (20734 data points).\n",
      "Split on feature grade.G. (20638, 96)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (20638 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (96 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (932 data points).\n",
      "Split on feature grade.A. (702, 230)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (702 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (230 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (358 data points).\n",
      "Split on feature emp_length.8 years. (347, 11)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (347 data points).\n",
      "Split on feature grade.A. (347, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (347 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (11 data points).\n",
      "Split on feature home_ownership.OWN. (9, 2)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (9 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1276 data points).\n",
      "Split on feature grade.A. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (1276 data points).\n",
      "Split on feature grade.B. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (1276 data points).\n",
      "Split on feature grade.C. (1276, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1276 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (4701 data points).\n",
      "Split on feature grade.A. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (4701 data points).\n",
      "Split on feature grade.B. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (4701 data points).\n",
      "Split on feature grade.C. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (4701 data points).\n",
      "Split on feature grade.E. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (4701 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (9223 data points).\n",
      "Split on feature grade.A. (9122, 101)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9122 data points).\n",
      "Split on feature grade.B. (8074, 1048)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (8074 data points).\n",
      "Split on feature grade.C. (5884, 2190)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (5884 data points).\n",
      "Split on feature grade.D. (3826, 2058)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (3826 data points).\n",
      "Split on feature grade.E. (1693, 2133)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (1693 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2133 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2058 data points).\n",
      "Split on feature grade.E. (2058, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2058 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (2190 data points).\n",
      "Split on feature grade.D. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (2190 data points).\n",
      "Split on feature grade.E. (2190, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (2190 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1048 data points).\n",
      "Early stopping condition 2 reached. Reached minimum node size.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (101 data points).\n",
      "Early stopping condition 2 reached. Reached minimum node size.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (28001 data points).\n",
      "Split on feature grade.D. (23300, 4701)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (23300 data points).\n",
      "Split on feature grade.E. (22024, 1276)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (22024 data points).\n",
      "Split on feature grade.F. (21666, 358)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (21666 data points).\n",
      "Split on feature emp_length.n/a. (20734, 932)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (20734 data points).\n",
      "Split on feature grade.G. (20638, 96)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (20638 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (96 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (932 data points).\n",
      "Early stopping condition 2 reached. Reached minimum node size.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (358 data points).\n",
      "Early stopping condition 2 reached. Reached minimum node size.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1276 data points).\n",
      "Early stopping condition 2 reached. Reached minimum node size.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (4701 data points).\n",
      "Split on feature grade.A. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (4701 data points).\n",
      "Split on feature grade.B. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (4701 data points).\n",
      "Split on feature grade.C. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (4701 data points).\n",
      "Split on feature grade.E. (4701, 0)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (4701 data points).\n",
      "Early stopping condition 1 reached. Reached maximum depth.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 6 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 5 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 4 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (0 data points).\n",
      "Stopping condition 1 reached. All data points have the same target value.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 0 (37224 data points).\n",
      "Early stopping condition 2 reached. Reached minimum node size.\n"
     ]
    }
   ],
   "source": [
    "model_7 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6, min_node_size = 0, min_error_reduction=-1)\n",
    "model_8 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6, min_node_size = 2000, min_error_reduction=-1)\n",
    "model_9 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6, min_node_size = 50000, min_error_reduction=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us evaluate the models (**model_7**, **model_8**, or **model_9**) on the **validation_set**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation data, classification error (model 7): 0.3837785437311504\n",
      "Validation data, classification error (model 8): 0.38453252908229213\n",
      "Validation data, classification error (model 9): 0.503446790176648\n"
     ]
    }
   ],
   "source": [
    "print(\"Validation data, classification error (model 7):\", evaluate_classification_error(model_7, validation_set, target))\n",
    "print(\"Validation data, classification error (model 8):\", evaluate_classification_error(model_8, validation_set, target))\n",
    "print(\"Validation data, classification error (model 9):\", evaluate_classification_error(model_9, validation_set, target))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `count_leaves` function, compute the number of leaves in each of each models (**model_7**, **model_8**, and **model_9**). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "41\n",
      "19\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(count_leaves(model_7))\n",
    "print(count_leaves(model_8))\n",
    "print(count_leaves(model_9))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Using the results obtained in this section, which model (**model_7**, **model_8**, or **model_9**) would you choose to use?\n",
    "\n",
    "**A:** model_8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Finished parsing file /home/yomomma/Downloads/GOOG.csv</pre>"
      ],
      "text/plain": [
       "Finished parsing file /home/yomomma/Downloads/GOOG.csv"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Parsing completed. Parsed 100 lines in 0.023327 secs.</pre>"
      ],
      "text/plain": [
       "Parsing completed. Parsed 100 lines in 0.023327 secs."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------\n",
      "Inferred types from first 100 line(s) of file as \n",
      "column_type_hints=[str,float,float,float,float,float,int]\n",
      "If parsing fails due to incorrect types, you can correct\n",
      "the inferred type list above and pass it to read_csv in\n",
      "the column_type_hints argument\n",
      "------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre>Finished parsing file /home/yomomma/Downloads/GOOG.csv</pre>"
      ],
      "text/plain": [
       "Finished parsing file /home/yomomma/Downloads/GOOG.csv"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Parsing completed. Parsed 858 lines in 0.023114 secs.</pre>"
      ],
      "text/plain": [
       "Parsing completed. Parsed 858 lines in 0.023114 secs."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import turicreate as tc\n",
    "\n",
    "# Load the data\n",
    "data =  tc.SFrame('GOOG.csv')\n",
    "\n",
    "# Make a train-test split\n",
    "train_data, test_data = data.random_split(0.8)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data['day_trend'] = train_data['Close'] - train_data['Open'] > 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">Date</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">Open</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">High</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">Low</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">Close</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">Adj Close</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">Volume</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">day_trend</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004-08-16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">49.813286</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">54.336334</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">47.800831</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">53.95277</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">53.95277</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">67814100</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004-08-30</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">52.443428</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">52.548038</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">49.285267</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">49.818268</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">49.818268</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">39711400</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004-09-06</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">50.316402</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">53.081039</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">49.619015</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">52.468334</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">52.468334</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">23705500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004-09-13</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">53.11591</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">58.525631</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">53.031227</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">58.525631</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">58.525631</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">48353100</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004-09-20</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">58.256641</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">61.818291</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">58.166977</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">59.691261</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">59.691261</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">43302100</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004-09-27</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">59.556767</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">67.257904</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">58.680054</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">66.042458</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">66.042458</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">83790400</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004-10-04</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">67.372475</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">69.678825</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">65.873093</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">68.607841</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">68.607841</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">66877800</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004-10-11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">68.244202</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">72.478333</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">66.450928</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">71.785927</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">71.785927</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">65851700</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004-10-18</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">71.332626</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">89.748604</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">69.539352</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">85.893051</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">85.893051</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">158477900</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004-10-25</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">87.870636</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">99.601669</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">85.952827</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">94.96405</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">94.96405</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">209656800</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[10 rows x 8 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tDate\tstr\n",
       "\tOpen\tfloat\n",
       "\tHigh\tfloat\n",
       "\tLow\tfloat\n",
       "\tClose\tfloat\n",
       "\tAdj Close\tfloat\n",
       "\tVolume\tint\n",
       "\tday_trend\tint\n",
       "\n",
       "Rows: 10\n",
       "\n",
       "Data:\n",
       "+------------+-----------+-----------+-----------+-----------+-----------+\n",
       "|    Date    |    Open   |    High   |    Low    |   Close   | Adj Close |\n",
       "+------------+-----------+-----------+-----------+-----------+-----------+\n",
       "| 2004-08-16 | 49.813286 | 54.336334 | 47.800831 |  53.95277 |  53.95277 |\n",
       "| 2004-08-30 | 52.443428 | 52.548038 | 49.285267 | 49.818268 | 49.818268 |\n",
       "| 2004-09-06 | 50.316402 | 53.081039 | 49.619015 | 52.468334 | 52.468334 |\n",
       "| 2004-09-13 |  53.11591 | 58.525631 | 53.031227 | 58.525631 | 58.525631 |\n",
       "| 2004-09-20 | 58.256641 | 61.818291 | 58.166977 | 59.691261 | 59.691261 |\n",
       "| 2004-09-27 | 59.556767 | 67.257904 | 58.680054 | 66.042458 | 66.042458 |\n",
       "| 2004-10-04 | 67.372475 | 69.678825 | 65.873093 | 68.607841 | 68.607841 |\n",
       "| 2004-10-11 | 68.244202 | 72.478333 | 66.450928 | 71.785927 | 71.785927 |\n",
       "| 2004-10-18 | 71.332626 | 89.748604 | 69.539352 | 85.893051 | 85.893051 |\n",
       "| 2004-10-25 | 87.870636 | 99.601669 | 85.952827 |  94.96405 |  94.96405 |\n",
       "+------------+-----------+-----------+-----------+-----------+-----------+\n",
       "+-----------+-----------+\n",
       "|   Volume  | day_trend |\n",
       "+-----------+-----------+\n",
       "|  67814100 |     1     |\n",
       "|  39711400 |     0     |\n",
       "|  23705500 |     1     |\n",
       "|  48353100 |     1     |\n",
       "|  43302100 |     1     |\n",
       "|  83790400 |     1     |\n",
       "|  66877800 |     1     |\n",
       "|  65851700 |     1     |\n",
       "| 158477900 |     1     |\n",
       "| 209656800 |     1     |\n",
       "+-----------+-----------+\n",
       "[10 rows x 8 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(train_data[train_data['day_trend'] == 1]) / len(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Automatically generating validation set from 5% of the data.</pre>"
      ],
      "text/plain": [
       "Automatically generating validation set from 5% of the data."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Linear regression:</pre>"
      ],
      "text/plain": [
       "Linear regression:"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of examples          : 663</pre>"
      ],
      "text/plain": [
       "Number of examples          : 663"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of features          : 1</pre>"
      ],
      "text/plain": [
       "Number of features          : 1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of unpacked features : 1</pre>"
      ],
      "text/plain": [
       "Number of unpacked features : 1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of coefficients    : 663</pre>"
      ],
      "text/plain": [
       "Number of coefficients    : 663"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Starting L-BFGS</pre>"
      ],
      "text/plain": [
       "Starting L-BFGS"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+-----------+--------------+--------------------+----------------------+---------------------------------+-----------------------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+-----------+--------------+--------------------+----------------------+---------------------------------+-----------------------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Iteration | Passes   | Step size | Elapsed Time | Training Max Error | Validation Max Error | Training Root-Mean-Square Error | Validation Root-Mean-Square Error |</pre>"
      ],
      "text/plain": [
       "| Iteration | Passes   | Step size | Elapsed Time | Training Max Error | Validation Max Error | Training Root-Mean-Square Error | Validation Root-Mean-Square Error |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+-----------+--------------+--------------------+----------------------+---------------------------------+-----------------------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+-----------+--------------+--------------------+----------------------+---------------------------------+-----------------------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Warning: Reached max step size.</pre>"
      ],
      "text/plain": [
       "Warning: Reached max step size."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 0         | 5        | 25.000000 | 0.001310     | 1827.741515        | 1761.692120          | 691.053021                      | 707.199110                        |</pre>"
      ],
      "text/plain": [
       "| 0         | 5        | 25.000000 | 0.001310     | 1827.741515        | 1761.692120          | 691.053021                      | 707.199110                        |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 1         | 10       | 1.019844  | 0.020669     | 242.768564         | 1465.028666          | 77.307914                       | 522.790947                        |</pre>"
      ],
      "text/plain": [
       "| 1         | 10       | 1.019844  | 0.020669     | 242.768564         | 1465.028666          | 77.307914                       | 522.790947                        |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 2         | 15       | 1.019844  | 0.022078     | 229.776584         | 1478.020646          | 27.119287                       | 528.732643                        |</pre>"
      ],
      "text/plain": [
       "| 2         | 15       | 1.019844  | 0.022078     | 229.776584         | 1478.020646          | 27.119287                       | 528.732643                        |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 3         | 17       | 1.019844  | 0.023061     | 222.982952         | 1484.814278          | 8.681384                        | 531.939542                        |</pre>"
      ],
      "text/plain": [
       "| 3         | 17       | 1.019844  | 0.023061     | 222.982952         | 1484.814278          | 8.681384                        | 531.939542                        |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Warning: Reached max step size.</pre>"
      ],
      "text/plain": [
       "Warning: Reached max step size."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 4         | 25       | 25.000000 | 0.024887     | 219.798307         | 1487.998923          | 8.537936                        | 533.465986                        |</pre>"
      ],
      "text/plain": [
       "| 4         | 25       | 25.000000 | 0.024887     | 219.798307         | 1487.998923          | 8.537936                        | 533.465986                        |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 6         | 34       | 1.000000  | 0.027939     | 4.961209           | 1702.836021          | 0.192923                        | 664.179580                        |</pre>"
      ],
      "text/plain": [
       "| 6         | 34       | 1.000000  | 0.027939     | 4.961209           | 1702.836021          | 0.192923                        | 664.179580                        |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+-----------+--------------+--------------------+----------------------+---------------------------------+-----------------------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+-----------+--------------+--------------------+----------------------+---------------------------------+-----------------------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'max_error': 1582.0860210668332, 'rmse': 648.7685395458972}\n"
     ]
    }
   ],
   "source": [
    "# Automatically picks the right model based on your data.\n",
    "model = tc.regression.create(train_data, target='Adj Close',\n",
    "                                    features = ['Date'])\n",
    "\n",
    "# Save predictions to an SArray\n",
    "predictions = model.predict(test_data)\n",
    "\n",
    "# Evaluate the model and save the results into a dictionary\n",
    "results = model.evaluate(test_data)\n",
    "\n",
    "\n",
    "# Apply the classify(tree, x) to each row in your data\n",
    "# predictions = test.apply(lambda x: model.predicttree, x))\n",
    "\n",
    "# # Once you've made the predictions, calculate the classification error and return it\n",
    "# ## YOUR CODE HERE\n",
    "# mistakes = 0\n",
    "# for i, p in enumerate(predictions):\n",
    "#     if p != data[i][target]:\n",
    "#         mistakes += 1\n",
    "\n",
    "# return mistakes / len(data)\n",
    "\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}